Hourly Usage
Performance Metrics
Median Total Time
28.90s
Median TTFT
6.56s
Median Prefill TPS
1058.80
Median Gen TPS
6.25
Model Information
Context Size
262144
Quantization
r64 on INT8
Engine
vllm
Creation Method
Merge
Model Type
Gemma31B
Chat Template
Gemma4
Reasoning
Yes
Vision
Yes
Parameters
31B
Added At
9/26/2026
license: apache-2.0 base_model:
- google/gemma-4-31B-it
- ReadyArt/gemma-4-31B-it-scotoma-2
- bgg1996/Melinoe-Gemma4-31B-VL-heretic
- sophosympatheia/Glistening-Gem-31B-v2.1
- sophosympatheia/Glistening-Gem-31B-v2.0
- zerofata/G4-MeroMero-v2-31B
- Ateron/Gemma-4-Dark-Thoughts-V2-31B
- Gryphe/Pantheon-Reasoning-31B-1.1 base_model_relation: merge tags:
- merge
- mergekit
- roleplay
- character-rp
- creative writing
- storytelling
- conversational
- low-refusal
- nsfw
- 31B
- gemma-4
- untied-lm_head
by Nimbz
SPLIT UNTIED
31B · Gemma 4 · StyleSwap Merge
“Dismiss, and grow.”
FIVE MODELS
THREE PHASES
UNTIED HEAD
EXPERIMENTAL
DARK LEAN
ROLEPLAY MERGE
◖◗🔵 or 🔴
Basically the same, they share a brain, yet differ on their speaking vein.
SPLIT-31B◖ the other headPick this one if you want instruction adherence for things like trackers or header and footer blocks. Expect a more structured output, and staying on track for long sessions.
SPLIT-UNTIED-31B◗ you are herePick this one if you want more creativity, a darker lean and more lewdness, but only if you're ready to trade a little prompt adherence for it.
⬛GGUF quants
Thank you to anyone providing quants - mradermacher was diligent!
☰The merge
Two della_linear passes over gemma-4-31B-it, on zerofata’s gemma-4-support fork, plus the StyleSwap grafting on top.
I
Comprehension
Of The Residual StreamThe theory: Adjust q_proj + k_proj for the attention mechanism, and alter gate_proj + up_proj for the knowledge routing.
Model
Purpose
Scope
Melinoe-VL-heretic
persona tracking, world-state stability
q/k 0.45 → 0.30 by depth · spine q/k 0.32
gate/up 0.28 → 0.18
gate/up 0.28 → 0.18
Glistening-Gem v2.1
format adherence, in-character
q/k 0.14 → 0.10 · spine 0.0
gate/up 0.06
gate/up 0.06
MeroMero v2
swipe variety
gate/up only · 0.30 · layers 6-47 minus spine
▸ Phase 1: della_linear
merge_method: della_linear
base_model: ./google/gemma-4-31B-it
dtype: bfloat16
tokenizer_source: base
parameters:
normalize: false
epsilon: 0.05
lambda: 1.0
int8_mask: true
models:
- model: ./bgg1996/Melinoe-Gemma4-31B-VL-heretic
parameters:
density:
- { filter: norm, value: 1.0 }
- { filter: layer_scalar, value: 1.0 }
- { value: 0.85 }
weight:
- { filter: embed_tokens, value: 0.0 }
- { filter: lm_head, value: 0.0 }
- { filter: embed_vision, value: 0.0 }
- { filter: vision_tower, value: 0.0 }
- { filter: multi_modal_projector, value: 0.0 }
- { filter: model.language_model.layers.5.self_attn.q_proj, value: 0.32 }
- { filter: model.language_model.layers.5.self_attn.k_proj, value: 0.32 }
- { filter: model.language_model.layers.11.self_attn.q_proj, value: 0.32 }
- { filter: model.language_model.layers.11.self_attn.k_proj, value: 0.32 }
- { filter: model.language_model.layers.17.self_attn.q_proj, value: 0.32 }
- { filter: model.language_model.layers.17.self_attn.k_proj, value: 0.32 }
- { filter: model.language_model.layers.23.self_attn.q_proj, value: 0.32 }
- { filter: model.language_model.layers.23.self_attn.k_proj, value: 0.32 }
- { filter: model.language_model.layers.29.self_attn.q_proj, value: 0.32 }
- { filter: model.language_model.layers.29.self_attn.k_proj, value: 0.32 }
- { filter: model.language_model.layers.35.self_attn.q_proj, value: 0.32 }
- { filter: model.language_model.layers.35.self_attn.k_proj, value: 0.32 }
- { filter: model.language_model.layers.41.self_attn.q_proj, value: 0.32 }
- { filter: model.language_model.layers.41.self_attn.k_proj, value: 0.32 }
- { filter: model.language_model.layers.47.self_attn.q_proj, value: 0.32 }
- { filter: model.language_model.layers.47.self_attn.k_proj, value: 0.32 }
- { filter: model.language_model.layers.53.self_attn.q_proj, value: 0.32 }
- { filter: model.language_model.layers.53.self_attn.k_proj, value: 0.32 }
- { filter: model.language_model.layers.59.self_attn.q_proj, value: 0.32 }
- { filter: model.language_model.layers.59.self_attn.k_proj, value: 0.32 }
- { filter: model.language_model.layers.0.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.0.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.1.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.1.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.2.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.2.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.3.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.3.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.4.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.4.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.5.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.5.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.6.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.6.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.7.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.7.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.8.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.8.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.9.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.9.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.10.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.10.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.11.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.11.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.12.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.12.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.13.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.13.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.14.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.14.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.15.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.15.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.16.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.16.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.17.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.17.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.18.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.18.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.19.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.19.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.20.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.20.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.21.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.21.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.22.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.22.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.23.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.23.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.24.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.24.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.25.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.25.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.26.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.26.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.27.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.27.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.28.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.28.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.29.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.29.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.30.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.30.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.31.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.31.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.32.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.32.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.33.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.33.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.34.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.34.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.35.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.35.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.36.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.36.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.37.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.37.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.38.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.38.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.39.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.39.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.40.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.40.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.41.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.41.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.42.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.42.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.43.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.43.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.44.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.44.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.45.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.45.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.46.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.46.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.47.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.47.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.48.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.48.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.49.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.49.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.50.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.50.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.51.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.51.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.52.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.52.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.53.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.53.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.54.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.54.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.55.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.55.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.56.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.56.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.57.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.57.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.58.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.58.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.59.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.59.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.0.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.0.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.1.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.1.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.2.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.2.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.3.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.3.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.4.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.4.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.5.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.5.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.6.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.6.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.7.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.7.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.8.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.8.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.9.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.9.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.10.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.10.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.11.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.11.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.12.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.12.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.13.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.13.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.14.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.14.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.15.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.15.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.16.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.16.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.17.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.17.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.18.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.18.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.19.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.19.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.20.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.20.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.21.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.21.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.22.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.22.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.23.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.23.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.24.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.24.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.25.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.25.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.26.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.26.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.27.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.27.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.28.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.28.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.29.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.29.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.30.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.30.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.31.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.31.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.32.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.32.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.33.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.33.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.34.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.34.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.35.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.35.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.36.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.36.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.37.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.37.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.38.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.38.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.39.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.39.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.40.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.40.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.41.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.41.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.42.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.42.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.43.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.43.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.44.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.44.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.45.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.45.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.46.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.46.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.47.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.47.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.48.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.48.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.49.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.49.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.50.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.50.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.51.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.51.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.52.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.52.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.53.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.53.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.54.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.54.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.55.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.55.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.56.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.56.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.57.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.57.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.58.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.58.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.59.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.59.mlp.up_proj, value: 0.18 }
- { value: 0.0 }
- model: ./sophosympatheia/Glistening-Gem-31B-v2.1
parameters:
density:
- { filter: norm, value: 1.0 }
- { filter: layer_scalar, value: 1.0 }
- { value: 0.55 }
weight:
- { filter: embed_tokens, value: 0.0 }
- { filter: lm_head, value: 0.0 }
- { filter: embed_vision, value: 0.0 }
- { filter: vision_tower, value: 0.0 }
- { filter: multi_modal_projector, value: 0.0 }
- { filter: model.language_model.layers.5.self_attn.q_proj, value: 0.0 }
- { filter: model.language_model.layers.5.self_attn.k_proj, value: 0.0 }
- { filter: model.language_model.layers.11.self_attn.q_proj, value: 0.0 }
- { filter: model.language_model.layers.11.self_attn.k_proj, value: 0.0 }
- { filter: model.language_model.layers.17.self_attn.q_proj, value: 0.0 }
- { filter: model.language_model.layers.17.self_attn.k_proj, value: 0.0 }
- { filter: model.language_model.layers.23.self_attn.q_proj, value: 0.0 }
- { filter: model.language_model.layers.23.self_attn.k_proj, value: 0.0 }
- { filter: model.language_model.layers.29.self_attn.q_proj, value: 0.0 }
- { filter: model.language_model.layers.29.self_attn.k_proj, value: 0.0 }
- { filter: model.language_model.layers.35.self_attn.q_proj, value: 0.0 }
- { filter: model.language_model.layers.35.self_attn.k_proj, value: 0.0 }
- { filter: model.language_model.layers.41.self_attn.q_proj, value: 0.0 }
- { filter: model.language_model.layers.41.self_attn.k_proj, value: 0.0 }
- { filter: model.language_model.layers.47.self_attn.q_proj, value: 0.0 }
- { filter: model.language_model.layers.47.self_attn.k_proj, value: 0.0 }
- { filter: model.language_model.layers.53.self_attn.q_proj, value: 0.0 }
- { filter: model.language_model.layers.53.self_attn.k_proj, value: 0.0 }
- { filter: model.language_model.layers.59.self_attn.q_proj, value: 0.0 }
- { filter: model.language_model.layers.59.self_attn.k_proj, value: 0.0 }
- { filter: model.language_model.layers.0.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.0.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.1.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.1.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.2.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.2.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.3.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.3.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.4.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.4.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.5.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.5.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.6.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.6.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.7.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.7.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.8.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.8.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.9.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.9.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.10.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.10.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.11.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.11.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.12.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.12.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.13.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.13.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.14.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.14.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.15.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.15.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.16.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.16.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.17.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.17.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.18.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.18.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.19.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.19.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.20.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.20.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.21.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.21.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.22.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.22.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.23.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.23.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.24.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.24.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.25.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.25.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.26.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.26.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.27.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.27.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.28.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.28.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.29.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.29.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.30.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.30.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.31.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.31.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.32.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.32.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.33.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.33.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.34.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.34.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.35.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.35.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.36.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.36.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.37.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.37.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.38.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.38.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.39.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.39.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.40.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.40.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.41.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.41.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.42.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.42.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.43.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.43.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.44.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.44.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.45.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.45.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.46.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.46.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.47.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.47.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.48.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.48.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.49.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.49.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.50.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.50.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.51.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.51.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.52.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.52.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.53.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.53.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.54.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.54.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.55.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.55.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.56.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.56.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.57.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.57.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.58.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.58.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.59.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.59.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.0.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.0.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.1.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.1.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.2.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.2.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.3.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.3.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.4.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.4.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.5.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.5.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.6.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.6.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.7.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.7.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.8.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.8.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.9.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.9.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.10.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.10.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.11.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.11.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.12.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.12.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.13.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.13.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.14.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.14.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.15.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.15.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.16.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.16.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.17.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.17.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.18.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.18.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.19.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.19.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.20.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.20.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.21.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.21.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.22.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.22.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.23.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.23.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.24.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.24.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.25.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.25.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.26.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.26.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.27.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.27.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.28.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.28.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.29.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.29.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.30.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.30.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.31.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.31.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.32.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.32.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.33.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.33.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.34.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.34.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.35.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.35.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.36.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.36.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.37.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.37.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.38.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.38.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.39.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.39.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.40.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.40.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.41.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.41.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.42.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.42.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.43.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.43.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.44.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.44.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.45.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.45.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.46.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.46.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.47.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.47.mlp.up_proj, value: 0.06 }
- { value: 0.0 }
- model: ./zerofata/G4-MeroMero-v2-31B
parameters:
density:
- { filter: norm, value: 1.0 }
- { filter: layer_scalar, value: 1.0 }
- { value: 0.60 }
weight:
- { filter: embed_tokens, value: 0.0 }
- { filter: lm_head, value: 0.0 }
- { filter: embed_vision, value: 0.0 }
- { filter: vision_tower, value: 0.0 }
- { filter: multi_modal_projector, value: 0.0 }
- { filter: model.language_model.layers.5., value: 0.0 }
- { filter: model.language_model.layers.11., value: 0.0 }
- { filter: model.language_model.layers.17., value: 0.0 }
- { filter: model.language_model.layers.23., value: 0.0 }
- { filter: model.language_model.layers.29., value: 0.0 }
- { filter: model.language_model.layers.35., value: 0.0 }
- { filter: model.language_model.layers.41., value: 0.0 }
- { filter: model.language_model.layers.47., value: 0.0 }
- { filter: model.language_model.layers.53., value: 0.0 }
- { filter: model.language_model.layers.59., value: 0.0 }
- { filter: model.language_model.layers.6.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.6.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.7.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.7.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.8.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.8.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.9.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.9.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.10.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.10.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.12.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.12.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.13.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.13.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.14.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.14.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.15.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.15.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.16.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.16.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.18.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.18.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.19.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.19.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.20.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.20.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.21.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.21.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.22.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.22.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.24.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.24.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.25.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.25.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.26.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.26.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.27.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.27.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.28.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.28.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.30.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.30.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.31.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.31.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.32.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.32.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.33.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.33.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.34.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.34.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.36.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.36.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.37.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.37.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.38.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.38.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.39.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.39.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.40.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.40.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.42.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.42.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.43.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.43.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.44.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.44.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.45.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.45.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.46.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.46.mlp.up_proj, value: 0.3 }
- { value: 0.0 }
II
Voice
Of The Residual StreamIf that also checks out, then v_proj steers the expression of what q/k came up with, and o_proj + mlp.down_proj are the layers that coordinate how the output should be written.
Model
Purpose
Scope
scotoma-2
Anti-Gemma tics
weight 1.0 · density 1.0 · layers 7-41
alone on 7-11
alone on 7-11
Dark-Thoughts V2
darker lean, less sanitizing
v/o/down · 0.50 (24-47) → 0.30 (48-53)
Glistening-Gem v2.0
self-repetition, coherence
v/o/down · 0.22 · layers 12-41
Melinoe-VL-heretic
willingness
v/o/down · 0.30 · layers 42-53
▸ Phase 2: della_linear
merge_method: della_linear
base_model: ./Split-31B-read
dtype: bfloat16
tokenizer_source: base
parameters:
normalize: false
epsilon: 0.05
lambda: 1.0
int8_mask: true
models:
- model: ./ReadyArt/gemma-4-31B-it-scotoma-2
parameters:
density:
- { filter: norm, value: 1.0 }
- { filter: layer_scalar, value: 1.0 }
- { value: 1.0 }
weight:
- { filter: embed_tokens, value: 0.0 }
- { filter: lm_head, value: 0.0 }
- { filter: embed_vision, value: 0.0 }
- { filter: vision_tower, value: 0.0 }
- { filter: multi_modal_projector, value: 0.0 }
- { filter: model.language_model.layers.7.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.7.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.8.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.8.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.9.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.9.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.10.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.10.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.11.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.11.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.12.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.12.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.13.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.13.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.14.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.14.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.15.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.15.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.16.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.16.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.17.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.17.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.18.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.18.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.19.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.19.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.20.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.20.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.21.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.21.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.22.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.22.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.23.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.23.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.24.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.24.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.25.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.25.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.26.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.26.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.27.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.27.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.28.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.28.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.29.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.29.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.30.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.30.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.31.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.31.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.32.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.32.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.33.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.33.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.34.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.34.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.35.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.35.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.36.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.36.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.37.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.37.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.38.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.38.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.39.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.39.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.40.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.40.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.41.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.41.mlp.down_proj, value: 1.0 }
- { value: 0.0 }
- model: ./Ateron/Gemma-4-Dark-Thoughts-V2-31B
parameters:
density:
- { filter: norm, value: 1.0 }
- { filter: layer_scalar, value: 1.0 }
- { value: 0.75 }
weight:
- { filter: embed_tokens, value: 0.0 }
- { filter: lm_head, value: 0.0 }
- { filter: embed_vision, value: 0.0 }
- { filter: vision_tower, value: 0.0 }
- { filter: multi_modal_projector, value: 0.0 }
- { filter: model.language_model.layers.24.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.24.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.24.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.25.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.25.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.25.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.26.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.26.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.26.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.27.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.27.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.27.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.28.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.28.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.28.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.29.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.29.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.30.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.30.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.30.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.31.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.31.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.31.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.32.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.32.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.32.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.33.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.33.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.33.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.34.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.34.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.34.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.35.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.35.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.36.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.36.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.36.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.37.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.37.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.37.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.38.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.38.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.38.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.39.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.39.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.39.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.40.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.40.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.40.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.41.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.41.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.42.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.42.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.42.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.43.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.43.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.43.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.44.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.44.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.44.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.45.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.45.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.45.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.46.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.46.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.46.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.47.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.47.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.48.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.48.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.48.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.49.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.49.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.49.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.50.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.50.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.50.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.51.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.51.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.51.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.52.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.52.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.52.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.53.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.53.mlp.down_proj, value: 0.3 }
- { value: 0.0 }
- model: ./sophosympatheia/Glistening-Gem-31B-v2.0
parameters:
density:
- { filter: norm, value: 1.0 }
- { filter: layer_scalar, value: 1.0 }
- { value: 0.60 }
weight:
- { filter: embed_tokens, value: 0.0 }
- { filter: lm_head, value: 0.0 }
- { filter: embed_vision, value: 0.0 }
- { filter: vision_tower, value: 0.0 }
- { filter: multi_modal_projector, value: 0.0 }
- { filter: model.language_model.layers.12.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.12.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.12.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.13.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.13.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.13.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.14.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.14.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.14.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.15.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.15.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.15.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.16.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.16.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.16.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.17.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.17.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.18.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.18.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.18.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.19.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.19.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.19.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.20.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.20.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.20.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.21.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.21.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.21.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.22.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.22.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.22.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.23.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.23.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.24.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.24.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.24.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.25.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.25.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.25.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.26.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.26.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.26.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.27.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.27.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.27.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.28.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.28.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.28.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.29.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.29.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.30.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.30.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.30.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.31.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.31.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.31.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.32.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.32.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.32.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.33.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.33.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.33.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.34.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.34.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.34.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.35.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.35.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.36.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.36.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.36.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.37.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.37.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.37.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.38.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.38.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.38.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.39.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.39.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.39.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.40.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.40.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.40.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.41.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.41.mlp.down_proj, value: 0.22 }
- { value: 0.0 }
- model: ./bgg1996/Melinoe-Gemma4-31B-VL-heretic
parameters:
density:
- { filter: norm, value: 1.0 }
- { filter: layer_scalar, value: 1.0 }
- { value: 0.80 }
weight:
- { filter: embed_tokens, value: 0.0 }
- { filter: lm_head, value: 0.0 }
- { filter: embed_vision, value: 0.0 }
- { filter: vision_tower, value: 0.0 }
- { filter: multi_modal_projector, value: 0.0 }
- { filter: model.language_model.layers.42.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.42.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.42.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.43.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.43.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.43.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.44.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.44.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.44.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.45.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.45.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.45.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.46.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.46.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.46.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.47.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.47.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.48.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.48.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.48.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.49.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.49.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.49.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.50.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.50.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.50.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.51.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.51.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.51.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.52.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.52.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.52.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.53.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.53.mlp.down_proj, value: 0.3 }
- { value: 0.0 }
III
The Growth
head graftTo this point, the merge would just ship the tied output head. This variant unties it and grafts the untied StyleTune lm_head on, at t = 0.85.
This should result in a darker lean, more potent NSFW and ERP conversations, and obvious to the method, less slop, trading for some slight instruction-following capabilities. I tried four variants, but if 0.85 is the right call here ultimately depends on your feedback.
▸ Phase 3: StyleSwap
python graft_head.py \
--body ./Split-31B \
--donor ./Gryphe/Pantheon-Reasoning-31B-1.1 \
--t 0.85 \
--out ./Split-Untied-31B \
--clip-q 0.999 \
--protect-chars em_dash
◆ Afterthought
You can't stump over scotoma-2. Checking benchmarks like caliperbench objectively, almost none of the merges beat their best parent when it comes to adjective and adverb density. So, instead of just blending scotoma-2 in again, I compared the delta of it's layers to the base model and gave it a weight of 1.0 over those. No idea if that survived contact with the actual prose on the long run, though.
Samplers and Thinking. Samplers are... yeah, they are. I give you a serving suggestion below, but go with whatever works for you. Thinking also works, but some people always report that thinking breaks for some merges, while it doesn't for others, so I still don't know if it's truly merge, front-end or even back-end related. So; just assume it's a no-think model if it breaks for you and we're good.
☰Samplers
Setting
Value
Temperature
1.0
Min-P
0.03
Top-K / Top-P
off / 1.0
Repetition penalty
off
DRY multiplier
0.8
◖◗Usage Like Always
◆ Thinking. Put <|think|> at the start of the system prompt, or let your front-end handle the Gemma 4 template. Reasoning gives better recall capability, but you might lose a bit on ERP pacing.
◆ Give it a scene. A sophisticated card will always get more nuanced output of this model than a few lines, just saying.
◖◗Thankus Maximus
- AesSedai / ReadyArt: scotoma-2 is just a banger, and you know it Aes.
- bgg1996: Melinoe-VL-heretic, doing two different jobs here on two different tensor families, cool thing.
- sophosympatheia: Glistening-Gem v2.0 and v2.1, kind of being the glue here.
- zerofata: MeroMero v2, and the mergekit fork obviously, your tunes going into anything should also speak for itself.
- Ateron: Dark-Thoughts V2. The rat guy that cursed me with the habit of merging. Thank you Mr. Rat, this was a good one.
- Gryphe: for the untied head.
- Charles Goddard/ArceeAI: For the mergekit and its continuation.
- The BeaverAI community: for the feedback that once again lead to actually shaping this one significantly.
- Google DeepMind & the open-source community: for the base model and everything that ended up inside this merge.
◖ · ◗