Start here
About 60 secondsThe idea, in a minute.
- The question
- Does a synthetic MoE proxy rank sparse architectures in the same order as a larger-model evaluation?
- What we found
- No. It shows useful mechanistic signals and a natural-text direction, but medium proxy rankings diverge from the 86M evaluation.
- Why it matters
- A proxy can help diagnose mechanisms without being reliable for architecture selection.
The external comparison has four architectures and uses a distinct synthetic task versus a 60-case evaluation; the zero correlation is a warning, not a precise generalization estimate.
A concrete example
Conventional MoE versus dense
At small proxy scale, three paired seeds and a WikiText-2 anchor favored conventional MoE; the existing 86M task suite reversed that comparison.
- 01
Proxy evidence
Small-scale V3 mean NLL was 1.400 for conventional MoE and 1.455 for dense; WikiText-2 was 2.221 ± 0.021 versus 2.272 ± 0.035 bits per byte.
- 02
Larger-model check
On the 86M frozen suite, conventional MoE passed 30/60 cases while dense passed 34/60.
- 03
Implication
The proxy’s local advantage did not preserve the tested architecture ordering at 86M.
Positive mechanistic and natural-text checks do not by themselves establish rank validity.
Read the formal abstract
A proxy can be useful without being predictive. V3 was designed to isolate expert routing, capacity, and composition in a controlled synthetic task. Its routing controls behave meaningfully: oracle routing establishes headroom, and overriding a learned router degrades its checkpoint. Conventional and grouped sparse models also beat dense on the three-seed small-scale WikiText-2 anchor. But the proxy’s medium-scale architecture ordering did not match an existing 86M comparison. The four-architecture rank correlation was zero; notably, the proxy expected every sparse variant to beat dense, while at 86M only grouped-4 beat dense in fully correct cases. The evidence supports V3 as a mechanistic diagnostic, not a validated architecture selector.
01 Research question
Does the V3 MoE architecture proxy preserve architecture rankings across synthetic evaluation, a natural-text anchor, and an existing 86M task-suite comparison?
02 Experiment setup
The full procedure, model settings, datasets, and evaluation choices are recorded here.
Explore the complete methodology
V3 uses answer-only loss over atomic answer symbols, with a frozen six-category suite covering common and rare capacity recall, domain rules, shared control, trained compositions, and held-out ordered compositions. The validation used tiny, small, and medium active-parameter scales (about 123,712; 4,824,320; and 25,381,376). Tiny and small comparisons used seeds 101, 202, and 303; medium was a single seed-101 screen. Architectures included dense, conventional MoE, grouped-2, and grouped-4 where applicable. The primary report preserves a Pareto profile rather than collapsing these outcomes into one score.
The natural-text anchor scored the first 1,000,000 bytes of checksum-pinned WikiText-2 raw validation text, at shared learning rate 3e-4 for 4,000 steps and three paired seeds. A separate existing 86M frozen evaluation suite supplied the larger-model comparison. These evaluations are not identical tasks: one is natural text and one is a 60-case task suite. The rank comparison is therefore an external validity check, not a matched benchmark.
03 What happened?
No. It shows useful mechanistic signals and a natural-text direction, but medium proxy rankings diverge from the 86M evaluation.
Proxy mean NLL · scale shown from 0 to 2
How to read thisThe proxy orders grouped-2, grouped-4, conventional, dense by mean NLL; 86M full-correct counts order grouped-4, dense, grouped-2, conventional.
The limitOnly four architectures; the metrics and tasks differ, so this is a limited rank-validation check rather than a universal proxy estimate.
Proxy values are medium seed-101 overall NLL (lower is better); 86M values are fully correct cases out of 60 (higher is better). These are different metrics and are shown as recorded, without rescaling.
Full results and data table
| Architecture | Proxy mean NLL | 86M fully correct cases |
|---|---|---|
| Dense | 1.5728797408049406 | 34 |
| Conventional MoE | 1.4861577834012678 | 30 |
| Grouped-2 | 1.4628125817665325 | 31 |
| Grouped-4 | 1.4673615612859219 | 36 |
At small proxy scale, three-seed mean NLL was 1.455 for dense, 1.400 for conventional MoE, and 1.428 for grouped-2. Conventional MoE beat dense in every paired seed. At medium scale, single-seed means were 1.573 dense, 1.486 conventional MoE, 1.463 grouped-2, and 1.467 grouped-4. Sparse models paid a held-out-composition cost: medium held-out NLLs were 4.292 dense, 4.336 conventional, 4.550 grouped-2, and 4.573 grouped-4.
On WikiText-2, bits/byte (lower is better) averaged 2.272 ± 0.035 for dense, 2.221 ± 0.021 for conventional MoE, and 2.241 ± 0.035 for grouped-2. Both sparse arms beat dense in each of three seeds. Grouped-4 was not part of this natural-text comparison.
The existing 86M suite had grouped-4 at 36/60 fully correct, dense at 34/60, grouped-2 at 31/60, and conventional MoE at 30/60. The medium proxy ranked grouped-2, grouped-4, conventional, dense by mean NLL. The rank correlation between these four orderings was zero. Thus it identified grouped sharing as a promising candidate but incorrectly predicted that each sparse architecture would outperform dense. No architecture met the study’s conservative promotion gate.
04 What does this mean?
What this suggests
- V3 can diagnose whether routing is causal in its own synthetic task and can expose seen versus held-out composition behavior.
- The natural-text anchor supports some sparse-versus-dense direction at small scale.
What we still don’t know
- Whether additional scales, architectures, and natural-language/knowledge tasks could yield useful predictive rank correlations.
- Whether the rank mismatch is driven by task differences, scale, training regimes, or a combination.
Read the full discussion
The validation distinguishes mechanistic validity from rank validity. V3 showed that routing can causally affect its proxy task: oracle routing had headroom, random and single-expert controls were worse, and forced route overrides damaged a learned checkpoint. These results justify using the proxy to diagnose routing and memorization within its own design. They do not show it predicts which architecture wins elsewhere.
The WikiText result offers a positive external direction for conventional and grouped-2 MoE at one small scale under a fixed text budget. It is not enough to rescue the rank claim because the natural-text evaluation did not include grouped-4, spans one scale, and differs from the 86M suite. The zero rank correlation among four architecture points is limited evidence, but directly contradicts treating the observed medium proxy order as a reliable predictor.
The practical classification is “mechanistic diagnostic,” not “rank-order predictor.” A future validation should add grouped-4 text results, multi-seed medium comparisons, sealed natural-language and knowledge tasks, and multiple larger model sizes before fitting or trusting architecture-specific rankings. Communication, memory, and throughput also require separate measurement.
05 Limitations
The external comparison has four architectures and uses a distinct synthetic task versus a 60-case evaluation; the zero correlation is a warning, not a precise generalization estimate.
Read all study limitations
The 86M comparison includes only four architectures and comes from a different 60-case suite. A zero rank correlation over four items is a clear mismatch in this sample but is statistically coarse.
Medium proxy results are seed-101 screens, whereas tiny/small and WikiText results use three paired seeds. Held-out composition regressions show that aggregate proxy wins can reflect capacity or seen-composition fit rather than transfer.
The V3 validation artifacts preserve configurations and evaluation JSON, but no trained model tensor checkpoints were found in the evidence directory. The E36 unrelated multi-gigabyte checkpoint should not be treated as part of this study.
06 References & materials
Keep the complete record
Take the research with you.
The full write-up, source code, and recorded experiment materials.
Download everything together or choose individual files.
Rerun dependencies include all shared model, proxy, scoring, evaluation, and metrics packages plus configs, generation/fetch scripts, and frozen suites. The validation bundle preserves analysis and run evaluations, not model tensor checkpoints; no model weights were found in its evidence directory. Separate 86M artifact directories provide the dense baseline and sparse comparison evidence. Evaluation artifacts are not trained weights.