Start here
About 60 secondsThe idea, in a minute.
- The question
- Does increasing a learned product-key bank from 16 to 64 experts deliver repeatable quality gains with usable exposure?
- What we found
- The 32-domain experiment produced a repeatable mean-NLL benefit, but all E64 layers had zero routes at the 10th percentile and excessive route concentration. A balanced 64-domain test still failed semantic assignment and training-cost gates.
- Why it matters
- A larger nominal bank does not help if optimization leaves many experts underused or without distinct work.
These are tiny synthetic proxies with 123,712 active parameters; they do not validate frontier-scale MoE behavior or deployment.
A concrete example
Three-seed scale-validity result
This example comes from the completed E16/E64 confirmation on the checksum-frozen 384-case suite.
- 01
Matched comparison
Both models had 123,712 active parameters, trained 4,000 steps at learning rate 6e-4, with seeds 101, 202, and 303.
- 02
Quality
Mean NLL declined from 3.3423 at E16 to 3.2215 at E64, a 3.62% relative improvement. All three paired seeds improved.
- 03
Exposure
Every E64 layer failed registered concentration criteria: maximum route shares exceeded 0.15, and each layer had zero assignments at its 10th percentile.
The quality signal was reproducible, but expert-bank exposure remained insufficient to validate scaling.
Read the formal abstract
A larger expert bank can improve aggregate quality and still fail as a scaling result if the router concentrates its work. In a three-seed E16/E64 experiment with active parameters held at 123,712, E64 reduced mean NLL by 3.62%, and every paired seed improved. Yet all six E64 layers violated exposure limits: maximum route share exceeded 0.15, and the tenth percentile of assignments was zero. A second, deliberately balanced 64-domain experiment made demand much less sparse and increased training to 8,000 steps. It still missed the required 1% NLL gain, assigned dominant domain jobs to only 10 experts in one layer and 3 in the other, and took 3.62 times as long to train. The results justify investigating exposure mechanisms, not claiming that bigger banks reliably scale.
01 Research question
Does E64 provide a reproducible quality gain over active-parameter-matched E16, and do experts receive sufficiently distributed and distinct work to validate the bank increase?
02 Experiment setup
The full procedure, model settings, datasets, and evaluation choices are recorded here.
Explore the complete methodology
The primary scale-validity experiment compared tiny proxy models using learned product-key top-1 routing. E16 and E64 each applied exactly 123,712 active parameters per token. The frozen 384-case suite had SHA-256 6e852d4a66c3252f18328f243bb32c28fcbb2b13c0f6b343e8f971b6bfc1f8ab. All runs used seed 101, 202, or 303, 4,000 training steps, and learning rate 6e-4. Preregistered gates required at least a 1% mean-NLL improvement, stable per-seed direction, held-out quality and cost guards, and acceptable route exposure.
A separate demand64 screen increased the synthetic task to 64 uniformly sampled domains with 512 facts per domain, used 3,072 evaluation cases, and trained 8,000 steps (65,536,000 tokens) at learning rate 6e-4. It had only seed 101 because its screening failure stopped confirmation. Its E16/E64 arms again had 123,712 active parameters. The dataset adequacy audit passed. The semantic gate required every expert to own a distinct dominant domain job, sufficient training traffic, and at least 1% mean-NLL improvement.
03 What happened?
The 32-domain experiment produced a repeatable mean-NLL benefit, but all E64 layers had zero routes at the 10th percentile and excessive route concentration. A balanced 64-domain test still failed semantic assignment and training-cost gates.
Mean NLL · scale shown from 0 to 4
How to read thisE64 improved mean NLL from 3.342296504061244 to 3.2214714662327353 across paired seeds, with mean training time rising from 185.1448006203333 to 236.87874667266667 seconds. Exposure failures are described in the text and not represented as a passing E16 gate.
The limitTiny proxy only; all six E64 layers failed exposure criteria, and the separate demand64 screen used one seed and failed its primary gates.
Means across seeds 101/202/303 from scale-validity-result.json. Lower NLL is better; training time is seconds.
Full results and data table
| Bank size | Mean NLL | Mean training seconds |
|---|---|---|
| E16 | 3.342296504061244 | 185.1448006203333 |
| E64 | 3.2214714662327353 | 236.87874667266667 |
Across the three-seed scale-validity run, mean NLL was 3.3423 for E16 and 3.2215 for E64, an average relative improvement of 3.62%; all three E64 seeds improved. Mean teacher-forced token accuracy rose from 21.23% to 23.54%. Held-out-composition and episode NLL passed their guards, and E64 training time averaged 236.88 seconds versus 185.14 seconds for E16 (1.28x).
Exposure failed consistently. Across six E64 layers, experts-used counts ranged from 42/64 to 55/64, but every layer had zero assignments at the 10th percentile and a maximum route share above the allowed 0.15. The second layers in seeds 101 and 303 also fell below the 0.60 normalized-entropy floor. The aggregate decision was investigate, with expert-bank exposure as its sole failed gate.
In the 64-domain demand screen, the dataset audit passed. E64 mean NLL improved 0.47% (3.4343 versus 3.4506), below the 1% gate; token accuracy increased by 0.33 percentage points and exact cases increased by 14 of 3,072. Held-out composition regressed 0.53%, within its 5% guard. Answer routing used 62/64 experts in layer 0 and 48/64 in layer 1, but only 10 and 3 experts respectively owned a dominant domain job; one expert owned 20 and 54 jobs. Though all experts received some training traffic, the least-used expert received only 1.86% and 8.18% of the uniform mean across layers. E64 took 675.33 seconds versus 186.32 seconds for E16 (3.62x), over the 1.5x cost limit. The decision was fail; seeds 202/303 and E256 were not run.
04 What does this mean?
What this suggests
- For this proxy, the E64 quality gain replicated across seeds when active parameters and evaluation suite were matched.
- Route counts and semantic task assignments are necessary complements to aggregate quality when judging a larger bank.
What we still don’t know
- Whether an exposure intervention can correct concentration without sacrificing quality or training cost.
- How the observed proxy behavior transfers to natural language, larger models, or distributed inference.
Read the full discussion
The two experiments answer related but distinct questions. The three-seed study supports a repeatable quality improvement for a 64-expert bank in the smaller 32-domain setting, but does not validate the bank’s route distribution. The larger demand intervention tests whether inadequate or skewed demand was the main cause. Better balance and more examples improved E64 only modestly and did not distribute distinct domain jobs. “Every expert was touched during training” was not equivalent to balanced training or useful specialization.
Route exposure is therefore a bottleneck in the evidence chain. Aggregate NLL can improve while some experts receive few assignments, and broad count coverage can hide concentration among a small subset. Evaluation assignments, domain-job ownership, and the minimum training traffic surfaced failures that total bank size and average score could not.
The defensible next step from the three-seed study was one targeted exposure-mechanism experiment. The demand64 screen later showed this specific data-expansion intervention was not enough and its preregistered failure ended further seeds and E256. These results do not prove that all larger banks fail; they show that these configurations did not meet the registered criteria.
05 Limitations
These are tiny synthetic proxies with 123,712 active parameters; they do not validate frontier-scale MoE behavior or deployment.
Read all study limitations
The models are tiny synthetic-task proxies, and the 64-domain follow-up uses one seed. Synthetic domain assignments do not establish real-language expertise.
The 3,072-case demand64 screen is not a multi-seed confirmation. Its failure is a stopping result under stated thresholds, not a universal estimate of E64 quality.
Checkpoint metadata and full evaluation payloads are locally preserved for both experiments; the trained tensor weights were intentionally not retained. Metadata is provenance, not a model checkpoint.
06 References & materials
Keep the complete record
Take the research with you.
The full write-up, source code, and recorded experiment materials.
Download everything together or choose individual files.
All six scale-validity evaluation payloads and checkpoint metadata records exist; demand64 has seed-101 evaluations and metadata only. Metadata files are provenance evidence, not tensor weights, and are therefore listed with evidence rather than checkpoints. Trained tensor weights are not retained. Seeds 202/303 were not run for the failed demand64 screen.