Start here
About 60 secondsThe idea, in a minute.
- The question
- Does a larger continual-learning model benefit more from limiting its total confidence?
- What we found
- The 20× certainty cap improved recent retention by 9.97 points in the narrow model and just 0.22 points in the wide model.
- Why it matters
- A larger model can remember more overall while receiving less benefit from a particular forgetting-control rule.
One paired seed over 100 Permuted-MNIST tasks; the two widths spent different amounts of training under the active cap.
A concrete example
The wider model remembers more, but gains less from the cap
Recorded 100-task terminal evaluations.
- 01
Narrow matched comparison
The cap raises recent-five retention from 79.00% to 88.97%.
- 02
Wide matched comparison
The cap raises it from 92.81% to 93.03%.
- 03
Look at the longer horizon
The wide capped model retains 53.25% across all tasks versus 30.96% for the narrow capped model.
Overall capacity and the benefit of a specific intervention are distinct.
Archived experiment reports and measurementsRead the formal abstract
When a Bayesian learner becomes increasingly certain, new learning can become difficult. We tested a simple constraint: allow total precision to grow to twenty times its initial value, then keep that total fixed while preserving its relative distribution.
Across 100 Permuted-MNIST tasks, the cap improved recent-five retention versus its matching no-forgetting control by 9.97 percentage points at width fifty but only 0.22 points at width 1,024. The wider constrained model remembered more tasks overall, yet the capacity-by-treatment interaction was negative. This single-seed pilot separates capacity benefits from certainty-budget benefits and exposes the importance of the chosen memory horizon.
01 Research question
Does a larger continual-learning model benefit more from limiting its total confidence?
02 Experiment setup
The full procedure, model settings, datasets, and evaluation choices are recorded here.
Explore the complete methodology
Executed September 20, 2026 on DGX Spark GB10. Source is pinned to upstream commit 4aafc77e2a53aadad5b96de2bed426e76db79b69. Widths fifty and 1,024 have 39,700 and 813,056 Bayesian weights. Both use no biases, ReLU, float32, batch one, ten Monte Carlo samples, one full 60,000-example epoch per task.
Each width compares ordinary MESU with N=300,000, effectively no-forgetting MESU with N=10^15, and N=10^15 with a global 20-times initial-precision cap. Once the cap activates, every update projects total precision back to the same budget, even if a proposal would reduce the total. The mean-update rule and relative precision ratios remain unchanged by projection.
A shared 200-task permutation and evaluation-key stream is generated before training. Within each width, all arms start identically and share the exact pre-activation trajectory. Model shapes necessarily change parameter-sized random draws across widths. No task boundary enters the learning rule.
The frozen endpoint is the first ten-task checkpoint at or above thirty allowing ten task-equivalents after wide-model activation, capped at 200. Both widths consequently finish at 100 tasks. Recent accuracy uses the same 1,000-image subset during training; terminal retention uses all 10,000 test images on every seen task. Final model states, input permutations, raw metrics, code, and pinned reference source are included.
03 What happened?
The 20× certainty cap improved recent retention by 9.97 points in the narrow model and just 0.22 points in the wide model.
Recent-five retention (%) ↑ · scale shown from 0 to 100
How to read thisThe cap improved recent retention substantially at width fifty and barely at width 1,024. All-task retention shows the associated tradeoff with older memory.
The limitOne paired seed over 100 Permuted-MNIST tasks; the two widths spent different amounts of training under the active cap.
One paired seed, common 100-task endpoint; control is MESU N=10^15. Values are full-test accuracy percentages.
Full results and data table
| Width / arm | Recent-five retention (%) ↑ | All-task retention (%) ↑ |
|---|---|---|
| 50 / control | 79 | 47.6 |
| 50 / cap | 88.97 | 30.96 |
| 1024 / control | 92.81 | 55.17 |
| 1024 / cap | 93.03 | 53.25 |
At width fifty, final-five full-test retention was ordinary 91.79%, no-forgetting 79.00%, and capped 88.97%. At width 1,024 it was 88.07%, 92.81%, and 93.03%. Treatment benefit versus its matched no-forgetting control was therefore +9.97 and +0.22 percentage points; the declared interaction was -9.75 points.
All-task retention gives a different view. Width-fifty ordinary/no-forgetting/capped means were 21.55%/47.60%/30.96%; width-1,024 means were 16.52%/55.17%/53.25%. The cap traded distant memory for recent learning at both widths. Higher all-task retention at the larger width is a capacity effect, not evidence of a stronger cap benefit.
The narrow cap activated at task-equivalent 18.56575 and the wide cap at 83.70345. Post-activation exposure was about 81.43 versus 16.30 tasks, despite the common total horizon. Maximum active relative budget error was 3.58e-7 at each width.
Earlier thirty-task width-fifty evidence found a +1.10-point recent-retention benefit over the N=10^15 control but a -3.27-point difference versus ordinary MESU. That earlier horizon is not pooled with the 100-task result. Finite-value, inactive-identity, threshold, persistent-latch, and precision-invariant checks passed in the archived execution.
04 What does this mean?
What this suggests
- The 20× certainty cap improved recent retention by 9.97 points in the narrow model and just 0.22 points in the wide model.
- Capacity and treatment effects must be separated. A wider model can improve overall retention without making a total-certainty cap more useful relative to the correct within-width control.
What we still don’t know
- One paired seed does not establish statistical significance, an optimal budget, or generality beyond Permuted MNIST.
- Different cap-activation times confound equal active-exposure interpretation; equal total training is not equal time under treatment. This is not a full reproduction of the MESU paper.
Read the full discussion
Capacity and treatment effects must be separated. A wider model can improve overall retention without making a total-certainty cap more useful relative to the correct within-width control.
Recent-five and all-task retention answer different questions. The apparent desirability of forgetting changes when the evaluation rewards recent adaptation rather than distant history.
05 Limitations
One paired seed over 100 Permuted-MNIST tasks; the two widths spent different amounts of training under the active cap.
Read all study limitations
One paired seed does not establish statistical significance, an optimal budget, or generality beyond Permuted MNIST.
Different cap-activation times confound equal active-exposure interpretation; equal total training is not equal time under treatment. This is not a full reproduction of the MESU paper.
06 References & materials
Keep the complete record
Take the research with you.
The full write-up, source code, and recorded experiment materials.
Download everything together or choose individual files.
The complete archive has 2 parts. Download every part into the same folder, then open complete-study.zip with an app that supports split ZIP archives. The article, source archive, and individual checkpoints can also be downloaded separately.