Start here
About 60 secondsThe idea, in a minute.
- The question
- Can simple input transformations reveal that a small model has learned a rule well enough to stop?
- What we found
- Across 90 parity episodes, geometry stopping preserved the fixed baseline’s 71.1% success rate and similar accuracy while reducing counted work by 33.8%.
- Why it matters
- A training monitor can sometimes use known structure in the task instead of repeatedly consulting labels.
This result uses synthetic parity problems and a declared task-specific transformation; it does not establish a general stopping rule.
A concrete example
A four-bit parity episode
In a parity episode, flipping one relevant input bit should flip the predicted class; flipping an irrelevant distractor should leave it unchanged. The recorded study evaluated this relation during training.
- 01
Probe
At each 50-step checkpoint, evaluate a small batch and transformed copies, without using their labels.
- 02
Measure
Combine complement behavior, distractor invariance, and a confidence penalty into a residual.
- 03
Stop
Stop at the first residual at or below 0.20; the fixed comparison trains to 800 steps.
The transformation tests task behavior directly, but only because the relevant transformation is known.
Read the formal abstract
A model can fit a small validation set for the wrong reason. Could its responses to carefully chosen input changes provide a more direct stopping signal? In 90 synthetic parity episodes, an unlabeled counterfactual-geometry rule stopped at a mean of 453.3 optimizer steps, compared with 800 for fixed training. Success was 71.1% for both, and the paired mean test-accuracy difference was +0.00096 (95% bootstrap interval −0.00121 to +0.00324). After charging each observer example as if it were a training example, counted work fell by 33.8%. The result supports a task-aware monitor on this benchmark; it leaves open whether useful transformation rules can be identified reliably in broader settings.
01 Research question
For small parity networks, can an unlabeled counterfactual-geometry threshold stop training earlier than fixed or validation-based policies while preserving held-out performance?
02 Experiment setup
The full procedure, model settings, datasets, and evaluation choices are recorded here.
Explore the complete methodology
The experiment ran 90 episodes, 30 for each parity order 2, 3, and 4 over 24 input bits. A two-layer parity MLP had width 32 or 64; training used AdamW, minibatches of 64, and a maximum horizon of 800 optimizer steps. Episode-level choices varied training size (512, 1,024, or 2,048), learning rate (0.003, 0.01, or 0.03), and label noise (0 or 5%). The study code creates separate training, validation, probe, and test samples; the test set contains 8,192 examples per episode.
Every 50 steps, the geometry observer evaluated 64 probe inputs and their transformations. It measured whether flipping one relevant bit reversed the output, whether flipping sampled distractor bits preserved it, and whether the prediction was confident enough to avoid a constant guess. Training stopped when residual reached 0.20. Comparators were fixed 800 steps, validation loss at most 0.20, and validation accuracy at least 0.95. The validation rules used labels; the geometry probe did not.
The observer cost was counted conservatively as one training-example equivalent per forward example, although training examples also require backward computation.
The archived report files carry 2026-08-30 filesystem timestamps, but the result JSON does not record the exact execution date; that timestamp is artifact provenance, not a verified run date. To rerun the recorded default protocol from the repository root, use `python projects/geometric-meta-optimization/experiments/geometric_early_stopping.py --episodes-per-family 30 --input-bits 24 --horizon 800 --interval 50 --geometry-threshold 0.20 --output /tmp/geometric-early-stopping.json`. The driver requires Python with PyTorch and NumPy; this reruns rather than reads the frozen report.
03 What happened?
Across 90 parity episodes, geometry stopping preserved the fixed baseline’s 71.1% success rate and similar accuracy while reducing counted work by 33.8%.
Mean test accuracy (%) · scale shown from 0 to 100
How to read thisGeometry retained a 71.1% success rate across 90 episodes and reduced mean train-plus-observer example-equivalents from 51,200 to 33,899 versus fixed training.
The limitThe observer cost is a conservative example-equivalent count; the benchmark is parity only and the paired accuracy interval includes zero.
Episode means; accuracy is shown as percent and steps are optimizer updates. The 90% success threshold is separate from mean accuracy.
Full results and data table
| Policy | Mean test accuracy (%) | Mean optimizer steps |
|---|---|---|
| Fixed training | 86.6543 | 800 |
| Geometry | 86.7502 | 453.3 |
| Validation loss | 86.451 | 437.2 |
| Validation accuracy | 86.342 | 432.2 |
Fixed training averaged 0.86654 test accuracy, with 71.11% of episodes reaching 90% accuracy. Geometry stopping averaged 0.86750 accuracy and the same 71.11% success rate, using 453.3 steps on average. The paired accuracy difference was +0.00096, with a 95% percentile-bootstrap interval from −0.00121 to +0.00324.
The validation-loss and validation-accuracy policies averaged 437.2 and 432.2 steps, with accuracy 0.86451 and 0.86342; both also had 71.11% success. Thus geometry was not the earliest rule in raw optimizer steps. Its point is that it does not use labeled validation targets.
Geometry’s observer averaged 4,886 forward-only examples per episode. Total accounted work was 33,899 example-equivalents versus 51,200 for fixed training, a 33.8% reduction. This accounting treats an observer forward as equal to a training example, so it is conservative in favor of the fixed baseline.
04 What does this mean?
What this suggests
- For tasks with known valid and invalid transformations, behavioral consistency may serve as a low-cost training signal.
- On this parity benchmark, geometry did not improve success over fixed training but reduced training steps and accounted example-equivalents.
What we still don’t know
- Whether the rule transfers to realistic tasks, other architectures, or noisy transformations.
- How an observer’s forward-pass cost compares with training on modern accelerators.
Read the full discussion
The parity rule supplies unusually clear behavioral invariants: relevant-bit complement should reverse the answer, while distractor flips should not. The observer can therefore check whether learned behavior matches the task’s structure without asking for the correct label at every probe.
The study’s comparison with validation stopping is instructive: validation policies stopped somewhat earlier, but required labeled measurements. Geometry offers a different kind of signal rather than an unconditional speed advantage over every comparator. Accuracy uncertainty also means the small positive mean difference should not be read as a quality improvement.
05 Limitations
This result uses synthetic parity problems and a declared task-specific transformation; it does not establish a general stopping rule.
Read all study limitations
The evidence concerns synthetic parity with explicitly known relevant bits and transformations. A real task may not have a trusted transformation set, and incorrect invariants could stop training prematurely.
Only one 90-episode cohort is represented here. The bootstrap interval describes paired episode variability under this generated setup, not uncertainty across task families or hardware. Cost is measured in example-equivalents, not wall-clock time, energy, or accelerator utilization. No trained weights/checkpoints were found alongside the report; the per-checkpoint metrics are recorded in results.json, not model weights.
06 References & materials
Keep the complete record
Take the research with you.
The full write-up, source code, and recorded experiment materials.
Download everything together or choose individual files.
Source and full episode metrics are present. No weight/checkpoint files were found in the scoped project or evidence folders; results.json is observational output, not a checkpoint. Shared source dependencies are listed explicitly. Reproduction commands and dependency details are included in the study setup.