# Can a Model Stop Training When Its Behavior Has the Right Shape? A 90-episode parity study compares an unlabeled counterfactual-geometry stopping rule with fixed training and validation-based stopping. Author: yoyo Publication date: 2026-10-18 Version: 1.0 ## Takeaway Across 90 small parity tasks, an unlabeled geometry rule kept the success rate unchanged while cutting conservatively counted total work by 33.8% versus fixed training. ## The idea, in a minute ### The question Can simple input transformations reveal that a small model has learned a rule well enough to stop? ### What we found Across 90 parity episodes, geometry stopping preserved the fixed baseline’s 71.1% success rate and similar accuracy while reducing counted work by 33.8%. ### Why it matters A training monitor can sometimes use known structure in the task instead of repeatedly consulting labels. ### Keep in mind This result uses synthetic parity problems and a declared task-specific transformation; it does not establish a general stopping rule. ## A concrete example: A four-bit parity episode In a parity episode, flipping one relevant input bit should flip the predicted class; flipping an irrelevant distractor should leave it unchanged. The recorded study evaluated this relation during training. ### Probe At each 50-step checkpoint, evaluate a small batch and transformed copies, without using their labels. ### Measure Combine complement behavior, distractor invariance, and a confidence penalty into a residual. ### Stop Stop at the first residual at or below 0.20; the fixed comparison trains to 800 steps. The transformation tests task behavior directly, but only because the relevant transformation is known. ## Reading the main result 33.8% — accounted work reduction Geometry retained a 71.1% success rate across 90 episodes and reduced mean train-plus-observer example-equivalents from 51,200 to 33,899 versus fixed training. The observer cost is a conservative example-equivalent count; the benchmark is parity only and the paired accuracy interval includes zero. ## What this suggests - For tasks with known valid and invalid transformations, behavioral consistency may serve as a low-cost training signal. - On this parity benchmark, geometry did not improve success over fixed training but reduced training steps and accounted example-equivalents. ## What remains unknown - Whether the rule transfers to realistic tasks, other architectures, or noisy transformations. - How an observer’s forward-pass cost compares with training on modern accelerators. ## Abstract A model can fit a small validation set for the wrong reason. Could its responses to carefully chosen input changes provide a more direct stopping signal? In 90 synthetic parity episodes, an unlabeled counterfactual-geometry rule stopped at a mean of 453.3 optimizer steps, compared with 800 for fixed training. Success was 71.1% for both, and the paired mean test-accuracy difference was +0.00096 (95% bootstrap interval −0.00121 to +0.00324). After charging each observer example as if it were a training example, counted work fell by 33.8%. The result supports a task-aware monitor on this benchmark; it leaves open whether useful transformation rules can be identified reliably in broader settings. ## Research question For small parity networks, can an unlabeled counterfactual-geometry threshold stop training earlier than fixed or validation-based policies while preserving held-out performance? ## Experiment setup The experiment ran 90 episodes, 30 for each parity order 2, 3, and 4 over 24 input bits. A two-layer parity MLP had width 32 or 64; training used AdamW, minibatches of 64, and a maximum horizon of 800 optimizer steps. Episode-level choices varied training size (512, 1,024, or 2,048), learning rate (0.003, 0.01, or 0.03), and label noise (0 or 5%). The study code creates separate training, validation, probe, and test samples; the test set contains 8,192 examples per episode. Every 50 steps, the geometry observer evaluated 64 probe inputs and their transformations. It measured whether flipping one relevant bit reversed the output, whether flipping sampled distractor bits preserved it, and whether the prediction was confident enough to avoid a constant guess. Training stopped when residual reached 0.20. Comparators were fixed 800 steps, validation loss at most 0.20, and validation accuracy at least 0.95. The validation rules used labels; the geometry probe did not. The observer cost was counted conservatively as one training-example equivalent per forward example, although training examples also require backward computation. The archived report files carry 2026-08-30 filesystem timestamps, but the result JSON does not record the exact execution date; that timestamp is artifact provenance, not a verified run date. To rerun the recorded default protocol from the repository root, use `python projects/geometric-meta-optimization/experiments/geometric_early_stopping.py --episodes-per-family 30 --input-bits 24 --horizon 800 --interval 50 --geometry-threshold 0.20 --output /tmp/geometric-early-stopping.json`. The driver requires Python with PyTorch and NumPy; this reruns rather than reads the frozen report. ## Results Fixed training averaged 0.86654 test accuracy, with 71.11% of episodes reaching 90% accuracy. Geometry stopping averaged 0.86750 accuracy and the same 71.11% success rate, using 453.3 steps on average. The paired accuracy difference was +0.00096, with a 95% percentile-bootstrap interval from −0.00121 to +0.00324. The validation-loss and validation-accuracy policies averaged 437.2 and 432.2 steps, with accuracy 0.86451 and 0.86342; both also had 71.11% success. Thus geometry was not the earliest rule in raw optimizer steps. Its point is that it does not use labeled validation targets. Geometry’s observer averaged 4,886 forward-only examples per episode. Total accounted work was 33,899 example-equivalents versus 51,200 for fixed training, a 33.8% reduction. This accounting treats an observer forward as equal to a training example, so it is conservative in favor of the fixed baseline. ## Discussion The parity rule supplies unusually clear behavioral invariants: relevant-bit complement should reverse the answer, while distractor flips should not. The observer can therefore check whether learned behavior matches the task’s structure without asking for the correct label at every probe. The study’s comparison with validation stopping is instructive: validation policies stopped somewhat earlier, but required labeled measurements. Geometry offers a different kind of signal rather than an unconditional speed advantage over every comparator. Accuracy uncertainty also means the small positive mean difference should not be read as a quality improvement. ## Limitations The evidence concerns synthetic parity with explicitly known relevant bits and transformations. A real task may not have a trusted transformation set, and incorrect invariants could stop training prematurely. Only one 90-episode cohort is represented here. The bootstrap interval describes paired episode variability under this generated setup, not uncertainty across task families or hardware. Cost is measured in example-equivalents, not wall-clock time, energy, or accelerator utilization. No trained weights/checkpoints were found alongside the report; the per-checkpoint metrics are recorded in results.json, not model weights. ## Summary results | Policy | Mean test accuracy (%) | Mean optimizer steps | | --- | --- | --- | | Fixed training | 86.6543 | 800 | | Geometry | 86.7502 | 453.3 | | Validation loss | 86.451 | 437.2 | | Validation accuracy | 86.342 | 432.2 | Episode means; accuracy is shown as percent and steps are optimizer updates. The 90% success threshold is separate from mean accuracy. ## Available materials Source and full episode metrics are present. No weight/checkpoint files were found in the scoped project or evidence folders; results.json is observational output, not a checkpoint. Shared source dependencies are listed explicitly. Reproduction commands and dependency details are included in the study setup. The evidence archive preserves selected reports and result records. Source paths and SHA-256 hashes are listed in provenance.json. This publication rechecked recorded outputs and archive integrity; no research experiment was rerun. ## References - [Source reports and measured results](/downloads/geometric-early-stopping/evidence.zip)