Start here
About 60 secondsThe idea, in a minute.
- The question
- Can repeating exploration and group consolidation avoid the early commitment problem of one-shot parameter averaging?
- What we found
- Cyclic consensus greatly outperformed independent and one-shot controls through 20 dimensions on this synthetic objective, then degraded sharply at 30D.
- Why it matters
- The experiment isolates how a population can preserve its best discovery while continuing to search nearby basins.
Rastrigin is symmetric and separable. The result does not show that direct parameter consensus works for neural networks.
A concrete example
A 10-dimensional confirmation group
The frozen confirmation aggregates three seeds, each with 5,000 paired groups, for 15,000 total groups at dimension 10.
- 01
Explore
Run five replicas with private gradients; four followers receive decaying perturbations around the archived best point.
- 02
Consolidate
After each short exploration phase, select the best current or archived point and contract replicas toward it.
- 03
Repeat
Run 120 cycles for 1,200 gradients per replica, retaining the elite so its objective value cannot regress.
The cyclic method succeeded in 14,781 groups; independent runs succeeded in none, and one-shot soft-best succeeded once.
Read the formal abstract
Averaging a population once can erase the exploration that made its members useful. This Rastrigin study tests a repeated alternative: preserve the best point found so far, perturb followers around it, let each replica follow its own gradient, and reconsolidate the group. Across three frozen seeds and 15,000 groups per dimension, cyclic consensus reached exact-global success in 98.54% of 10D groups, compared with zero for five independent runs and one success for one-shot soft-best consensus. It remained at 53.93% in 20D but fell to 3.50% in 30D. The evidence supports repeated explore-and-consolidate behavior on this synthetic surface, while the sharp high-dimensional decline and the distance from neural-network evidence remain central limits.
01 Research question
With equal gradient budgets, does repeated exploration and elite-preserving consensus improve exact-global success over five independent trajectories or a one-shot soft-best consensus on Rastrigin?
02 Experiment setup
The full procedure, model settings, datasets, and evaluation choices are recorded here.
Explore the complete methodology
The experiment minimizes the separable Rastrigin function in dimensions 2, 5, 10, 20, and 30. Each group contains five replicas with paired initial points across methods. Exact-global success is final loss at most 1e-6. The cyclic method runs 120 cycles and 1,200 gradient steps per replica using learning rate 0.002. In each cycle, followers begin near the best archived point with Gaussian perturbations decaying from 0.35 to 60% of that value; replicas take private gradients before the best current or archived point becomes leader. Two contraction steps then merge replicas. The elite archive prevents loss of the best objective value.
Confirmation used seeds 91001, 91002, and 91003, each with 5,000 groups per dimension (15,000 paired groups per row). Comparators included five independent runs, one-shot soft-best consensus, and single gradient descent. The one-shot method and cyclic method both use five replicas; single-run is an additional reference. Execution settings recorded in the result report are Python 3.12.13, PyTorch 2.13.0, CPU float64.
The confirmation is identified as the 2026-09-01 Rastrigin cyclic-consensus run in its artifact directory and report. Reproduction from the repository root is `python projects/rastrigin-consensus/cyclic_consensus.py --dimensions 2 5 10 20 30 --seeds 91001 91002 91003 --trials-per-seed 5000 --device cpu --output /tmp/rastrigin-cyclic-consensus.json`. The recorded environment was Python 3.12.13 and PyTorch 2.13.0 on CPU float64; the driver requires PyTorch. The output path must not already exist.
03 What happened?
Cyclic consensus greatly outperformed independent and one-shot controls through 20 dimensions on this synthetic objective, then degraded sharply at 30D.
Cyclic success (%) · scale shown from 0 to 100
How to read thisCyclic consensus solved 14,781 of 15,000 groups at 10D, while five independent runs solved none and one-shot consensus solved one.
The limitPerformance drops to 53.93% at 20D and 3.50% at 30D; this remains evidence on a symmetric, separable synthetic function.
Percentage of 15,000 groups (5,000 per seed) reaching Rastrigin loss ≤ 1e-6; each cyclic replica receives 1,200 gradient steps.
Full results and data table
| Dimension | Cyclic success (%) | Groups |
|---|---|---|
| 2D | 100 | 15000 |
| 5D | 99.967 | 15000 |
| 10D | 98.54 | 15000 |
| 20D | 53.933 | 15000 |
| 30D | 3.5 | 15000 |
Cyclic success was 100.000% at 2D, 99.967% at 5D, 98.540% at 10D, 53.933% at 20D, and 3.500% at 30D. The 10D result was stable across the three held-out seeds: 98.28%, 98.64%, and 98.70%. Aggregated across 15,000 groups at 10D, cyclic consensus succeeded 14,781 times; five independent runs succeeded zero times and one-shot soft-best consensus succeeded once. All 14,781 cyclic successes were rescues of groups where no independent replica succeeded, and no independent success was lost.
At 10D, mean final loss was 0.0153 for cyclic consensus, versus 58.91 for the best-of-five independent comparator and 16.15 for one-shot consensus. Cyclic groups used exactly 1,200 gradients per replica; all returned replicas were bit-identical, and elite-loss monotonicity violations were zero.
The dimensional guard matters: cyclic success was still 53.93% at 20D, but only 3.50% at 30D. This is a steep decline, not a dimension-free optimizer improvement.
04 What does this mean?
What this suggests
- Repeated communication between trajectories can rescue groups whose independent replicas all miss the global basin on this objective.
- The mechanism has a clear dimensional boundary: exact success drops after 20D and nearly disappears at 30D.
What we still don’t know
- Whether repeated parameter-space consensus is useful on nonseparable objectives or neural networks.
- How the method compares under wall-clock or memory constraints beyond equal gradient counts.
Read the full discussion
The contrast between one-shot and cyclic consensus is consistent with a repeated-search explanation. A single consolidation can commit to a local basin early. In the cyclic schedule, an archived elite cannot be lost, followers probe nearby alternatives, and the group reconsolidates around improvements. Equal gradient counts make this a comparison of trajectory organization rather than extra gradient budget.
The result justified moving to a bounded neural-network proxy in the experiment card, not claiming neural-network gains. In neural networks, nearby weights may represent different functions because of hidden-unit permutations and other symmetries; a low-loss leader is also harder to identify cheaply. The next step requires function-space validation before accepting parameter interpolation.
05 Limitations
Rastrigin is symmetric and separable. The result does not show that direct parameter consensus works for neural networks.
Read all study limitations
Rastrigin is separable, symmetric, deterministic, and has a centered global optimum. The easy-to-score landscape makes it an intentionally narrow mechanism test. The success rate collapses at 30D.
Trials are grouped and paired by initializations, so percentages summarize this frozen synthetic setup rather than independent real-world tasks. The experiment has no neural network, learned objective, wall-clock comparison, or energy accounting. No checkpoints are needed for this deterministic optimizer; the available artifacts are source code, results JSON, and run documentation, not trained weights.
06 References & materials
Keep the complete record
Take the research with you.
The full write-up, source code, and recorded experiment materials.
Download everything together or choose individual files.
Deterministic synthetic optimization has no learned checkpoint. Included are the candidate implementation, its base objective/comparator implementation, focused tests, confirmation report, experiment card, and actual result JSON. These are real source and evidence artifacts. Reproduction commands and dependency details are included in the study setup.