Why an exact gradient trick failed to replace real training data

Start here

About 60 seconds

The idea, in a minute.

The question: Can a trained model provide synthetic inputs that teach a fresh model faster than ordinary data?
What we found: Frozen synthetic embeddings beat Gaussian noise, but real TinyStories updates remained substantially better at matched processed positions.
Why it matters: An exact algebraic result at one layer can hide a difficult reconstruction problem across an entire network.

Keep in mind

Two held-out student seeds in the controlled pilot; synthetic embeddings are not ordinary text and do not establish foundation-model speedups.

See the evidence

A concrete example

Accuracy vanishes as the inverse reaches the input

Recorded known-realizable-gradient diagnostic, separate from student-transfer endpoints.

01
Inject at the output head
The head-gradient cosine reaches 0.996 using a controlled adjoint.
02
Require valid raw inputs
Valid soft targets at raw embeddings yield full-gradient cosine about 0.104.
03
Test actual learning
The controlled student loss remains worse for synthetic embeddings than real data.

A local reconstruction check is necessary but not enough.

Archived experiment reports and measurements

Read the formal abstract

If a trained model knows where learning should go, perhaps a new model can skip some rediscovery. We tested a direct, low-rank inversion from desired weight updates to synthetic embedding and output-target pairs, without optimizing those synthetic inputs in an outer loop.

The linear-layer reconstruction passed, and the output-head gradient could be reconstructed with cosine 0.996 at direct injection. But valid soft targets at raw embeddings reproduced known-realizable full gradients with cosine only about 0.10. In the corrected, fixed-validation pilot, twenty matched updates gave held-out losses 3.884 for real data, 4.793 for frozen synthetic embeddings, and 5.584 for Gaussian inputs. The promising algebra did not survive the transformer reconstruction bottleneck well enough to replace ordinary training.

01 Research question

Can a trained model provide synthetic inputs that teach a fresh model faster than ordinary data?

02 Experiment setup

The full procedure, model settings, datasets, and evaluation choices are recorded here.

Explore the complete methodology

The controlled pilot uses TinyStories with a shared 256-token byte vocabulary, four causal transformer blocks, width 144, four attention heads, and MLP width 576. The saved controlled-run summary records 1,078,560 parameters; this actual artifact supersedes a different default parameter count in the general README.

Stage A verifies exact gradient factorization for a linear map. Later stages use low-rank factors of oracle directions to construct activation/adjoint pairs, then map them toward valid synthetic inputs and probability targets. Synthetic construction uses seeds 101–102; held-out student seeds 201–202 do not participate in construction or selection.

The corrected comparison uses one fixed 512-sequence validation set. Every update processes sixteen sequences of forty-eight positions. Twenty updates therefore process 15,360 positions per condition. Conditions are ordinary TinyStories, frozen constructed embeddings, and Gaussian-embedding controls. Corrected output-target naming distinguishes soft/hard targets from input tokens.

The oracle trains to step 300, with checkpoints at 0, 50, 150, and 300. Complementary known-realizable-gradient diagnostics use three separate students 301–303 and measure reconstruction at multiple injection depths. All controlled-pilot checkpoint and synthetic-dataset files are included alongside the diagnostic CSVs.

03 What happened?

Frozen synthetic embeddings beat Gaussian noise, but real TinyStories updates remained substantially better at matched processed positions.

Figure 01 / Observed resultsReal data remains the stronger teacher

4.793 vs 3.884Synthetic versus real-data validation loss

630

3.884

Real data

4.793

Synthetic

5.584

Gaussian

Validation cross-entropy ↓ · scale shown from 0 to 6

How to read thisSynthetic embeddings help compared with noise but remain worse than ordinary data. The exact linear-layer result does not imply an exact network-level inverse.

The limitTwo held-out student seeds in the controlled pilot; synthetic embeddings are not ordinary text and do not establish foundation-model speedups.

Controlled rerun: two held-out seeds, one fixed 512-sequence validation set, and 15,360 processed positions per twenty-update arm.

Full results and data table

Complete summary results
Training input	Validation cross-entropy ↓	Matched updates
Real data	3.884	20
Synthetic	4.793	20
Gaussian	5.584	20

Stage A cosine was 1.0 with relative error 3.95e-7. The oracle reached fixed-validation cross-entropy 2.1696 at step 300. These successful checks do not establish student transfer.

At twenty matched updates, controlled held-out validation cross-entropy averaged 3.884 for real data, 4.793 for frozen synthetic embeddings, and 5.584 for Gaussian controls. Lower is better. Earlier pilot values used a different comparison and are preserved as historical rather than pooled into this endpoint.

For known-realizable gradients, controlled head injection achieved head cosine 0.996, but full-gradient cosine only 0.353. Before final LayerNorm these were 0.910/0.322. At raw embeddings with valid soft targets they fell to 0.194/0.104. Oracle-direction raw-embedding full cosine was lower still, about 0.016.

Trajectory-span diagnostics separately show that short actual optimizer trajectories are more aligned with ordinary-gradient spans than a raw endpoint-weight direction. This helps reject the idea that a weight difference should automatically be treated as an ideal gradient; it does not repair the synthetic-input reconstruction.

04 What does this mean?

What this suggests

Frozen synthetic embeddings beat Gaussian noise, but real TinyStories updates remained substantially better at matched processed positions.
The failure is not explained solely by choosing an impossible desired gradient. Even a genuine language-model gradient became poorly reconstructed when forced through valid raw inputs and output probabilities.

What we still don’t know

Two held-out students provide a pilot, not a statistically powered speedup claim. Byte-level TinyStories models are far from pretrained foundation models.
Processed positions are matched, but synthetic construction has additional costs and the embeddings do not correspond to ordinary printable training text.

Read the full discussion

The failure is not explained solely by choosing an impossible desired gradient. Even a genuine language-model gradient became poorly reconstructed when forced through valid raw inputs and output probabilities.

Useful synthetic transfer must overcome activation realizability, target constraints, and cross-layer distortion. Beating Gaussian noise is a weak baseline; the relevant efficiency comparison is ordinary data under an explicit compute and exposure budget.

05 Limitations

Two held-out student seeds in the controlled pilot; synthetic embeddings are not ordinary text and do not establish foundation-model speedups.

Read all study limitations

Two held-out students provide a pilot, not a statistically powered speedup claim. Byte-level TinyStories models are far from pretrained foundation models.

Processed positions are matched, but synthetic construction has additional costs and the embeddings do not correspond to ordinary printable training text.