Changelog · first model
Watch it learn
Training never shows the same problem twice. Each step draws 8 new episodes: a random relation, 999 new examples and 16 new questions, so it is trained to find a procedure for any relation rather than memorise answers for one.
This page covers the first, bit-by-bit model, kept in thechangelog. The current model is on theModel page.
Exact answers during training
Measured every 100 updates on fixed validation episodes (128 questions per relation). Only the trained relations were used to pick the saved checkpoint; the never-seen ones are just watched.
Relation by relation
Bitwise relations (and, or, a<<1) click within the first 200 updates; carries and max follow by about 400. Among the never-seen relations, a+b+1 andnand transfer almost at once, 3a and a−2b creep up slowly, andmin never takes off. Random rules plateau below 100%. Hover a cell for the value.
Replay the checkpoints
The same question and the same 999 examples, answered by the model saved at each stage. At first attention is spread evenly and the answer is a blur of probabilities. As training goes on, attention snaps onto the examples in the question's situation and the bits become certain.
Relation
Where the first question looks at bit 3 (low-bit-first pass)
Training loss
Per-bit cross-entropy, averaged over each 100-update window.
Training recipe
Relations drawn during training
a+const and b^const draw a new constant each episode. transducer is a random 1–4 state rule reading one bit position at a time, in either direction — a nearly endless supply of new relations.
Never trained (held out)
Settings
| Updates | 3,000 |
| Episodes per update | 8 |
| Questions per episode | 16 |
| Optimizer | AdamW, one-cycle, peak 1e-3 |
| Loss | per-bit cross-entropy |
| Wall time | ~8 min, RTX 3050 laptop |
uv run teenie-bits --device cuda --updates 3000 \ --log-every 100 --save checkpoints/bits.pt \ --snapshots checkpoints/bits-snapshots \ --log experiments/bits-seed0-train.jsonl