Changelog · first model
How good is it, honestly?
Every method answers the same 1,024 questions per relation (64 fresh episodes × 16 questions, seed 60000, never used in training or checkpoint selection). A question counts only if all 8 bits are right.
This page covers the first, bit-by-bit model, kept in thechangelog. The current model is on theModel page.
- Teenie — the trained model, weights frozen, never told the relation.
- Closest example — copy
cfrom the example whose(a, b)differs in the fewest bits. - Best hand-written lookup — the bit-by-bit lookup from the Journey page, looking back 1 or 2 bits, in whichever direction does best for that relation. It is told the direction, so it's anupper reference for fixed rules, not a fair rival.
Relations it trained on
Fresh examples and questions, same kinds of relation as in training.
Relations it never saw
Close cousins of trained ideas, never sampled during training.
Stress test
Multiplication has no small bit-by-bit summary; nothing here solves it yet.
When it says it's sure
Confidence = the probability of the model's least certain bit. Above 0.99, its answers were not always correct. How often it gets that sure varies a lot: on max it is 100% exact but only rarely above the bar, because the 6% weight left on the wrong-direction pass softens every bit.
| relation | exact | bits right | sure (≥ 0.99) | exact when sure | closest example | lookup L1 / L2 / H1 / H2 |
|---|---|---|---|---|---|---|
add | 100% | 100% | 100% | 100% | 2.1% | 100% / 99% / 12% / 11% |
sub | 100% | 100% | 100% | 100% | 3.5% | 100% / 99% / 11% / 11% |
b-a | 100% | 100% | 100% | 100% | 3.5% | 100% / 99% / 10% / 10% |
xor | 100% | 100% | 100% | 100% | 4.7% | 100% / 100% / 100% / 100% |
and | 100% | 100% | 100% | 100% | 28.3% | 100% / 100% / 100% / 100% |
or | 100% | 100% | 100% | 100% | 29.9% | 100% / 100% / 100% / 100% |
a+2b | 100% | 100% | 100% | 100% | 3.9% | 34% / 95% / 9% / 7% |
2a+b | 100% | 100% | 99.9% | 100% | 5.0% | 36% / 95% / 8% / 8% |
max | 100% | 100% | 0.6% | 100% | 21.1% | 30% / 40% / 43% / 71% |
a<<1 | 100% | 100% | 100% | 100% | 33.4% | 100% / 100% / 1% / 1% |
a+const | 100% | 100% | 94.6% | 100% | 30.3% | 100% / 98% / 30% / 28% |
b^const | 100% | 100% | 100% | 100% | 26.9% | 100% / 100% / 100% / 100% |
transducer | 73.7% | 94.6% | 42.1% | 99.5% | 20.3% | 56% / 63% / 56% / 66% |
a+b+1 held out | 100% | 100% | 100% | 100% | 2.1% | 100% / 99% / 11% / 11% |
a-2b held out | 71.1% | 94.0% | 25.3% | 100% | 4.2% | 34% / 95% / 8% / 8% |
min held out | 32.2% | 80.1% | 0.8% | 100% | 21.9% | 30% / 41% / 43% / 72% |
nand held out | 100% | 100% | 100% | 100% | 28.3% | 100% / 100% / 100% / 100% |
3a held out | 75.8% | 94.0% | 13.9% | 100% | 28.0% | 18% / 96% / 5% / 5% |
a*b stress | 5.8% | 72.1% | 0% | — | 6.6% | 4% / 8% / 2% / 3% |
What these numbers do and don't show
Shows
- On the trained carry relations it is exact from the examples alone, where copying the closest example gets 2–5% (the hand-written bit lookup, told the reading direction, is exact there too).
- It picks the right reading direction by itself (low-first for carries, high-first for max).
- It generalises to some unseen relations (
a+b+1,nand: 100%).
Doesn't show (yet)
- One training seed; numbers are from a single run.
- New carry patterns (
3a,a−2b) and the mirror of a trained idea (min) are still unreliable. - Only 8-bit numbers, where 999 examples cover 1.5% of all inputs and lookup does well. Wider numbers and multiplication were untested or unsolved for this model.
Reproduce
uv run teenie-bits-eval --load checkpoints/bits.pt --device cuda \ --episodes 64 --queries 16 --seed 60000 \ --output experiments/bits-seed0-eval.json uv run python -m teenie.bits.export # refresh this site's data cd site && pnpm install && pnpm build