Model
How the bit-token model works
It is never told which relation it faces. It reads the 999 examples once, then guesses every bit of the answer and corrects the guess, round after round, by looking back at the examples. The same weights run every round.
This page shows the 8-bit version, which is small enough to inspect. The width-free model on the overview uses the same design with inputs that do not depend on the width, so the same weights answer 8- to 32-bit questions (939K parameters; src/teenie/bits/wide_model.py).
Architecture
1 · Read the examples once
Each example's 24 bits (a, b, c) go through a small encoder whose first layer is bilinear — two projections multiplied together — and become one vector. The 999 vectors stay fixed for every round: no reading order, no direction to choose.
2 · Hold a guess
The question starts with all 8 answer bits undecided. Its input layer is bilinear too, so products of its own bits such as ai·bj — the pieces of a multiplication — are one step away. The question is split into 8 tokens, one per answer bit, each knowing which bit it is.
3 · Look and correct
Each round, in each of 2 bit layers, the bit tokens first attend to each other — so carries and partial products can move between bit positions — then each reads the 999 example vectors, then a gated (SwiGLU) layer that can multiply signals updates them. A memory per token carries to the next round, and each token writes its own bit of the new guess.
4 · Answer honestly
After round 8 every bit is rounded to 0 or 1. Confidence is the certainty of the least certain bit: on relations it has never seen it seldom claims to be sure, and when it does it is right 99.5% of the time on hidden compositions.
Where the parameters are
| part | what it does | parameters | share |
|---|---|---|---|
| Example encoder | 24 bits → bilinear layer → GELU → linear → norm (run once) | 26,368 | 4.0% |
| Example keys and values | one projection pair per layer (2) | 66,048 | 9.9% |
| Question input | (a, b, current guess) → bilinear layer, plus 8 bit-position vectors | 10,624 | 1.6% |
| Bit layer 1 | bit tokens attend to each other (bias per bit distance), read the examples (4 heads), gated feed-forward | 231,868 | 34.8% |
| Bit layer 2 | bit tokens attend to each other (bias per bit distance), read the examples (4 heads), gated feed-forward | 231,868 | 34.8% |
| Memory | GRU cell per bit token, carried to the next round | 99,072 | 14.9% |
| Readout | each bit token → its own bit | 385 | 0.1% |
| Total | 666,233 | 100% |
Watch it answer
Recorded from the current model on fresh episodes. Play through the rounds: the guess starts undecided and sharpens as the question keeps looking back at the examples. add and max settle almost immediately; a*b takes longer (91.8% exact for this model);a*b+a and the hidden sum loop show where it still fails.
Relation (hidden from the model)
Question
The question
The guess, round by round
Rows are rounds, columns are answer bits (most significant on the left). The bluer, the surer the bit is 1; the paler, the surer it is 0. Undecided bits stay in between.
Where it looks
Each dot is one of the 999 examples (a → right, b → up), sized and shaded by the question's attention in the last look layer of this round (gray = no more than an even share). The ring is the question.
Most attended examples this round
Why one token per answer bit
The previous model held each question as one vector. It could multiply and it could add, but not one after the other: a product followed by another step needs carries to move between bit positions through an intermediate result. Trained on nothing but such relations, each design reached:
How it is trained
Never the same problem twice
Each step draws 8 fresh episodes: a random relation, 999 new examples and 16 new questions. Relations include the named ones, a*b, random expressions of 1–5 operations, loops whose repeat count comes fromb, and bit loops. Certain operation combinations and loop bodies are never generated, so they can be tested as truly unseen.
Effort where it is still wrong
Every 500 updates the model is checked on each relation kind, and the next episodes are drawn in proportion to how often it is still wrong. That alone took multiplication from 45% to 99.6% in an earlier mixed run. This model: 10,000 updates on mix4, per-bit cross-entropy, AdamW.
Source: src/teenie/bits/bit_token_model.py (--architecture bittoken). Results on the difficulty ladder: Results. How it got here, including the first bit-by-bit model: Changelog.