teenie

Changelog · first model

The first model: bit by bit

Kept for the record. This model read every example one bit position at a time — lowest bit first in one pass, highest first in another. It was exact on the named relations it trained on, but could not handle multiplication or decide which direction to read for min; the current model replaced it. Its training and benchmark pages:how it learned ·benchmarks.

The first learned model (447K parameters, superseded) walked through the bit positions, remembered a little about the bits so far, and at each bit copied the answer from the examples in the same situation as the question. It is still the easiest to look inside, so the replay below uses it.

1 · Loop over bits

A small recurrent network (GRU) reads (ak, bk, ck) one position at a time, for every example and for the question. The same weights run at every position.

2 · Remember what matters

Its memory has to hold just enough about the bits already read — a carry for add, a borrow forsub, “a is already bigger” for max. Nobody tells it which; training does.

3 · Look up similar situations

At bit k, the question attends over the 999 examples. Examples whose current bits and memory match get the attention; their known ck becomes the vote. The question's own guess feeds back into its memory for the next bit.

4 · Check both directions

One pass reads from the lowest bit, one from the highest. Each re-predicts 64 of the examples with that example hidden from itself. The pass that explains them better gets more say in the final answer.

One bit step

This repeats 8 times per pass. During training the question's earlier bits are the true ones; when answering, they are its own predictions. Weights never change when answering.

One bit step of a passThe question and every example describe their situation at bit k. Attention matches the question to examples in the same situation and reads their c_k. The readout gives the probability that the question's c_k is 1. A GRU step then folds the bits into each memory and the loop moves to the next bit.askskeysvaluesround to 0/1next bitQuestion's memorybits read so far999 examples' memoriesbits read so farExamples' bit c_ktheir known answersdescribememory + a_k, b_kdescribememory + a_k, b_kattentionwhich examples are inthe same situation?readoutp(c_k = 1)GRU step: fold (a_k, b_k, c_k) into each memory, then move to the next bit

Inside real predictions

These are recordings from the trained model on fresh episodes. Pick a relation and step through the bits. Onadd, low-bit-first, the attention gathers in the shaded intersection — examples whose bitk of a and b match the question — and, within it, only those with the same carry. Try min or a*b to see it fail.

Relation

Question

Pass

The question

Where it looks at bit 0

Each dot is one of the 999 examples (a → right, b → up). Gray = no more than an even share of attention; bigger and bluer = more (log scale, up to 256× an even share). Shaded bands: examples whose bit k of a or of b matches the question. The ring is the question.

Most attended examples

Which direction does it trust?

Average weight given to the high-bit-first pass over 16 episodes per relation. It picks low-first for carries, high-first for max, and doesn't care for bitwise relations. On the held-out min it hesitates — which is exactly where it fails. * never seen in training.

← low bit firsthigh bit first →addsuba+2ba<<1xorandmaxtransducera+b+1 *3a *a-2b *nand *min *a*b

Size

Parameters447,491
Memory (GRU hidden size)128
Attention heads4
Self-check examples per pass64
Examples per question999 (fixed)
Bit width trained / supported8 / up to 64

Source: src/teenie/bits/model.py.