Changelog

From fuzzy decimals to exact bits

The goal never changed: given exactly 999 examples of an unknown relation c = f(a, b) and one new(a, b), produce c — with no operation name, no menu of formulas, and no retraining at answer time. What changed was how numbers are shown to the model, and what counts as right.

1. Attempt 1 · float32 · within 1% counts as right

   A small recurrent network on decimal numbers

   Inputs and outputs were float32 numbers. A recurrent network read the 999 examples and had to output c, counted correct within 1% relative error. After 1,000 training updates it was essentially never right.

   add	1.2%
   sub	1.0%
   mul	0%
   div	0%

   Fails · experiments/fixed999-seed0-eval.json

2. Attempt 2 · float32 · within 1% counts as right

   Curriculum, memory slots, more training

   Small numbers first, several memory slots, 3,000 updates and 133K parameters. Better, but just copying the c of the closest example still did better on every basic relation.

   add	30.9% (closest example 42.6%)
   sub	11.7% (14.1%)
   mul	16.4% (19.5%)
   div	1.6% (16.4%)

   Below the closest-example baseline · experiments/curriculum-long-m4-eval.json

3. Attempt 3 · float32 · within 1% counts as right

   A built-in curve fitter (residual loop)

   Replaced the network with a fixed curve-fitting loop (conjugate gradient) over a hand-chosen set of features, with 61 trainable parameters. It passed 90% on all four operations — but an untrained copy scored the same. The fitter did the work, not learning. And “within 1%” is still a guess, not an answer.

   add	99.7% (untrained 99.7%)
   sub	99.2% (99.4%)
   mul	94.5% (94.7%)
   div	97.4% (97.7%)

   High score, but not learned · experiments/residual-pilot-basic-seed4000{0,1,2}.json (pooled)

4. The switch · 8-bit integers · all bits right or wrong

   Write numbers as bits, score only exact answers

   Instead of one decimal number, each of a, b, c becomes 8 bits, and the model predicts each bit of c as 0 or 1. Results wrap at 256. An answer is right only when all 8 bits are. This trades “close enough” for a yes-or-no answer, and the model can say how sure it is about each bit.

   Before building anything, a quick test with no learning asked whether 999 examples are even enough. A hand-written lookup walks the bits from lowest to highest. At bit k it keeps the examples whose bits atk and at the previous position match the question, and copies their ck:

   relation	closest example	lookup, looking back 1 bit	looking back 2 bits
   add	2.1%	100%	99.2%
   sub	3.5%	100%	99.1%
   xor	4.7%	100%	100%
   a+2b	3.9%	34.3%	94.5%
   max	21.1%	43.2%	71.2%

   Three lessons: copying the closest example is useless with bits (2–5% for add/sub); going bit by bit, remembering just the previous position, is exact for carry-style relations; and looking further back fails because 999 examples don't cover all the combinations. So the model has to learn a short summary of the bits so far — like a carry — and that is what its loop does.

5. Bit-serial model · first learned model

   A learned loop over bit positions, plus a lookup over the examples

   One shared recurrent network reads every example one bit at a time; at each bit the question looks up the examples in the same situation; one pass runs from the lowest bit, one from the highest. 100% exact on all 12 named relations it trained on — but only 32.2% on min, the mirror of a trained idea, because it could not decide which direction to read. Inside this model →

6. All bits at once · loop on the answer

   Drop the reading order, keep a loop on the answer

   Each example becomes one vector from all 24 of its bits, fixed for the whole run. The question keeps a guess of all 8 answer bits and corrects it every round. Never-seen relations improved: 3a75.8% → 92.3%,min 32.2% → 46.3%, and confidence became far more useful.

7. Multiplication · multiplicative units

   Networks mostly add; give them a way to multiply

   Trained on a*b alone, the plain loop stalls at 52.3%. Adding bilinear inputs (two projections multiplied) and gated feed-forward layers lifts it to 98.1%. Among 14 relations, though, multiplication got only 1/14 of training and reached 45.3%. Two other designs were dead ends: letting the 999 example tokens act as working memory was too slow on a 4 GB GPU, and inferring the weights of a tiny calculator with a periodic (cosine) readout could represent multiplication exactly but could not be trained.

8. Adaptive sampling · random expressions

   Spend training where the model is still wrong

   Drawing relations in proportion to how often they are still wrong fixed the starvation: a*b99.6% inside mixed training, every earlier relation 98–100% (min97.7%). Random 1–3 operation expressions joined training, with eight compositions never generated — checked by behaviour, so rewrites cannot leak. Those hidden compositions reached50.0% on average (copying the closest example: about 9%).

9. Product, then another step · a computation limit

   It multiplies, it adds — but not one after the other

   a*b+a stayed near 19.0%: the low four bits right, the high bits wrong — carries through an intermediate result. More multi-step training data lifted hidden compositions to53.5% but not this; a round counter did not help either. Trained on nothing but four such relations, the model still reached only 39.1%: the loop has no place to hold an intermediate result.

10. Loops and the difficulty ladder

    Functions defined by loops, measured on a ladder

    Training now includes loops whose repeat count comes from b (sums of sequences, repeated application, Fibonacci-like recurrences, bit counting), and the tests are a ladder from 2-operation expressions to hidden loop bodies run longer than ever trained. The model may either execute the loop or find a shortcut — both count. See the results →

11. Scratch slots · memory that is never scored

    Somewhere to keep intermediate results

    Four memory vectors per question, carried between rounds, lifted multiplication inside mixed training from53.9% to 80.8% and the Fibonacci-like loop from33.1% to 47.9% — but a product followed by another step stayed near 19.2%, even when trained on nothing else (46.3%).

12. Bit tokens · step-by-step training tried and dropped

    One token per answer bit

    The limit was never parameters: carries have to move between bit positions through an intermediate result, and one question vector has no structure for that. Giving every answer bit its own token — attending to the other bits and to the examples each round — took that capacity check from 39.1% to99.4%. Training each round against one intermediate step (an operation, or a loop iteration) did not help, kept or faded out: the intermediate values never appear in the examples. Bit tokens trained on final answers only became the model. The design →

13. Guardrails · a sealed test set

    Checking that it is not fitted to our own tests

    Every test had been used to choose architectures and data. A sealed set of 24 everyday functions, written separately, is evaluated only at the end of a round and never used to decide anything. Every 8-bit candidate cleared it with honest confidence — but at 8 bits the 999 examples cover 1.5% of all inputs, and lookup methods that understand nothing answer about a third of it. That, and the next chapter, moved the measure to 16 and 32 bits.

14. Bit-aligned reading

    Each answer bit reads the same bit of the examples

    On our generator's held-out combinations (T1–T3) it scored 77.0% / 55.1% / 54.3%, 8–16 points above the plain bit tokens — but nothing extra on the sealed set. The gain was specific to relations shaped like our generator's.

15. Wider numbers

    One model for any width up to 32 bits

    The 8-bit design trained on 12-bit numbers collapsed (hidden compositions 15.0%). A width-free version sees every bit relative to its own position and from the lowest bit, so the same weights answer any width up to 32 bits. At 32 bits it gets some unseen relations far above lookup — a conditional it never trained on 77.0% against 1.1% — and fails others completely.Where it stands →

16. Composition at 16 bits

    Identifying the relation, not computing it

    It computes a*b alone almost perfectly at 16 bits, but a*b+b was 0.4% even after training only on products followed by a step. Trained on just four such relations it gets a*b+b 98.2%: the arithmetic is fine; recognising which relation the examples come from is not. Deeply worked probe examples (0.1%), four layers of per-example processing, and a pool of 16 relations did not change that. The model learns these relations one at a time, each needing about a thousand episodes, instead of one way to compose them.

17. Mixed training, longer

    No late jump

    Multiplication at 16 bits had stayed near zero for 7,000 updates of mixed training before rising to 70–80%, so a long run trained on every kind of composition at random, at 16, 24 and 32 bits. Six thousand updates in, every composition built on a product was still at 0–2% at every check, and it was stopped.

18. A composition grid

    It memorises combinations instead of combining parts

    Eight product-like first steps × eight second steps: 52 combinations trained, 12 held out, every part still trained in other combinations. The trained combinations it learned reached 87–97%, but held-out ones whose parts it knew stayed low: (a*b)+a 7.7% and ((a+1)*b)^b 10.1%.

19. Chains · a new measure

    Multi-step chains work — except through a product

    Chains are a start such as a+b or a*b followed by one or two of 21 kinds of step, with a fresh random constant each time. A new measure counts only the questions every lookup method gets wrong. At 32 bits the model answers chains of additions, bitwise operations and shifts mostly where lookup fails, and its sure answers are always right; chains through a product it hardly ever gets, and it is never sure of them — although products followed by a step were in its training. The composition failure of the last chapters was a product failure.Numbers in the findings →

20. Tried · a relation-code bottleneck

    Guess the relation first, then apply it

    The answer bits read the 999 examples directly, which lets the model look up similar examples and learn each relation on its own. The current experiment adds a bottleneck: a few relation-code tokens per bit position read the examples, and the answer bits read only that code. Still one pass, nothing fitted at answer time, and the code is free vectors rather than a choice among fixed operations. It learned trained combinations faster, in half the time, but combined known parts no better.

Where it goes next

Every experiment

Exact match on 1,024 fresh questions per relation (64 episodes × 16, seed 60000), at 8 bits unless the run says otherwise. "Trained" is the mean over the relations each run trained on, so rows with different training sets are not directly comparable. Every run is described in the findings.