teenie

Results

The difficulty ladder

The goal: recognise the rule behind increasingly complex functions — including ones defined by loops — and get at least half of the never-seen ones exactly right, while staying honest about when it is sure. Each run is scored at the width in its title (8 bits unless stated). At 8 bits the examples cover 1.5% of all inputs and lookup methods that understand nothing reach about a third of the sealed set, so 8-bit scores overstate what the model has inferred; the overview compares the 16- and 32-bit results with lookup directly. A tier is won at ≥50% exact with its sure answers right ≥95% of the time.

Tiers won by the latest run1 of 6+ 24- and 32-bit episodes — scored at 32 bits
Hidden compositions31.3%never generated for training
Unseen loop bodies (T5a)13.2%at the trained length; loops run longer than trained (T5b, a stretch): 26.3%

Scoreboard

bit tokens, step-by-step hints faded outbit tokens + division, remainder, comparisons, variable shifts, squares, bit countswidth-free bit tokens, 8/12/16 bits — scored at 16 bits+ 24- and 32-bit episodes — scored at 32 bits
0%25%50%75%100%targetT1T2T3T4T5aT5bhiddennear
Table view
tierbit tokens, step-by-step hints faded outbit tokens + division, remainder, comparisons, variable shifts, squares, bit countswidth-free bit tokens, 8/12/16 bits — scored at 16 bits+ 24- and 32-bit episodes — scored at 32 bits
target · unseen multi-step expressions— sure-right —— sure-right —11.9% sure-right 100.0%9.9% sure-right 100%
T1 · 2 operations50.2% ✓ sure-right 100%60.9% ✓ sure-right 100%33.5% sure-right 100%31.7% sure-right 100.0%
T2 · 3–4 operations36.2% sure-right 100%47.3% sure-right 100%24.1% sure-right 100%26.5% sure-right 100%
T3 · 5–7 operations39.0% sure-right 94.7%47.8% sure-right 98.9%16.7% sure-right 100%18.6% sure-right 100.0%
T4 · new operation types64.3% ✓ sure-right 99.8%74.2% ✓ sure-right 100%70.8% ✓ sure-right 99.9%63.9% ✓ sure-right 99.9%
T5a · unseen loop bodies32.4% sure-right 99.7%40.7% sure-right 99.2%13.8% sure-right 100.0%13.2% sure-right 100.0%
T5b · longer loops (stretch)42.1% sure-right 99.1%48.7% sure-right 99.5%30.2% sure-right 98.3%26.3% sure-right 98.4%
hidden compositions38.0% sure-right 100%56.0% ✓ sure-right 99.8%36.4% sure-right 100%31.3% sure-right 99.6%
product, then a step18.1% sure-right 87.5%27.9% sure-right 100%2.2% sure-right —0% sure-right —

T1–T3: random expressions of 2, 3–4 and 5–7 operations that always contain a combination never shown in training (a xor of a product, a product of a difference, a min of a sum, a shift of a xor). T4: operation types never trained — division, modulo, rotations, conditionals. T5a: loop bodies never generated (repeat count from b, up to 15 as in training) and unseen bit loops — recognising a new loop. T5b: loops run 63 or 255 times, longer than anything trained — repeating a step further than trained, a research stretch. Product, then a step: a*b+b, a*b+5, a*b−a, (a*b)^a.

During training

Mean exact match per tier on fixed validation episodes (128 questions per relation), every 500 updates.

target · unseen multi-step expressions

target during training
0%25%50%75%100%02,0004,0006,0008,00010,00012,00014,00016,00018,00020,000training updates50%0%

T1 · 2 operations

T1 during training
bit tokens, step-by-step hints faded outbit tokens + division, remainder, comparisons, variable shifts, squares, bit countswidth-free bit tokens, 8/12/16 bits — scored at 16 bits+ 24- and 32-bit episodes — scored at 32 bits
0%25%50%75%100%02,0004,0006,0008,00010,00012,00014,00016,00018,00020,000training updates50%52.7%58.6%28.1%64.8%

T2 · 3–4 operations

T2 during training
bit tokens, step-by-step hints faded outbit tokens + division, remainder, comparisons, variable shifts, squares, bit countswidth-free bit tokens, 8/12/16 bits — scored at 16 bits+ 24- and 32-bit episodes — scored at 32 bits
0%25%50%75%100%02,0004,0006,0008,00010,00012,00014,00016,00018,00020,000training updates50%56.3%63.7%28.1%25%

T3 · 5–7 operations

T3 during training
bit tokens, step-by-step hints faded outbit tokens + division, remainder, comparisons, variable shifts, squares, bit countswidth-free bit tokens, 8/12/16 bits — scored at 16 bits+ 24- and 32-bit episodes — scored at 32 bits
0%25%50%75%100%02,0004,0006,0008,00010,00012,00014,00016,00018,00020,000training updates50%38.3%49.6%23.4%45.7%

T4 · new operation types

T4 during training
bit tokens, step-by-step hints faded outbit tokens + division, remainder, comparisons, variable shifts, squares, bit countswidth-free bit tokens, 8/12/16 bits — scored at 16 bits+ 24- and 32-bit episodes — scored at 32 bits
0%25%50%75%100%02,0004,0006,0008,00010,00012,00014,00016,00018,00020,000training updates50%61.2%72%71.1%66.3%

T5a · unseen loop bodies

T5a during training
bit tokens, step-by-step hints faded outbit tokens + division, remainder, comparisons, variable shifts, squares, bit countswidth-free bit tokens, 8/12/16 bits — scored at 16 bits+ 24- and 32-bit episodes — scored at 32 bits
0%25%50%75%100%02,0004,0006,0008,00010,00012,00014,00016,00018,00020,000training updates50%31.4%40.7%15.8%12.8%

T5b · longer loops (stretch)

T5b during training
bit tokens, step-by-step hints faded outbit tokens + division, remainder, comparisons, variable shifts, squares, bit countswidth-free bit tokens, 8/12/16 bits — scored at 16 bits+ 24- and 32-bit episodes — scored at 32 bits
0%25%50%75%100%02,0004,0006,0008,00010,00012,00014,00016,00018,00020,000training updates50%43.6%49.1%36.5%21.5%

hidden compositions

hidden during training
bit tokens, step-by-step hints faded outbit tokens + division, remainder, comparisons, variable shifts, squares, bit countswidth-free bit tokens, 8/12/16 bits — scored at 16 bits+ 24- and 32-bit episodes — scored at 32 bits
0%25%50%75%100%02,0004,0006,0008,00010,00012,00014,00016,00018,00020,000training updates50%39.9%55%37.6%31.5%

product, then a step

near during training
bit tokens, step-by-step hints faded outbit tokens + division, remainder, comparisons, variable shifts, squares, bit countswidth-free bit tokens, 8/12/16 bits — scored at 16 bits+ 24- and 32-bit episodes — scored at 32 bits
0%25%50%75%100%02,0004,0006,0008,00010,00012,00014,00016,00018,00020,000training updates50%18.2%26.9%2.1%0%

Loops in detail

Latest run, exact match. Training only ever used repeat counts up to 15; 63 and 255 are longer loops. Some loops have short closed forms (the sum is a·n + n(n−1)/2) — finding either a procedure or a shortcut counts.

loopcount ≤ 15count ≤ 63count ≤ 255
sum a..a+n-1 hidden body21.5%0.3%0%
a^n hidden body19.0%30.1%44.7%
fib(a,a) hidden body22.4%0%0%
x->3x+1 hidden body7.9%0%0%
acc=(acc<<1)^a hidden body12.6%0%0.1%
accumulate random bodies71.3%49.3%56.8%
repeat random bodies49.3%63.5%76.3%
recurrence random bodies59.7%48.5%51.3%