Regression gates

The frozen reference for the hardware-agnostic pivot. Every number here was measured on the pre-pivot tree at commit c3351d1 (v0 .trit, scalar kernel, single-threaded) before any pivot code landed.

Rule: nothing in the pivot may change a number in this file without an explicit, argued entry in the "Sanctioned changes" section below.

Environment of record

Other hosts will differ on timing and RSS. The correctness gates are host-independent and must hold everywhere.

G1 — Unit and integration tests

cargo test --workspace

All pass. trit-core 12, tritsim lib 19, tiny_model.rs 6.

G2 — Real-checkpoint logit agreement (the headline gate)

tritsim compare --model models/bitnet-2b4t.trit --dump logits.json

6 positions: mean cosine 0.9991, top1 match 100.0%

Last position, top-3 both sides:

ours   [(12366, 18.34601), (539, 14.126393), (264, 13.815544)]
theirs [(12366, 18.5),     (539, 14.25),     (264, 13.75)]

Single-position variant (--dump bos_dump.json): 1 positions: mean cosine 0.9977, top1 match 100.0% — identical under TRITSIM_INT_MLP=1.

CLI acceptance thresholds (crates/tritsim/src/main.rs): cosine >= 0.98, top-1 >= 0.90. The measured values sit far above them; the gate is the measured value, not the threshold.

G3 — Greedy text, byte-exact

tritsim run --model models/bitnet-2b4t.trit \
            --tokenizer models/bitnet-2b4t/tokenizer.json \
            --prompt "The capital of France is" --steps 16

 Paris. Paris is a city in the north of France, and it is the

Must reproduce byte-for-byte in Reference, Folded and IntMlp modes, and on every backend (cpu, rtl) and every CPU kernel.

G4 — RTL bit-exactness

make -C rtl lint          # clean
make -C rtl test          # all sets, exact i32 equality

SET vectors/exact_64x64:      64 rows OK
SET vectors/padded_5x100:      5 rows OK
SET vectors/extremes_4x128:    4 rows OK
SET vectors/zeros_3x64:        3 rows OK
SET vectors/wide_2x6912:       2 rows OK
SET vectors/model_k_proj_l0:   8 rows OK
SET <invalid-code>:            err raised OK

extremes_4x128 feeds x = -128 deliberately. Any CPU kernel that negates in the i8 domain fails here. This set must never be weakened.

G5 — Multiplier-free synthesis

make -C rtl synth

Yosys asserts zero $mul and zero $macc cells. Baseline cell count 33,659 (MAX_COLS chparam'd to 512; $_DFFE_PP_ 4096 is the flattened activation memory). Generic synthesis, no device target.

G6 — Hardware-in-the-loop parity

cargo test -p tritsim --features rtl --release

random_shapes_match_cpu_exactly compares the Verilated core against the CPU kernel on 8 shapes (1x64, 3x64, 2x100, 5x129, 7x640, 2x2560, 2x6912, 16x61) and requires exact equality. All pass.

G7 — Resource baseline (the numbers the pivot exists to move)

These are targets to improve, not invariants. G1-G6 are invariants.

G8 — trit-core against the tritsim oracle

Two independent implementations of the same model must agree.

cargo test -p tritsim --release --test cross_implementation              # tiny fixtures
cargo test -p tritsim --release --test cross_implementation -- --ignored # real checkpoint

On the real checkpoint, 8 positions, all three numerics rungs: cosine 1.000000 and identical top-1 at every position.

This gate found a real bug. The production path had precomputed w_scale * x_scale into a single constant and multiplied once, where the reference multiplies twice: acc as f32 * w_scale * x_scale. f32 multiplication is not associative, so acc * (w * x) and (acc * w) * x round differently. The drift was invisible at short contexts (cosine 1.000000 for four positions), grew with position to 0.999624 by position 7, and changed the 16th generated token from "the" to "known". Ordering the multiplications to match the reference restored cosine 1.000000 everywhere.

Post-pivot results

The pivot's purpose was to move the G7 numbers without moving G1-G6. Measured on the environment of record with models/bitnet-2b4t.trit:

Decode is at 4 threads, which is where this host peaks.

The roofline fraction is a ratio against a probe run in the same invocation, and that probe is itself noisy: across the five thread counts in the committed CSV it read 48.25, 58.27, 57.24, 52.67 and 49.78 GB/s. At 4 threads, 32.85 against 57.24 is 57%. Quote the pair, never the fraction alone, and never a fixed ceiling for this host. The pivot's "66% of roofline" was computed against a single-pass probe that under-reported, so it was optimistic on top of that.

Kernel throughput on a 2560x6912 tensor, planes resident in L3:

Against other runtimes

Same host, same session, greedy, 32 tokens, median of three. bitnet.cpp runs the identical checkpoint; llama.cpp runs Qwen2.5-3B Q4_K_M because mainline cannot load i2_s, so that column is a different model at a different quality point.

Ahead at 1, 2 and 4 threads, 1.43x bitnet.cpp per core. Still behind above 4 threads: our curve peaks and declines where bitnet.cpp keeps climbing to 32.49. That gap is the remaining scaling work.

Threading

Measured, not assumed, and the measurement changed twice.

Before a persistent pool existed, splitting each of the 210 ternary matvecs per token across the machine cost more than it saved: 14.13 tok/s at one thread against 2.40 at 32. PARALLEL_MIN_BYTES was set to 32 MiB to prevent it.

With the pool, dispatch is a sequence-counter bump and an unpark, and the threshold drops to 64 KiB. Four mechanisms were measured in the real decode loop; a dedicated work-stealing pool and a broadcast primitive both stayed slower than not parallelising at all.

Layer time in ms. All produce byte-identical output.

G1-G6 are unchanged throughout. G2 still reports 0.9991/100%, G3 is still byte-identical in every numerics mode on both implementations, and the real checkpoint still decodes byte-identically through the Verilated RTL core -- 24.4 s/token in simulation, via tritd --backend rtl.

Sanctioned changes

Numbers here move only with an entry below, naming the phase, the reason, and the before/after.

The one change already anticipated: RoPE currently computes theta.powf(-2.0 * i / head_dim) in f32 (crates/tritsim/src/math.rs), giving angle errors up to ~1e-3 rad at position 2048. Moving both implementations onto a shared f64 inv_freq/rope_angle is expected to shift G2 slightly — in the direction of the reference, since the reference is more accurate. That gets its own isolated phase and its own row above.