Architecture

1. The core insight

A BitNet b1.58 linear layer computes y = W·x where every weight is -1, 0 or +1 and activations are int8. The inner loop is therefore not multiply-accumulate but select-accumulate:

for each weight w, activation a:
    w == +1  ->  acc += a
    w == -1  ->  acc -= a
    w ==  0  ->  skip

The original reading of this was "so build hardware without multipliers." That is still true and still the endgame. What the pivot established is that the same observation pays on silicon people already own, provided the weights are stored so that a machine can act on them without decoding first:

• On a CPU, "does this weight participate, and with which sign" becomes a mask over a vector of activations. On AVX-512 the mask is a k register and the whole operation is two masked loads plus two vpdpbusd. Measured 32.5x the scalar path on this host.
• On fabric, it is a mux and an adder tree, needing no DSP blocks, so a modest part can host thousands of lanes.

Both want the identical thing from memory. That is what section 2 is about, and it is the single structural idea this codebase is organized around.

2. The one design commitment: bit planes

A ternary weight has three states, so two bits is the natural budget however you spend them. Version 0 of the .trit format spent them as an interleaved code per weight (00 = 0, 01 = +1, 10 = -1). Version 1 spends the same two bits planar: a 16-byte beat carries 64 consecutive columns of one row as {u64 pos, u64 neg}.

The bit count is identical. What changes is what one load can do with it.

y[r] = sum of x[c] where W[r][c] = +1
     - sum of x[c] where W[r][c] = -1

With planes, each of those sums is a mask applied to a vector of activations — no branch, no table lookup, no per-weight arithmetic. With interleaved codes, extracting the sign of lane k means shifting by 2k and masking, per weight, which defeats vectorization entirely and is exactly why the pre-pivot scalar kernel ran a branch per weight at 0.16 tok/s.

The consequence that matters most: one beat is simultaneously one iteration of the CPU kernel's inner loop and one cycle's weight input to the RTL core. A memory-mapped .trit is not a container the runtime decodes into some other layout. It is the layout. Nothing between the page cache and the accumulator rewrites a byte, on either target.

That is what makes "hardware-agnostic" a structural property rather than an aspiration. The CPU and the fabric are not two ports of the same idea kept in sync by discipline; they consume the same bytes.

Full specification: 01-TRIT-FORMAT.md.

3. The honest bandwidth math (read this first)

Autoregressive decoding at batch size 1 touches every weight once per token, so throughput is bounded by:

tok/s ~= effective_memory_bandwidth / bytes_touched_per_token

Measured on BitNet b1.58 2B4T, reported by tritd info:

The dense head is still the largest single stream, even at bf16. On a tied-embedding model with a 128256-word vocabulary, the non-ternary tail rivals the entire ternary body. It used to dominate it 2.5:1, because tritc widened the checkpoint's bf16 tensors to f32 and stored 1,313,341,440 bytes where 656,670,720 carry the same values. Restoring the source precision removed 656 MB per token and is exact: see 02-ROADMAP.md A3.

This is also the shape of the pre-pivot mistake, which is worth keeping in view. That version of this section estimated a "~600 MB packed" model and reasoned entirely about ternary weight streaming, and was wrong about where the bytes were.

So the runtime's dense f32 matvec is not incidental. MatvecBackend::f32_matvec exists as a first-class trait method for exactly this reason — a backend that vectorizes the ternary kernel and leaves the head scalar has not moved end-to-end throughput at all.

Where that leaves us

Measured on an AMD Ryzen 9 8945HX (Zen 4, AVX-512 VNNI), 4 threads, benches/prompts/short.jsonl, median of three:

The roofline figure is a measured ratio, not an assumption about what the memory system can do: tritd bench runs a streaming-read probe on the same machine in the same invocation, best of five passes, and divides by it. The probe itself reads 48-58 GB/s across runs, so the pair belongs together and this host has no single ceiling worth quoting.

This machine drifts by up to 30% between runs, so absolute figures here are not comparable across sessions. Anything claiming a speedup should be an interleaved paired measurement against the thing it is faster than; 06-GATES.md is the frozen reference.

Consequences baked into the design:

• Weight streaming, not weight caching. Weights pass through once per token; only activations, norms and KV pages want to be resident.
• Prefill is the exception. With N prompt tokens you amortize one weight pass over N tokens of work, so prefill runs compute-bound. TTFT looks disproportionately good; steady-state decode is the honest number.
• Bytes are not time. Measure both. This document once argued from the byte share alone and concluded the head was the thing to attack. An instrumented decode disagreed: with an f32 head it was 26.9 ms of a 64.4 ms token, 42% of the time against 72% of the bytes, because it ran at 48.8 GB/s while the ternary path ran at 13.9. The two halves sat at opposite ends of the machine's efficiency range, so the byte split misled about where time went. Both have since moved — the head is bf16 and the ternary path can use cores — but the lesson holds: a byte count predicts time only when the two paths run at comparable efficiency, and here they did not.

Where the time actually goes

Measured with an f32 head and a single-threaded ternary path, which is the configuration the two findings below came from:

Two opposite conclusions came out of that split, and both held.

The head was already at roughly 92% of this machine's ceiling, so no kernel work would move it: forcing the dense kernel to scalar, AVX2 and AVX-512 in turn gave 28.8, 28.1 and 27.1 ms. The only thing that could help was moving fewer bytes, which is what storing it in bf16 does.

The ternary path was the opposite: compute-bound in the kernel at about 15.8 GB/s per core, barely above what it achieved from DRAM, which meant it would scale with cores if the runtime could use them. Section 7 is how.

4. System overview

   HF checkpoint
        |
        v
   +---------+   folds norms, quantizes, packs planes, verifies
   |  tritc  |
   +---------+
        |  .trit v1  (mmap'd; the payload IS the beat stream)
        v
   +-------------------------------------------------------+
   |  trit-core: format, transformer, KV cache, RoPE,       |
   |             sampler, tokenizer traits, numerics ladder |
   +-------------------------------------------------------+
        |  MatvecBackend  (the one seam: bit-exact by contract)
        +----------------------+----------------------+
        v                      v                      v
   +----------+          +-----------+         +---------------+
   | trit-cpu |          | trit-rtl  |         | ScalarBackend |
   | AVX-512  |          | Verilated |         | portable ref  |
   | AVX2     |          | tritcore  |         +---------------+
   | NEON     |          +-----------+
   | scalar   |                |
   +----------+                v
        |               rtl/trit_matvec.sv
        |               (multiplier-free, 64 lanes)
        v
   +---------+   run | serve | bench | info,  plus a C ABI
   |  tritd  |
   +---------+

   tritsim: an independent second implementation of the whole model,
            deliberately naive. Not in this data path -- it is the
            oracle the data path is diffed against.

4.1 The backend seam

MatvecBackend is the one abstraction that matters, and its contract is unusual: implementations must be bit-exact with the portable reference for every input, including xq = -128. Not "within tolerance" — identical.

That is achievable by construction rather than by tuning, because the ternary accumulators are i32 and integer addition is exact and order-independent. A backend that cannot meet it is wrong, not approximate. It is what lets the CPU kernel, the RTL core and the reference be swapped for one another without moving a logit, and it is why the differential tests can demand equality instead of a threshold.

The trait is injected, not global. A process-wide selector works for a single-shot CLI and breaks the moment two sessions want different backends.

4.2 Kernel selection

Resolved once by runtime feature detection, cached for the process. --kernel (or TRIT_CPU_KERNEL) pins one and errors if this CPU cannot run it, rather than falling back. A silent fallback would let a CI runner without AVX-512 report green while actually re-testing the scalar path, and would let a benchmark measure a different kernel than the one it names.

Measured on a 2560x6912 tensor with the planes resident in L3:

4.3 About popcount

Popcount is the natural primitive when activations are also 1-bit. Here weights are 1.58-bit but activations are int8, so a popcount of a weight mask only counts terms — it cannot recover the sum. The production kernels mask activation bytes and accumulate the two planes separately.

The bit-serial popcount formulation is implemented anyway, behind a feature flag, because it is the one that maps directly onto the RTL adder tree. It measures 22x slower than AVX-512 at int8 activations. It stays as a third independent implementation for the differential tests, and because it becomes the right kernel if activations ever drop below 8 bits.

5. Correctness architecture

Three implementations of the same arithmetic must agree:

• tritsim — the oracle. Naive, scalar, deliberately obvious. It holds one i8 per trit rather than reading planes, because its job is to be evidently correct rather than fast.
• trit-core + trit-cpu — the runtime. Zero-copy, vectorized, threaded.
• rtl/tritcore — the hardware, under Verilator.

Agreement is demanded exactly on the integer path. Lane order, thread count and the RTL's beat-serial accumulation must all produce identical bits. Only the f32 tail (attention, LM head) admits reassociation, and even there the cross-implementation check measures cosine 1.000000 at every position on the real checkpoint.

This is not ceremony. It has caught real defects that no eyeball would:

• A folded w_scale * x_scale constant that multiplied once where the reference multiplies twice. f32 multiplication is not associative, so the two rounded differently: invisible for four token positions, cosine 0.999624 by position 7, and a changed word by token 16.
• An oracle that fell back to the embedding matrix for a missing lm_head while the runtime refused to — meaning the two sides of the parity gate could have been comparing different models, in a way the parity test itself could never see, because it never got as far as running.

The frozen reference is 06-GATES.md. G1-G6 and G8 are invariants: they may not move without an argued entry. G7 holds the resource numbers the pivot exists to improve, and is expected to move.

A note on what counts as coverage: the aarch64 CI job spent its entire existence failing at clippy before reaching a single assertion, while the test matrix advertised ARM coverage. Two real defects were sitting in code that matrix claimed to cover. A job that cannot reach its assertions is not coverage.

6. The numerics ladder

Three rungs, each a strictly more exact evaluation of the same model, selected automatically to the best the architecture supports:

Both upper rungs were proved against the real checkpoint before being made the default. Norm folding matters beyond speed: it removes the rsqrt and the divide from the hardware datapath, which is a synthesis result as much as a numerics one.

7. Threading

Threading splits rows, never a reduction, so results are bit-identical regardless of thread count or scheduling.

The dispatch mechanism turned out to matter more than the parallelism. A decode step issues 210 ternary matvecs whose largest is 4.4 MB, and handing each to a work-stealing pool costs more than the matvec itself: layer time went 37.4 ms at one thread to 173 ms at eight. For a long time the runtime avoided the problem by refusing to parallelise anything smaller than 32 MiB, which meant the ternary path was effectively single-threaded.

Four mechanisms were measured in the real decode loop. A dedicated pool sized to the thread count and a broadcast primitive both helped substantially and both remained slower than not parallelising at all. Only a persistent pool on a sequence counter beat serial execution, at 2.9x on the layer path.

crates/trit-cpu/src/pool.rs is that pool: workers spawned once, waiting on a counter, with no allocation and no scheduler involvement per job. They spin briefly and then park, because jobs arrive roughly every 300 us and spinning through that gap wastes a core per worker, which the four-core parts this runtime targets cannot afford.

Two properties are load-bearing and both are tested:

• run blocks until every slot it dispatched has reported, which is what makes the pointers it hands out sound. Nothing escapes the call.
• Slots with no chunk report nothing. When idle workers incremented the completion counter, their increments satisfied the barrier before the working slots had finished and run returned while another thread was still writing.

Slot count is bounded by the work as well as the pool, at 256 KiB of planes per slot, and the dense head shares the same pool rather than running a second one. The automatic thread count caps at eight: sixteen measures 0.94x against the previous implementation, so one per core is the wrong answer to "decide for me" on a large machine.

What remains is that the curve peaks at four threads and declines, where bitnet.cpp keeps climbing to eight. That is the open gap.

8. The silicon track

The RTL is not deprecated by the pivot and is not a museum piece. It is a backend, exercised by CI on every push, and its bit-exactness is a frozen invariant.

Proven today:

• rtl/trit_matvec.sv consumes .trit v1 beats directly — w_pos/w_neg, 64 lanes, a combinational select-accumulate tree, i32 accumulator.
• Bit-exact against golden vectors including the x = -128 extremes set, which any implementation that negates in the i8 domain fails.
• Yosys generic synthesis asserts zero $mul and zero $macc cells on every run. Baseline 33,659 cells at 64 lanes, of which ~4.1k DFF are the flattened activation memory (BRAM on any real part).
• The real 2B4T model decodes end-to-end through the Verilated core, byte- identical to the CPU path, at 24.4 s/token. That number is what says silicon is the next step, not a defect.

Not proven:

• No timing closure. The 64-term single-cycle reduction and 64 parallel activation reads are fine under Verilator and are not yet placed on a part.
• MAX_COLS is 8192, which every projection in this model class fits under. The wrappers now reject anything wider rather than wrapping the address silently.
• No board. The selection memo (04b-BOARD-MEMO.md) recommends a Kria KV260 on roofline-versus-synthesis grounds; the gate passed and the purchase decision is unmade.

9. Non-goals (v1)

• Training or QAT. We consume checkpoints, we do not make them.
• Batching, multi-user serving, speculative decoding.
• Long context beyond 2048; vision or multimodal.
• Beating a GPU on absolute tok/s. The metric is tokens per joule per dollar.

10. Open questions

• The f32 head. int8 is the obvious move and the roofline says ~2.2x. What it costs in quality is unmeasured, and the answer decides whether it ships as the default or as a flag.
• Base-3 packing (1.6 b/w, ~20% less traffic on the ternary body) against the unpack cost, on both CPU and fabric. Worth less than it looks while the head dominates bytes/token.
• KV cache precision. f32 and preallocated at 315 MB. int8 pages would cut resident footprint substantially and matter more as context grows.
• ARM throughput. The NEON kernels are correct on hardware and have never been timed. Until they are, no ARM performance claim exists.
• Where the RTL's activation memory should live once a board is real — on-chip banking versus DDR spill — which is a placement question the simulator cannot answer.