Ternary networks: getting started

A ternary network stores each weight as one of {-1, 0, +1} — two bits instead of thirty-two. This tutorial takes a small classifier from the sequential { } DSL through FP32 training to a 2-bit packed model running through SKaiNET’s exact ternary kernel on a Raspberry Pi 4 class board (Cortex-A72), where the kernel was tuned in the first place.

Three things make this path different from ordinary quantization:

  • ~16× less weight memory. The BITNET_B1_58 layout packs four weights per byte plus one FP32 scale per tensor — 0.25 bytes per weight against 4 for FP32.

  • Exact results. Activations stay FP32 end to end. A ternary weight is an add, a subtract, or nothing, so the kernel’s output equals the FP32 matmul against the decoded weight (only float summation order differs). There is no activation-quantization error to budget for — the alternative W1.58A8 int8 path trades ~1.5 % error for sdot throughput; this path trades nothing.

  • Baseline NEON only. The SIMD kernel (vendored from NeoGPU, MIT) uses a 4 KB decode LUT and plain vfmaq_f32 — no FEAT_DotProd required. That makes it the fast path precisely on Pi-4/Cortex-A72 class cores, where dotprod-dependent kernels fall back to scalar.

The weight layout

TensorEncoding.BITNET_B1_58 is the storage format (the same packing BitNet b1.58 uses). For a [n, k] weight matrix, the flattened n·k ternary codes are packed sequentially, four per byte, low bit-pair first, and the buffer ends with one little-endian FP32 scale:

byte i:  bits[1:0] -> element 4i      code 0 -> -1
         bits[3:2] -> element 4i+1    code 1 ->  0
         bits[5:4] -> element 4i+2    code 2 -> +1
         bits[7:6] -> element 4i+3
payload = ceil(n·k / 4) bytes, then 4 bytes FP32 per-tensor scale

TernaryCodec.encodeBitNet writes this layout; the native kernel reads the payload directly and the dispatcher applies the scale to the output.

Loading a BitNet.cpp-quantized GGUF (not this sequential layout)

A GGUF’s I2_S tensor isn’t always already in the sequential order above. BitNet.cpp’s own quantizer (quantize_i2_s) packs a different, grouped layout instead — 128-element blocks on its x86/AVX pipeline, 64-element blocks on ARM/NEON — which the kernel above cannot read directly. SKaiNET’s loader detects this and repacks it into the sequential layout automatically at load time; the result is correct, but that repack is a real cost paid on every load unless the file is converted once, ahead of time.

If you’re preparing a model for repeated loads (a shipped app, a benchmark harness), convert it once instead of paying the repack cost every time: I2sAotConverter (GGUF → GGUF, built from source like the rest of the toolchain) or the IREE-facing equivalent in SKaiNET-IREE-tools re-encode the grouped layout into the sequential one ahead of time, so the load path never repacks at all and gets the same zero-copy mmap treatment every other packed format has.

A native kernel that decodes the grouped layout directly — skipping the repack (and any AOT conversion) entirely — was considered and explicitly not built (#1205): AOT conversion already gets a converted file to the same zero-copy fast path with far less ongoing kernel-maintenance cost (one layout to decode, not three, across every SIMD backend). See #1205’s closing comment for the full reasoning and what a grouped-layout kernel would actually need if a real workload someday can’t tolerate an AOT step at all.

Step 1 — Define and train the model in FP32

Nothing about the architecture changes for ternary. This is the same MNIST-shaped classifier as the Kotlin getting started tutorial — define it, train it there, and keep the trained FP32 weights:

val model = sequential<FP32, Float>(ctx) {
    input(784)                                   // 28x28 flattened
    dense(128) { activation = { it.relu() } }    // hidden layer
    dense(10) { activation = { it.softmax(1) } } // class scores
}

Training stays FP32 (see the training section); ternary weights are produced from the trained model. For a small classifier, straight post-training ternarization (next step) typically costs a few points of accuracy; quantization-aware training recovers most of it, but start simple and measure.

Step 2 — Ternarize the trained weights

TernaryCodec.encodeBitNet performs absmean ternarization: it scales the tensor by the mean absolute value, rounds each weight to {-1, 0, +1}, packs four codes per byte, and appends the scale.

import sk.ainet.lang.memory.TernaryCodec

// weights: FloatArray of the trained [n, k] dense weight, row-major
val packed: ByteArray = TernaryCodec.encodeBitNet(weights)

The memory arithmetic for the classifier above — biases stay FP32, they are noise at this scale:

Tensor FP32 BITNET_B1_58

dense1.weight [128, 784]

392 KB

24.5 KB + 4 B scale

dense2.weight [10, 128]

5 KB

0.32 KB + 4 B scale

total weights

~397 KB

~25 KB (≈16×)

The same ratio holds at any scale: a 2.4 B-parameter BitNet model’s ternary tensors drop from ~9.6 GB FP32 to ~0.6 GB packed.

Step 3 — Run it through the exact kernel

Wrap the packed bytes and the FP32 activations as views, and dispatch. The weight’s format is what selects the kernel: KernelDispatch.matmul checks the exact key matmul(FP32 dense × BITNET_B1_58) before anything else, so the ternary fast path engages with no changes to your model code, your DSL, or the dispatcher.

import sk.ainet.backend.api.kernel.KernelDispatch
import sk.ainet.lang.memory.Storage
import sk.ainet.lang.memory.TensorView
import sk.ainet.lang.memory.TernaryBlockDecoder
import sk.ainet.lang.tensor.Shape
import sk.ainet.lang.tensor.storage.TensorEncoding
import sk.ainet.lang.types.FP32

val weight = TensorView.packed(
    Storage.Heap.wrap(packed), Shape(n, k), TensorEncoding.BITNET_B1_58,
    TernaryBlockDecoder(TensorEncoding.BITNET_B1_58, n * k),
)
val activation = TensorView.dense(Storage.Heap.wrap(x), Shape(1, k), FP32)
val logits = TensorView.dense(Storage.Heap.floats(n), Shape(1, n), FP32)

KernelDispatch.matmul(activation, weight, logits)

Without any native pack installed this already works — dispatch requantizes the activation to int8 and serves through the portable bitnet_gemv reference. Installing a pack upgrades it to the exact NEON path.

Step 4 — Native on the Raspberry Pi

A Pi deployment is a Kotlin/Native linuxArm64 binary. The kernels archive it links carries the LUT kernel compiled at -march=armv8-a — deliberately below the archive’s usual armv8.2 flags, so it runs (and runs fast) on the A72. One install call at startup wires it into dispatch:

import sk.ainet.exec.kernel.NativeKnTernaryF32Gemv

fun main() {
    NativeKnTernaryF32Gemv.install()   // "ternary_f32_gemv/cinterop" now serves
    // ... load packed weights, serve requests ...
}

Add the target and dependency in build.gradle.kts:

kotlin {
    linuxArm64 { binaries.executable() }
    sourceSets.commonMain.dependencies {
        implementation("sk.ainet.core:skainet-backend-native-cpu")
    }
}

The same kernel, same C file, reaches every other deployment shape. On the JVM and Android the install is automatic: the ternary packs are ServiceLoader entries (FfmTernaryKernelPackFactory / JniTernaryKernelPackFactory), so KernelDispatch.ensureInstalled() — which runs on the first dispatch — wires them with no bootstrap call. Kotlin/Native has no ServiceLoader, so the explicit call remains:

Target Bridge Install

Raspberry Pi / linuxArm64, iOS, macOS

Kotlin/Native cinterop

NativeKnTernaryF32Gemv.install() (explicit)

Android

JNI (baseline libskainet_jni.so — every arm64 device, A72 included)

automatic (ServiceLoader; JniTernaryF32Gemv.install() still works)

Desktop/server JVM

FFM (java.lang.foreign)

automatic (ServiceLoader; NativeTernaryF32GemvKernel.install() still works)

Removing a native artifact is never an error: the pack warns and dispatch falls back to the portable path — slower, still correct.

What to expect

The kernel’s upstream measurements on a Raspberry Pi 4 (Cortex-A72 @ 1.8 GHz): 6.78 GOPS on the fused full-vocab projection with 4 threads against 3.20 GOPS for the int8 path (which also carries quantization error). It threads internally with pthreads once a projection has ≥ 512 output rows; below that it stays on the calling thread. For a small classifier the honest summary is: the memory win is the headline (your whole model fits in L2), and the exactness means ternarization is the only accuracy decision you have to make.

Status and roadmap

The kernel, its dispatch pack, and all three bridges are merged (#1136 tracks the effort; the C kernel is vendored verbatim from NeoGPU under MIT, agreed in neogpu#1). Landed in 0.49.0:

  • GGUF I2_S import with keep-packed loading (#1140) — load a BitNet-b1.58 GGUF and get packed BITNET_B1_58 tensors instead of FP32-widened ones, which will let a dense layer’s weight arrive packed without the manual encodeBitNet step above.

  • Benchmark scenario + tuning notes (#1141).

  • BitNet model support in SKaiNET-transformers (transformers#335).

  • Off-heap storage, zero-copy mmap for sequential-layout files, and an AOT GGUF converter (#1198, #1202, #1203, #1207) — landed in 0.51.0; see "Loading a BitNet.cpp-quantized GGUF" above.

  • A native decode kernel for BitNet.cpp’s grouped layout was considered and closed, not deferred (#1205) — AOT conversion covers the real use case at a fraction of the kernel-maintenance cost.