Weight Quantization and Memory Residency

Since the engine-adoption arc (issues #338–#346, engine 0.50/0.51 line), this repository owns no quantization machinery of its own. Every family’s weight loader is a thin wrapper over the SKaiNET engine’s StreamingGgufParametersLoader, and what a weight looks like in memory is a declared, resolved form — not the emergent result of per-family converter code. The historical multi-stage pipeline this page used to describe (MemSegWeightConverter, QuantPolicy, pre-transposed FP32 copies, hand-managed `Arena`s) is gone.

A loader states what it wants per tensor; the engine resolves and delivers it — or visibly refuses. WeightForm has four axes:

The default every decoder-family loader requests:

A request the platform or encoding cannot honour is not an error — the engine heap-stages that tensor and the memory plan says so. The precedence is always: caller override > resolver > file.

A mapped weight is mmap-backed page cache: resident in RSS while hot, evictable under pressure, and outside the JVM/ART heap cap. Measured consequences:

The zero-copy part is the kernel story, not just the storage story: the FfmRowMajorKernelPack (JVM) and JniMappedKernelPack (Android) register matmul kernels under BLOCKED_ROW_MAJOR dispatch keys that read canonical packed blocks where they lie — mapped buffer or un-prepacked heap array — so nothing is copied or dequantized on the hot path. Installing them is two lines at startup (the unified CLI does exactly this):

Without the pack, dispatch falls back to the decoding reference kernel — correct for every format, visibly slower, and (since engine 0.50) emitted as a trace event rather than silently absorbed.

Embedding gathers rows; matmul kernels never see it. A packed embedding table must not be read through per-element get() — for packed tensor data that returns the raw quantization code, not the value (an engine source-compat quirk, and the root of the historical #993 class of bugs). Instead the decoder loader rewraps a packed delivery as PackedRowDequantTensorData (transformer-core): a RowDequantSource that dequantizes exactly the rows a step looks up via PackedBlockStorage.dequantizeBlock.

This keeps a 152k × 1536 vocabulary at its ~131 MB packed footprint instead of a ~933 MB dense FP32 heap array — and because the wrapper forwards PackedBlockStorage and the underlying view, a tied output.weight that aliases the embedding still routes matmulWeightTransposed through the packed kernel chain. (Flipping SmolLM2’s tied lm_head from dense-transpose to the packed kernel took the CLI smoke from ~3.6 to ~50 tok/s.)

Every transformer DSL module projects through linearProject (transformer-core), whose entire body is:

The engine primitive serves x · Wᵀ for a [out, in] weight without materializing a transpose: mapped packed → zero-copy row-major kernel; heap packed → relayout-once cache + packed kernels; dense → the standard transpose path. Dispatch selects by the weight’s storage format, never by model family.

MemoryPlans.plan prices weights (split mapped vs heap-charged), KV cache, forward slab, and headroom from the GGUF header — before a byte of payload is read. The unified CLI prints it on every run and warns when the plan exceeds the heap cap; --explain-load additionally prints one line per weight saying where it lands and why:

Weights are the big number but not the only one. OptimizedLLMRuntime runs DIRECT decode inside an engine ForwardScope (#343): step activations bump-allocate from one pre-sized slab and are recycled at the step boundary, so steady-state decode allocates zero new heap bytes per token. The KV caches copy their kept history out to ambient storage (detachFromStep); a caller holding a logits tensor across steps fails loudly (StorageClosedException), never silently. forwardScopeMetrics exposes peakFloats / overflowBytes for plan-vs-actual checks — ForwardScopeSteadyStateTest pins bit-identity with the unscoped path and a flat, overflow-free slab.

All seven GGML block formats the engine loader delivers — Q4_0, Q5_0, Q5_1, Q8_0, Q4_K, Q5_K, Q6_K — flow through the packed dispatch chain and are pinned against the engine’s canonical dequant by LinearProjectionPackedParityMatrixTest (W4A8 tolerance: packed kernels may quantize activations to int8 per block, so parity is an abs-floor/relative band, not bitwise equality). Ternary BitNet I2_S loads as packed BITNET_B1_58 (0.25 B/weight) with a BITNET_PLANES lm_head — see the BitNet module. Formats outside the loader’s fail-fast set (Q2_K, Q3_K, IQ-series) are rejected at load rather than silently mishandled.