The Quantization Process: Weights, Activations, and Calibration

Quantization in SKaiNET comes in two flavours that are easy to conflate. This page draws the line between them, then walks the activation post-training quantization (PTQ) pipeline β€” calibration, scales, and the int8 matmul shape β€” that an int8 NPU or accelerator target needs.

For the weight side, this article is an overview; the depth lives in How quantized SIMD kernels are built and TurboQuant KV-cache compression.

Two regimes, one word

Weight-only quantization Activation PTQ

What is quantized

The weights only. Activations stay fp32.

Both operands of a matmul β€” activations and weights β€” are int8.

Where it runs

Eager CPU/JVM kernels (Q8_0, Q4_K, Q6_K…), TurboQuant for the KV cache.

Integer accelerators (NPUs) that do i8 Γ— i8 β†’ i32 MACs.

The matmul

out[fp32] = Ξ£ act[fp32] Β· dequant(w_block) β€” dequant fused into the inner loop.

out = requantize( Ξ£ q_act[i8] Β· q_w[i8] ) with an i32 accumulator.

Needs calibration?

No. Block scales are derived from the weights at quantize time.

Yes. Activation ranges are data-dependent and must be measured.

In the codebase

QuantizedMatmul, the quantized SIMD kernels, TurboQuant.

Calibration in the framework; int8 emission in the target backend codegen.

The rest of this page is about the second column.

The int8 matmul shape

An integer accelerator computes a matmul as i8 Γ— i8 β†’ i32, so a single fp32 projection y = x Β· Wα΅€ becomes a three-part pattern:

q_x  = clamp(round(x / Sa), -127, 127)        # quantize activation  (fp32 -> i8)
acc  = matmul_i8(q_x, q_w)                     # i8 x i8 -> i32 accumulator
y    = acc * (Sa * Sw)                         # requantize           (i32 -> fp32)

with symmetric per-tensor scales

  • Sa = max|x| / 127 β€” the activation scale (data-dependent β†’ calibrated),

  • Sw = max|W| / 127 β€” the weight scale (static β†’ computed from the weights directly),

  • So = max|y| / 127 β€” the output scale, used when the result feeds another int8 matmul.

Zero-point is 0 under symmetric quantization, which keeps the requantize a single multiply. Softmax, LayerNorm and GELU stay in fp32 β€” only the matmuls are quantized.

Sw is free: the weights are known ahead of time. Sa and So are the problem β€” they depend on the activations, which depend on the input. That is what calibration measures.

Calibration: measuring activation ranges

Calibration runs the model eagerly in fp32 over a handful of representative inputs and records, per matmul, the absolute-max of each operand and of the result. The per-tensor scale is then S = max|x| / 127, accumulated (by max) across all calibration inputs.

The clean way to tap the activation stream is a TensorOps decorator that intercepts matmul and delegates everything else, using Kotlin interface delegation:

class ObservingTensorOps(
    private val base: TensorOps,
    private val onMatmul: (a: Tensor<*, *>, b: Tensor<*, *>, out: Tensor<*, *>) -> Unit,
) : TensorOps by base {                              (1)
    override fun <T : DType, V> matmul(a: Tensor<T, V>, b: Tensor<T, V>): Tensor<T, V> {
        val out = base.matmul(a, b)
        onMatmul(a, b, out)                          (2)
        return out
    }
}
1 Every op except matmul flows straight through to the real backend.
2 a is the activation, b the weight (or, for attention score matmuls, a second activation); out is the result. Read their values with tensor.data.copyToFloatArray() and fold into a running max|Β·|.

Wrap a context so the model’s ctx.ops.matmul(…​) calls route through the tap β€” the model itself is unchanged:

class ObservingExecutionContext(base: ExecutionContext, onMatmul: MatmulTap)
    : ExecutionContext by base {
    override val ops: TensorOps = ObservingTensorOps(base.ops, onMatmul)
}

// drive it:
val ranges = CalibrationRanges()
val ctx = ObservingExecutionContext(DirectCpuExecutionContext.create(), ranges::observe)
for (input in representativeInputs) model.forward(input, ctx)   // fp32 eager, taps every matmul
val scales = ranges.symmetricInt8Scales()                       // S = max|x| / 127 per tensor

SKaiNET also exposes the ExecutionObserver hooks (onOpStart / onOpEnd) on an ExecutionContext. Those are the natural home for this logic, but on the plain eager contexts the per-op notifyOp dispatch is not wired, so a TensorOps decorator is the reliable tap today. An observer implementation can share the same range-accumulation code.

The output is one scale triple per matmul β€” keyed by deterministic op-invocation order so the emitter can line them up with the graph:

# op_index   Sa            Sb            So            absmax_act  absmax_rhs  absmax_out
op0          0.124454215   0.003971611   0.053741008   15.805685   0.504395    6.825108
op1          0.124454215   0.004852055   0.063222170   15.805685   0.616211    8.029216
...

Sb is the second-operand scale. For the q/k/v/out/MLP projections it simply matches the static Sw; for the attention score matmuls (QΒ·Kα΅€, attnΒ·V) the second operand is also an activation, so Sb is a genuine calibrated scale β€” which is why the calibrator records it for every matmul rather than assuming operand two is a weight.

Why QuantizedMatmul is not the activation-int8 op

A natural question: can the existing QuantizedMatmul carry this? No β€” and the reason is worth stating, because it shapes where the int8 emission has to live.

  • It is weight-only and eager. QuantizedMatmul dequant-fuses Q8_0/Q4_K weight blocks against fp32 activations and returns a materialised fp32 tensor. It never records into the tape, so it produces no int8 graph for a backend to lower.

  • The tape can’t express the pattern either. TensorOps.matmul is dtype-homogeneous β€” matmul(a: Tensor<T, V>, b: Tensor<T, V>): Tensor<T, V>. There is no i8 Γ— i8 β†’ i32 accumulation form, and there is no round primitive (only clamp / convert / sign). So the quantize β†’ i8 matmul β†’ requantize pattern cannot be written at the op level today.

The consequence is a clean split of responsibilities:

  • The framework calibrates β€” a pure read-side tap over the existing fp32 eager run, fully expressible with today’s API (above).

  • The target backend emits int8 β€” the quantize / i8-matmul / requantize lowering belongs in the accelerator’s codegen, parameterised by the Sa / Sb / So the calibrator produced.

The pipeline end to end

  representative inputs
          β”‚
          β–Ό
  fp32 eager run  ──tap every matmul──▢  per-tensor max|Β·|
  (ObservingExecutionContext)                    β”‚
                                                 β–Ό
                                   symmetric scales  Sa, Sb, So  (= max|x| / 127)
                                                 β”‚
        static weights ──▢ Sw, int8 weight constants
                                                 β”‚
                                                 β–Ό
                       backend codegen:  quantize β†’ i8 matmul (i32) β†’ requantize
                                                 β”‚
                                                 β–Ό
                                     int8 model on the accelerator

Calibration is the SKaiNET-side half and is target-agnostic: the same scales drive any int8 backend.

Follow-ups

  • A tape-level int8 matmul op (i8 Γ— i8 β†’ i32) plus a round primitive would let the int8 pattern be expressed in the graph itself, rather than only in backend codegen.

  • Per-channel weight scales (a Sw per output column) and percentile / KL calibration (instead of plain max|Β·|) are the usual accuracy levers once the per-tensor path works.

  • Wiring the ExecutionObserver notifyOp dispatch on the eager contexts would let calibration use the observer hooks directly instead of a TensorOps decorator.