Mixed Precision: Why a Layer’s Dtype Is Worth Choosing

A network built with the SKaiNET DSL has one dtype: sequential<FP32, Float> { … } fixes it for every layer. Mixed precision (SKEEP-008, tracked in #1433) loosens that — one layer or one block can run in another dtype while the network around it stays unchanged:

val model = sequential<FP32, Float> {
    input(784)
    dense(256, dtype = BF16) { }     // one BF16 layer in an FP32 network
    activation { it.relu() }
    stage("head", dtype = FP16) {    // a whole FP16 block
        dense(64) { }
        dense(10)
    }
}

This page explains why that is worth having, what it costs, and where the current implementation stops.

Precision is not uniformly needed

The observation behind every mixed-precision system — PyTorch AMP, JAX’s dtype promotion, ONNX Runtime’s float16 transforms — is that a network’s tolerance for rounding is wildly uneven. The matmul-heavy bulk of a model (dense and convolution layers) tolerates 16-bit floats well: their error statistics average out over thousands of multiply-accumulates. A few places do not: softmax and normalization (small differences of large numbers), accumulators, and the loss. Forcing the whole network to the precision its most sensitive operation needs wastes memory and bandwidth on the 95% that doesn’t care.

Per-layer dense(dtype = …) and per-block stage(id, dtype = …) are exactly that dial: keep the skeleton FP32, drop the bulk into a cheaper dtype, and keep the choice local — a narrow layer is opt-in per call, never a network-wide default.

What the two 16-bit floats trade away

FP16 BF16

Layout

1 sign, 5 exponent, 10 mantissa bits

1 sign, 8 exponent, 7 mantissa bits

Range

overflows past 65 504

FP32’s full range

Precision

~3 decimal digits

~2 decimal digits

Failure mode

overflow to Inf in activations with large magnitudes

coarse rounding; sums of many small values drift

BF16 keeps FP32’s exponent, so anything that fits in FP32 fits in BF16 — it trades mantissa for safety and is usually the first thing to try. FP16 is more precise per bit but needs the values to stay small. The convert codecs handle overflow (to infinity) and NaN correctly, but a model can still silently degrade — which is why the dtype appears in the code at the call site, not in a config file.

Why it matters for SKaiNET’s targets

SKaiNET runs on Kotlin Multiplatform targets where the usual mixed-precision arguments are amplified:

  • Memory-constrained hosts. A browser tab (wasm), an Android app, an iOS app — environments where halving or quartering a model’s working set decides whether it runs at all.

  • Bandwidth-bound inference. Most inference time is spent moving weights, not multiplying them. Fewer bytes per element means fewer bytes through the cache hierarchy per token.

  • Models that arrive mixed. GGUF and ONNX models in the wild already mix precisions — quantized matmul weights next to FP32 norms. A DSL that can only express one dtype has no faithful home for them.

Today the memory and bandwidth benefits are not yet realized: FP16 and BF16 tensors are float-backed (the values are rounded to the 16-bit format, the storage is still 4 bytes per element). Real 2-byte storage belongs to the storage model (SKEEP-003). What mixed precision gives you now is the numerics — a faithful simulation of the narrow format, the DSL surface, and exports that carry the casts — so models can be validated against 16-bit behaviour before the storage catches up, and the API doesn’t change when it does.

The design: one visible boundary

The DSL wraps a mixed-precision layer or stage in a single module, PrecisionBoundary. Its forward pass is two casts around the wrapped module:

val x = ctx.ops.convert(input, innerType)    // cast in
ctx.ops.convert(inner.forward(x, ctx), outerType)  // cast back out

Three properties follow from using TensorOps.convert — an ordinary tensor op — rather than hiding the conversion inside the layer:

  • Free when degenerate. A tensor that already has the target dtype is passed through without a convert call, so a boundary between equal dtypes costs nothing and leaves no cast in a trace or an export.

  • Visible in traces and exports. The tracing wrappers record convert like any other op, and the StableHLO converter lowers it to stablehlo.convert — a compiled or exported graph shows exactly where precision changes, and CastFoldingPass folds the cast pair between adjacent boundaries.

  • Honest about cost. Every boundary copies its input and its output. A network that switches dtype every layer can be slower than all-FP32, especially while narrow dtypes are float-backed. Few, coarse boundaries (a stage around a block) beat many fine ones.

The dtype is passed as a value (dense(256, dtype = BF16)), not as a type argument. The earlier API sketched dense<BF16>(256), but an interface method’s type parameter cannot be reified, so the implementation had no KClass to build the layer’s tensors with — those overloads could never be implemented and have been removed. Passing the DType object gives the implementation both the instance (for convert) and dtype::class (for tensor creation).

What an export looks like

Tracing a network and exporting it to StableHLO gives one stablehlo.convert pair per boundary, with everything between them typed in the narrow format. For the network above without its relu, so that the BF16 layer is directly followed by the FP16 stage, the module has this shape (abridged: constants, transposes and bias adds are left out):

%v6  = stablehlo.convert %arg0 : (tensor<1x784xf32>) -> tensor<1x784xbf16>   // into the BF16 dense
%v10 = stablehlo.dot_general %v6, %v7, ... -> tensor<1x256xbf16>
%v13 = stablehlo.convert %v12 : (tensor<1x256xbf16>) -> tensor<1x256xf32>    // and back out
%v14 = stablehlo.convert %v13 : (tensor<1x256xf32>) -> tensor<1x256xf16>     // into the FP16 stage
%v15 = stablehlo.dot_general %v14, %v8, ... -> tensor<1x64xf16>
%v18 = stablehlo.dot_general %v17, %v9, ... -> tensor<1x10xf16>
%v21 = stablehlo.convert %v20 : (tensor<1x10xf16>) -> tensor<1x10xf32>       // and back out

The bf16 → f32 cast followed directly by f32 → f16 is what two adjacent boundaries with different inner dtypes record: each one restores the network’s dtype, so the tensor is widened only to be narrowed again. MixedPrecisionExportTest pins this output.

CastFoldingPass, part of the default optimization pipelines, removes that round trip. A widening from FP16 or BF16 to FP32 is exact, so narrowing the widened tensor gives the same values as narrowing the original, and the two casts fold into one:

%v13 = stablehlo.convert %v12 : (tensor<1x256xbf16>) -> tensor<1x256xf16>    // BF16 dense straight into the FP16 stage

Between two boundaries of the same inner dtype nothing is left at all. The pass never touches the other order, f32 → bf16 → f32: that pair is the rounding the layer asked for.

A declared dtype is a checked preference

Tracing marks every op that runs inside a boundary with the dtype the code declared for it: the graph node carries DTypePolicy.Prefer(dtype) and the boundary’s name as metadata. The boundary’s own two casts are not marked, since their job is to have a different dtype on one side.

DTypeConstraintResolutionPass, the first pass of the default pipelines, checks that preference against the node’s floating-point inputs. Nothing is reported for an untouched network. If a later pass retypes the region — DtypeForwardPropagationPass(targetFloatDtype = "BF16") coerces a whole graph to one float dtype, which is what some NPU targets need — a second run of the constraint pass says so:

Node 'n17_matmul' (matmul) in precision boundary 'head' prefers Float16 but input 0 has dtype 'BF16' — using the existing dtype.

It is a diagnostic, not an error: retyping a mixed-precision model for a single-dtype target is a legitimate choice, and it should be a visible one.

Training through a boundary

convert is differentiable between floating-point dtypes, so a mixed-precision layer or stage trains like any other. Its backward rule treats the cast as the identity and casts the gradient back to the source dtype:

  • where the gradient leaves a narrow region (the cast back out, seen from the loss side first) it is rounded to the narrow format, exactly as the forward values were on the way in;

  • where it enters the outer network again it is widened, which is exact.

The weights behind a boundary are stored in the narrow dtype, their gradients have that dtype too, and the optimizer updates them in place. That is a faithful simulation of training in the narrow format, with its limits: an update smaller than half a unit in the last place of a weight is lost. Keeping FP32 master weights and scaling the loss to avoid that is automatic mixed precision, which is out of scope (see below). A cast from or to an integer dtype has no gradient.

One thing to know when checking gradients by hand: input(n) clears requiresGrad on the tensor it receives unless asked not to (input(n, requiresGrad = true)). With the flag cleared, the first cast has nothing to propagate and the network input gets no gradient; the parameters are not affected.

What it does not do (yet)

  • Automatic mixed precision. There is no autocast policy, no FP32 master weights, no loss scaling. The dtype is always an explicit, local choice in the code.

  • Integer layers. A cast to Int8 truncates; it is not quantization. Integer inference goes through packed encodings and the quantized kernels — see The quantization process. Mixed precision and quantization are complementary, not competing: one changes the float format a layer computes in, the other changes the encoding weights are stored and multiplied in.

Where to look

  • sk.ainet.lang.nn.PrecisionBoundary — the boundary module.

  • DTypeConstraintResolutionPass and PRECISION_BOUNDARY_METADATA_KEY — the declared dtype as graph metadata, and the pass that checks it.

  • NeuralNetworkDsl.dense(outputDimension, dtype, …) and stage(id, dtype, …) — the DSL entry points.

  • TensorOps.convert — the cast, including FP16/BF16 rounding through the format codecs.

  • SKEEP-008 — the full design, including the storage-side repackTo/requantizeTo split and the phased rollout.