SKEEP-007: The browser lane — packed SIMD kernels and quantize-at-load

One Laya decision at 80–150 tokens is 10–19 G multiply-adds, 96 % of them in four dense matmulWeightTransposed calls per layer (Wqkv, Wi, mlp.Wo, Wo). The scalar [k, n] loop streams the whole weight once per activation row: about 75 GB of memory traffic per decision at 150 tokens. For comparison, the same kernel class written with wasm SIMD128 intrinsics runs a Llama-3.2-1B Q8_0 forward at roughly 270 GFLOPS on seven threads (llama.cpp #11453, measured in wllama), which puts a Laya decision at well under a second.

The kernel-pack seam (ViewKernelPack, KernelDispatch, KernelProvider) is how the JVM and Android got their packed kernels; a browser pack behind the same seam reaches every web build without application changes. Quantize-at-load is likewise a loader concern: today only GGUF files can be served packed, so every Hugging Face checkpoint — SafeTensors, F16 or BF16 — is dense FP32 on every target.

DefaultCpuOpsBase.matmulWeightTransposed orders the packed-weight paths by preference: an exact kernel for the canonical view (the JVM FFM, Android JNI and browser packs), then the one-time cached relayout into kernel feed order that the scalar packed kernels read (prepackedWeights was declared for this and never used in common code), then the decoding reference kernel through views. The dense branch computes x · Wᵀ straight from the weight’s rows with a 4 × 4 tile and one sequential accumulator per output, bit-identical to the transpose-then-matmul result. The JVM and Apple ops override preferDirectWeightTransposed() to keep their cached transpose, because their [k, n] kernels are vectorised.

NarrowFloatDenseTensorData (and so Fp16DenseTensorData, Bf16DenseTensorData) implements RowDequantSource; ops.gather and the transformers Embedding layer already branch on it.

skainet-io-core gains ByteArrayRandomAccessSource (single array or equal-sized chunks for files above the 2 GB array limit). JsBlobRandomAccessSource no longer logs to the console on every read; its byte-by-byte ArrayBuffer conversion stays until Tier 1 (Kotlin/Wasm 2.4 has no typed-array adapter), where the sidecar’s linear memory becomes the landing buffer for downloads.

skainet-backends/benchmarks/wasm-bench is a Kotlin/Wasm executable that times the matmul shapes of Laya, LEAF and SmolLM2 and prints [SKAINET_PERF_WASM] lines; it is the baseline every later tier is measured against.

A wasmJs-only module. Its native build compiles the existing kernel sources (skainet-backend-native-cpu/native/src/*.c, shared, not copied) with Emscripten to skainet_kernels_{mt,st}.{wasm,mjs}; skainet_simd.h gains wasm_simd128 bodies for the row-major Q8_0/Q4_0/Q4_K/Q5_K/Q6_K dot products (int8 lanes widened to int16, i32x4.dot_i16x8 accumulation — the technique of llama.cpp #11453, our code) and f32x4 bodies for the dense FP32, FP16 and BF16 kernels. skainet_row_threads.c learns the Emscripten pool size and degrades to the calling thread when the page is not cross-origin isolated. The artifacts ship as module resources and are reproducible from the pinned emsdk in CI.

On the Kotlin side a WasmRowMajorMatmulKernel : ViewKernel registers under the same keys as FfmRowMajorMatmulKernel (dense contiguous FP32 activation × Format(FP32, encoding) weight, BLOCKED_ROW_MAJOR, heap bytes), uploads a weight into sidecar memory once and copies activations per call; a WasmKernelProvider serves the dense kernels; WasmCpuOps : DefaultCpuOpsBase overrides matmul, matmulWeightTransposed, gelu and softmax while the sidecar is ready and falls back to the Tier 0 paths otherwise. Resident activations in sidecar memory (a Storage.WasmLinear kind) and a banded attention kernel follow once the copy overhead is measured.

lang-core gains Q8_0Quantizer (ggml-faithful, 32-element blocks, row-major) beside the existing Q4_0Quantizer, and a Requantizers registry from TensorEncoding to quantizer. SafeTensorsParametersLoader and the sharded loader take weightForm/weightFormFor; the materializer gains a RequantizeTo branch producing XBlockTensorData(shape, bytes, ROW_MAJOR) and honours WeightShapeOrientation and KERNEL_FEED. The GGUF loader’s RequantizeTo gate relaxes from "BITNET_PLANES only" to "any encoding with a registered quantizer". PlannerProfile gains quantizeDenseTo so a browser profile can resolve dense sources to RequantizeTo(Q8_0). skainet-quantize (in skainet-tensor-tools) writes a GGUF with the source tensor names kept, for downloads that should not carry F16.

The scalar column is the same run’s scalar pass (a quieter machine than the Tier 0 run above); st is the SIMD128 sidecar single-threaded, mt the pthreads build with four workers.

Read across one Laya layer on Q8_0: about 56 ms with pthreads, so roughly 1.2 s for the 22-layer encoder at 150 tokens — against 29 s scalar in the Tier 0 run, within the acceptance band of this proposal before the resident-activation step. The single-token dense rows (gemv, 1 GFLOPS) showed the cost of re-uploading a dense weight per call; with dense weights resident like the packed ones the same rows measure 8.6 GF (st) and 23.6 GF (mt) for smollm2.gemv and 6.6 / 23.0 GF for qwen06b.gemv. Small shapes (leaf.proj) are bound by the per-call copies and the thread hand-off, which is what resident activations address next.