Architecture

Become a full numerics library. SKaiNET targets the operators real models use, not the long tail in PyTorch / NumPy.

Run untrusted user code. Kernels are trusted code; security is about not corrupting memory, not about sandboxing.

DSL layer — Kotlin model DSL (nn { …​ }, tensor { …​ }) and imperative TensorOps / ExecutionContext API.

I/O layer — model loaders for GGUF, SafeTensors, ONNX in skainet-io-* modules. Reading is the common case, but skainet-io-gguf also writes: GGUFWriter/GgufExportFacade for arbitrary tensor export, and I2sAotConverter for the ternary AOT conversion in Ternary networks: getting started. The IREE-facing counterpart (GGUF → .irpa) lives outside this repo, in the public SKaiNET-IREE-tools — a deliberate boundary (§9): IREE/StableHLO-target-specific conversion does not belong in core, which stays target-neutral.

Compile layer — RecordingExecution records ops to a tape, then lowers to StableHLO / IREE bytecode in skainet-compile-* modules.

Backend layer — BackendProvider dispatches TensorOps calls to a concrete implementation (CPU, XNNPACK, future GPU). Inside a backend, the kernel SPI picks the SIMD recipe for the host hardware.

Eager execution — DirectCpuExecutionContext calls op implementations as the user invokes them. Used during development, testing, and on-device inference paths where AOT compilation is impractical (debug builds, dynamic graphs). This is the path the 0.21.0 SIMD work targets.

Recorded execution — RecordingExecution builds an op tape, which HloGenerator lowers to StableHLO MLIR. IREE compiles the MLIR to a portable bytecode for production deployment.

Lazy provider resolution. The kernel properties on the op set are by lazy; first access triggers KernelServiceLoader.installAll() when the registry is empty, then caches. Apps that pre-register providers via KernelRegistry.register(…​) bypass auto-discovery. Kotlin/Native has no ServiceLoader, so it registers manually with installNativeKernels().

Fall-through everywhere. Each routing decision returns null on a miss rather than throwing, so a new tensor type or SPI accessor is purely additive.

Rank is normalized once. A rank-1 decode step becomes [1, k] as a view before any kernel sees it, which is why a kernel written for rank 2 never meets a rank-1 tensor.

Adapters are visible. A gather, a dequantization, a requantization for a ternary kernel or a block relayout allocates in the caller’s scope and emits TraceEvent.AdapterInserted with its byte count. The costs that used to hide inside kernels are events.

Maven Central — every module published as sk.ainet.core:<module>-<target>:<version>. KMP variants land per target (-jvm, -android, -iosarm64, -macosarm64, -linuxx64, -linuxarm64, -js, -wasm-js, *-wasm-wasi).

Single BOM — sk.ainet:skainet-bom provides a platform() import for downstream Gradle. Note the group is sk.ainet, not sk.ainet.core, so downstream BOMs (e.g. sk.ainet.transformers:skainet-transformers-bom) can import it under the standard umbrella group.

Releases — tags 0.X.Y on the release branch trigger .github/workflows/publish.yml → ./gradlew publish on macOS-latest with JDK 25.

Numerical parity testing. Every accelerated kernel has a parity test against a scalar reference within a documented tolerance (typically 1e-5 * k or 1e-4 relative). Examples: PanamaVectorMatmulKernelTest, PanamaVectorQ4KMatmulKernelTest, Q6KMatmulTest. The scalar reference is the contract; SIMD speed is a non-functional improvement that must not break the contract.

Lazy resource lifetimes. MemorySegmentTensorDataFactory uses Arena.ofAuto() for per-op outputs so output segments are GC-reclaimable. The earlier Arena.ofConfined() builds leaked ~tens of MB per matmul, blowing 32+ GiB of direct memory in inference loops over a 35-layer Gemma 4 forward pass. Fixed in PR #556.

Kill switches via system properties. The Vector API code path respects -Dskainet.cpu.vector.enabled=false so a deployment can opt out of incubator code without a recompile. Same pattern for BLAS (-Dskainet.cpu.blas.enabled=true).

Scoped memory. Weights and the KV cache live in a ModelScope; one step’s activations live in a pre-sized ForwardScope slab that is reset between steps. Allocation and free events carry the TensorId they back, so a plan can be compared with what a run actually did. See The memory model.

Every conversion is an event. A gather, a dequantization, a requantization or a block relayout allocates in the caller’s scope and emits AdapterInserted. A conversion that does not appear in the trace is a bug — this is what turned the "hidden allocations" class of issue into something a test can assert.

One block-order contract. Packed weights exist in exactly two block orders, carried on the Layout and declared in each kernel’s KernelKey. Mixing them yields plausible wrong numbers rather than a crash, so the contract is normative and published: Packed weight layout.

Observability is one stream. Phases, kernel runs, adapters, allocations, scope resets, counters and the memory plan go to a TraceSink; the debugger, the Perfetto/JFR/android.os.Trace exporters, the derived generation metrics and the benchmark JSON are all consumers of it.

Performance. Panama FP32 matmul ≥1.5× scalar (M5 metric — met, ~10× at 1024² on Apple Silicon NEON). Native Q4_K matmul ≥2.5× scalar dequant baseline (M5 metric — deferred to native FFM PR).

Numerical equivalence. Every SIMD kernel matches its scalar reference within FP-rounding tolerance (1e-5 * k for matmul, 1e-4 relative for quantized). Pinned by parity tests.

Multi-target buildability. ./gradlew allTests (the release gate) must pass on every KMP target — JVM, JS, Wasm, macosArm64, iosSimulatorArm64, linuxX64, linuxArm64, Android.

Flat memory during generation. A decode step allocates nothing in the forward scope after warm-up, and the resident set does not grow with the number of steps. Asserted on every commit against a synthetic decode harness, and measured on an ARMv8.2 Cortex-A55 reference board: zero major page faults across a run, identical RSS after 4 and 48 steps.

Plan versus reality. The header-derived memory plan matches what a run actually allocated within 10 %, checked in CI rather than by inspection.

Bit-identical decode across refactors. A golden gate pins the decoded bytes of every packed encoding and the output of every scalar kernel, on JVM and Kotlin/Native. Any change to a decoder, a kernel or a byte layout has to move a recorded digest deliberately.

Vector API still incubator on JDK 25. JEP 508 keeps it that way through 2026; we depend on it heavily. If it breaks API in a future JDK, every JvmVectorKernels / JvmQuantizedVectorKernels file needs adjustment. Mitigation: thin wrappers, parity tests are a canary.

Whole-file mapping only. A file larger than 2 GB is refused rather than mapped in windows.

Ternary GGUFs still repack on every load unless converted ahead of time. I2sRepack transparently repacks BitNet.cpp’s grouped I2_S layouts (GROUP_128/GROUP_64) into BITNET_B1_58 order on each load — correct, but a real per-load cost for a repeatedly-loaded model. I2sAotConverter (§5.5) removes it for anyone who converts once; the automatic fallback sidecar cache for callers who don’t (#1198’s original proposal, tracked as a standalone follow-up, #1204) remains open and is intentionally low priority — it isn’t blocking, since the AOT path already covers the shipped-app case.

Whole-model plans only mapped-serve seven GGML formats plus dense FP32. StorageCapabilities.MAPPED_SERVABLE_DEFAULT — now cross-checked against kernel registrations (#1193, §5.3/§9) — still names a fixed set; a new packed encoding needs a MappedCapableKernel and an update to that constant before it can be served from a mapping. The guard test makes the two drift-safe, not automatic.

Hand-written NEON is a modest win over the compiler. On a Cortex-A55 the hand-written bitnet_gemv beats -O3 -ffast-math C by 1.08–1.24×; the large win is being native at all. Worth knowing before investing in more intrinsics on that core.

Two reverted optimizations on develop history. MemSeg pool (commit 8642b322) and intra-op matmul parallelism (commit 9ed633b6) were both tried and reverted. Re-attempts need a different strategy than what was tried; the revert commits explain why.