Compile a Model for Android (DSL → StableHLO → IREE)

Run the export task (or the equivalent for your own harness):

This traces the DSL model (LlamaNetworkLoader.fromWeights), strips per-layer KV caches (a single fixed-seq forward pass needs none — see the harness’s own doc comment for why KVCache.update() isn’t traceable), appends the DSL’s in-graph argMax so the compiled function returns small token ids instead of a full [seq, vocab] logits tensor, and emits two files: <model>-gen.mlir (the StableHLO graph) and <model>.safetensors (every weight, externalized — not baked into the graph).

The exported function follows a fixed contract: tensor<1xSEQxi32> → tensor<SEQxi32> — token ids in, one predicted next-token id per position out. llm-runtime/iree-android’s runtime is written against exactly this contract; if your own harness’s export doesn’t match it, `IreeRedecodeDecoder won’t be able to drive it — see IREE Android Runtime API for the contract in full.

This is architecture-agnostic tooling, not SKaiNET-specific — .irpa is IREE’s own parameter-archive format. One .irpa serves every ABI and every compile target (CPU or GPU) — it’s just tensor bytes, not machine code.

Unlike the .irpa, the .vmfb is architecture-specific machine code — you need a separate compile per Android ABI you ship:

Architecture, yes — core count, no. The .vmfb never encodes how many cores it will run on: iree-compile decides tiling and workgroups from shapes and the target ISA, and the device maps those workgroups onto cores at run time (Step 5). The skainet.schedule attribute the export harnesses put into the module header states which axes are independent and is advisory.

--target arm64 compiles for aarch64-linux-android29, cortex-a76, with dotprod; --target arm32 compiles for armv7a-linux-androideabi29, cortex-a55, NEON only. Both are defined in the toolchain’s compile-cpu subcommand — see docker/compiler/entrypoint.sh in SKaiNET-iree-toolchain if you need a different CPU target than either default.

If you’re targeting the redecode contract from Step 1, you don’t need to build anything here — llm-runtime:iree-android already ships pre-built `.so`s for both ABIs (checked in, not built per-model). Add it as a dependency:

You only need to touch llm-runtime/iree-android/native/ yourself if you’re changing the runtime’s contract (e.g. supporting the two-graph KV-cache pattern — see Eager vs. Compiled on Android). In that case, native/build-iree-redecode.sh <abi> [--vulkan] cross-builds via the same toolchain image; the module’s own README documents the --link flags required (external-weight vmfbs need three IREE targets beyond the runtime default — iree_modules_io_parameters_parameters, iree_io_parameter_index, iree_io_parameter_index_provider, iree_io_formats_irpa_irpa — because the weights are bound from a .irpa at session-create time, not baked into the vmfb).

Cores are chosen here, not in Step 3: IreeRedecodeDecoder.fromAssets(…​, taskTopologyGroupCount = n) — or the SKAINET_TASK_GROUPS environment variable it defaults to — sets the local-task worker group count when the device is created; leave it null for IREE’s own topology detection.

Copy the .vmfb(s) and .irpa into your app’s assets:

Then drive it:

fromAssets copies both files to filesDir on first run (IREE needs real file paths, not asset streams) and creates the native session — call it off the main thread.

If you’re exporting a model other than SmolLM2, don’t start from a blank file — copy SmolLm2ExportHarness’s shape (`llm-inference/smollm2/src/jvmMain/). The parts that are genuinely model-specific: which loader builds the DSL graph (LlamaNetworkLoader.fromWeights for llama-family checkpoints — substitute the right one for your architecture), and the checkpoint path/seq passed in. The parts that are architecture-agnostic and should be copied close to verbatim: stripping KVCache before tracing, appending ectx.ops.argMax(logits, dim = -1)
ectx.ops.squeeze(idx, 0), ConstantMaterializationPolicy.ExternalAlways, and the bf16 safetensors writer.