skainet-io-core/sk.ainet.io.tokenizer/QwenByteLevelBpeTokenizer/Companion/fromGgufFields

fromGgufFields

fun fromGgufFields(fields: Map<String, Any?>): QwenByteLevelBpeTokenizer(source)

Build from GGUF metadata fields (see GgufModelMetadata.rawFields).

Treats every token whose token_type code is 3 (control/special) as an atomic special token. Merges arrive as space-separated "first second" strings in GGUF.