skainet-io-core/sk.ainet.io.tokenizer/QwenByteLevelBpeTokenizer/Companion

Companion

object Companion(source)

Functions

fun fromGgufFields(fields: Map<String, Any?>): QwenByteLevelBpeTokenizer

Build from GGUF metadata fields (see GgufModelMetadata.rawFields).

fromTokenizerJson

fun fromTokenizerJson(root: JsonObject): QwenByteLevelBpeTokenizer

Build from a parsed HuggingFace tokenizer.json root object.