skainet-io-core/sk.ainet.io.tokenizer/SentencePieceTokenizer/Companion

Companion

object Companion(source)

Properties

WHITESPACE_ESCAPE

const val WHITESPACE_ESCAPE: Char = '\u2581'

SentencePiece whitespace-escape character: ▁ (U+2581).

Functions

fun fromGgufFields(fields: Map<String, Any?>): SentencePieceTokenizer

Build from GGUF metadata fields (see GgufModelMetadata.rawFields).

fromTokenizerJson

fun fromTokenizerJson(root: JsonObject): SentencePieceTokenizer

Build from a parsed HuggingFace tokenizer.json root object where model.type == "Unigram".