Tokenizer Internals

One stack, in the engine

Until 0.57 this repository carried a second tokenizer stack next to the engine’s: a legacy GGUFTokenizer that scraped tokenizer.json with regular expressions and merged greedily by score, four preprocessing strategies, a wrapper factory and two adapters. Measured against the Hugging Face tokenizers library the legacy path returned wrong ids on most inputs of the public ModernBERT and mmBERT files without raising an error (SKaiNET-transformers #471).

Since 0.58 the engine owns every algorithm (skainet-frontend-text, SKEEP-006): byte-level BPE, Metaspace BPE with byte fallback, SentencePiece, Unigram (Viterbi), WordPiece and Tekken, each guarded by a golden corpus recorded with the reference library. The engine also owns the resolution: which side file selects which algorithm, how added_tokens are matched atomically, how the normalizer and pre_tokenizer blocks apply, how tokenizer_config.json contributes bos/eos names and the chat template.

What this repository owns

  • Family policy. TransformersFamilies describes each family: architecture strings, stop markers, tool-calling support, chat-template family. A descriptor implements the engine’s FamilyFrontEnds and adds its end-of-turn ids to SpecialTokens.extraStop, resolved through the checkpoint’s own vocabulary — never hard-coded ids.

  • Chat templates (llm-agent), which render strings the tokenizer then encodes; the template source shipped with a checkpoint is available as TextFrontEnd.chatTemplateSource.

  • The Tokenizer sub-interface with the -1 sentinels and the decode(Int) spelling the runtimes still use, and TokenizerFactory as the bridge.

Special tokens

Every added_tokens entry is matched atomically before pre-tokenization, special or not; GGUF CONTROL and USER_DEFINED token types are treated the same way. The special flag decides what decode(ids, skipSpecialTokens = true) drops. encode(text) never adds bos/eos (the 0.57 behaviour the runtimes rely on); encode(text, addSpecialTokens = true) applies the checkpoint’s add_bos_token / add_eos_token policy.

Why the legacy class is deprecated, not fixed

Forwarding GGUFTokenizer to the engine silently would change token ids under consumers that changed no code; a loud deprecation is better than a quiet behaviour change. The class keeps its behaviour at WARNING level in 0.58 and 0.59, ERROR in 0.60, and is removed in 0.61. Goldens recorded against it must be re-recorded from the reference library before a call site migrates.