skainet-io-core/sk.ainet.io.tokenizer

Package-level declarations

Types

class QwenByteLevelBpeTokenizer(tokens: List<String>, merges: List<Pair<String, String>>, specialTokens: Map<String, Int>, val bosTokenId: Int? = null, val eosTokenId: Int? = null) : Tokenizer

GPT-2-style byte-level BPE tokenizer (Qwen, GPT-2, Mistral-Nemo, …).

SentencePieceTokenizer

class SentencePieceTokenizer(tokens: List<String>, scores: List<Float>, val unknownTokenId: Int? = null, val bosTokenId: Int? = null, val eosTokenId: Int? = null, val addSpacePrefix: Boolean = true) : Tokenizer

SentencePiece tokenizer for LLaMA, Gemma, TinyLlama, Mistral-v0.1 and other models whose GGUF tokenizer.ggml.model is "llama" and whose HuggingFace tokenizer.json has model.type == "Unigram".

TekkenTokenizer

class TekkenTokenizer(vocabTokenBytes: List<ByteArray>, vocabTokenStrings: List<String?>, specialTokens: Map<String, Int>, specialTokensById: Map<Int, String>, numSpecialTokens: Int = 1000, pattern: Regex) : Tokenizer

Mistral Tekken tokenizer implementation.

Tokenizer

interface Tokenizer

Common surface for all tokenizer implementations.

TokenizerFactory

object TokenizerFactory

Selects the right Tokenizer implementation for a model.

UnsupportedTokenizerException

class UnsupportedTokenizerException(message: String) : RuntimeException