tokenizerTokenTypes
Per-token type codes from tokenizer.ggml.token_type. GGUF convention: 1 = normal, 2 = unknown, 3 = control/special, 4 = user-defined, 5 = unused, 6 = byte.
Per-token type codes from tokenizer.ggml.token_type. GGUF convention: 1 = normal, 2 = unknown, 3 = control/special, 4 = user-defined, 5 = unused, 6 = byte.