fromGgufFields

Build from GGUF metadata fields (see GgufModelMetadata.rawFields).

Required keys:

• tokenizer.ggml.tokens — list of vocab strings

• tokenizer.ggml.scores — list of floats, same length

Optional keys:

• tokenizer.ggml.token_type — used only to flag the unknown token; type 2 means UNKNOWN.

• tokenizer.ggml.bos_token_id, tokenizer.ggml.eos_token_id, tokenizer.ggml.unknown_token_id

• tokenizer.ggml.add_space_prefix (bool, default true)