SKaiNET
Toggle table of contents
0.20.0-SNAPSHOT
common
Target filter
common
Switch theme
Search in API
Skip to content
SKaiNET
skainet-io-core
/
sk.ainet.io.tokenizer
/
TekkenTokenizer
/
totalTokens
total
Tokens
val
totalTokens
:
Int
(
source
)
Total token count (vocab + special tokens).