SKaiNET
Toggle table of contents
0.20.0-SNAPSHOT
common
Target filter
common
Switch theme
Search in API
Skip to content
SKaiNET
skainet-io-core
/
sk.ainet.io.tokenizer
/
QwenByteLevelBpeTokenizer
/
QwenByteLevelBpeTokenizer
Qwen
Byte
Level
Bpe
Tokenizer
constructor
(
tokens
:
List
<
String
>
,
merges
:
List
<
Pair
<
String
,
String
>
>
,
specialTokens
:
Map
<
String
,
Int
>
,
bosTokenId
:
Int
?
=
null
,
eosTokenId
:
Int
?
=
null
)
(
source
)