voidful
/

PangolinTokenizer

traditional-chinese

Model card Files Files and versions

PangolinTokenizer

Commit History

Make tokenizer config compatible with Transformers 4.x

8953d03
verified

voidful commited on 2 days ago

Add Open Formosa special tokens

989d8ec
verified

voidful commited on 2 days ago

Upload Pangolin tokenizer

82e506e
verified

voidful commited on 2 days ago

initial commit

aaed96a
verified

voidful commited on 2 days ago