ihateninjas/ollama
1
0
Fork 0
mirror of https://github.com/ollama/ollama synced 2026-04-23 08:45:14 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
ollama/tokenizer
History
Michael Yang f1373193dc
move tokenizers to separate package (#13825)
2026-02-05 17:44:11 -08:00
..
testdata move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
bytepairencoding.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
bytepairencoding_test.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
sentencepiece.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
sentencepiece_test.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
tokenizer.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
vocabulary.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
vocabulary_test.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
wordpiece.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
wordpiece_test.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00