Michael-A-Kuykendall/shimmytok

github.comtool

[shimmytok] - Pure-Rust tokenizer for GGUF models, compatible with llama.cpp tokenization.

from

Machine Learning, Rust

added

2026-10-10

likes

0

Machine Learning › Rust > Natural Language Processing: “Pure-Rust tokenizer for GGUF models, compatible with llama.cpp tokenization.”

Rust › Libraries > Artificial Intelligence > Machine learning: “[shimmytok] - Pure-Rust tokenizer for GGUF models, compatible with llama.cpp tokenization.”