ztok — a fast multithreaded tokenizer in Zig that loads tiktoken / HF / SentencePiece and is 2–5× faster

I built ztok, a tokenizer library focused on being fast and format-agnostic for local pipelines. - Loads what you already have — .tiktoken, HF tokenizer.json, SentencePiece .model, TokenMonster, Mistral Tekken. Auto-detected. - Bit-identical to tiktoken / HF / SentencePiece on the equivalence gate,