Been working on this a while! Should be useful for anyone trying to speed up their tokenization workflows. quicktok is a fast/exact BPE tokenizer written in C++. Token ids are byte-identical to tiktoken and encoding runs 2–3.6× faster than bpe-openai (the fastest alternative I know of) and 4–11× fas
