Best AI News โ€” Updated Every 3 Hours
Story Page
← All Stories
Home Community Story
Community

Can we reconstruct a closed-source LLM tokenizer using only two oracles from the chat API?

Via r/LocalLlama
Saturday, Jul 11, 2026 ยท 6:17AM
Summary

From the chat APIs alone, we can extract two useful oracles: 1. Token length oracle: Given any string s, return len(tokenize(s)). Prefix token oracle: Given a string s and integer n, return the string decoded from the first n tokens of tokenize(s). The first oracle (token count) is straightforward.

Continue reading the full article
Read at r/LocalLlama
www.reddit.com
Back to all stories