Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

Show HN: Quicktok, an exact BPE tokenizer 7x faster than tiktoken

**Brief summary (2‑3 sentences)** Quicktok is an open‑source BPE tokenizer that reproduces the exact output of OpenAI’s tiktoken but runs up to seven times fast

Original source