aicostbudget/tokenizer-ratios

How many characters is a token? 951 samples measured on OpenAI o200k_base (GPT-4o): English, code, JSON, CSV, IDs, emoji, Chinese. chars/4 misses JSON by half.