The problem with standard tokenizers

Standard tokenizers break on Roman Urdu due to inconsistent spelling and code‑mixing. A DFA tokenizer was built that normalizes input, classifies tokens, and vi