Do AI models understand daily life in Pakistan? lakh/crore, PKT and Urdu tested
Описание и применение
Проверка, понимают ли модели ИИ специфические для Пакистана знания, включая систему lakh/crore, часовой пояс PKT, географию и урду.
Задача
Assess whether LLMs grasp Pakistan‑specific concepts such as the lakh/crore number system, Pakistan Standard Time, local geography, and Urdu script.
Результат
The benchmark provides reproducible scores for LLMs on Pakistan‑specific tasks, highlighting strengths and weaknesses in numeric, temporal, geographic, and linguistic domains.
Рабочий процесс
- 1. Define four categories of local knowledge.
- 2. Create fixed‑answer questions with regex‑based scoring.
- 3. Implement dataset in pandas and kaggle_benchmarks.
- 4. Run models, collect only final answers, strip formatting.
- 5. Evaluate results automatically.
- 6. Publish benchmark for community use.
Доказательства
I built the Pakistan Local Context LLM Suite, a set of short questions with fixed, checkable answers in four categories:
Scoring is done by code (regex matching on a normalized response), with no judge model, so every result is reproducible.
- Дата записи: 2026-10-11T10:56:16.614Z