Do AI models understand daily life in Pakistan? lakh/crore, PKT and Urdu tested
说明与用途
测试AI模型是否理解巴基斯坦特定知识,包括 lakh/crore 系统、PKT、地理和乌尔都语脚本。
问题
Assess whether LLMs grasp Pakistan‑specific concepts such as the lakh/crore number system, Pakistan Standard Time, local geography, and Urdu script.
结果
The benchmark provides reproducible scores for LLMs on Pakistan‑specific tasks, highlighting strengths and weaknesses in numeric, temporal, geographic, and linguistic domains.
工作流
- 1. Define four categories of local knowledge.
- 2. Create fixed‑answer questions with regex‑based scoring.
- 3. Implement dataset in pandas and kaggle_benchmarks.
- 4. Run models, collect only final answers, strip formatting.
- 5. Evaluate results automatically.
- 6. Publish benchmark for community use.
证据
I built the Pakistan Local Context LLM Suite, a set of short questions with fixed, checkable answers in four categories:
Scoring is done by code (regex matching on a normalized response), with no judge model, so every result is reproducible.
- 记录日期: 2026-10-11T10:56:16.614Z