DarijaBench: Do AI Models Actually Understand Moroccan Darija?

说明与用途

DarijaBench 用于评估 AI 模型对摩洛哥达里贾方言的理解。四个顶尖模型得分 0.95，翻译是最难的技能。

问题

AI assistants perform poorly on Moroccan Darija, a low‑resource dialect, despite high performance on English and French.

结果

Top models achieve 0.95 DarijaBench score, indicating strong basic understanding; translation remains hardest, sentiment easiest. Benchmark highlights gaps in cultural knowledge and code‑switching handling.

工作流

• 1. Identify need for Darija evaluation.
• 2. Design benchmark with 3 skill categories (translation, sentiment, cultural knowledge).
• 3. Create 60 items (20 per category).
• 4. Implement automatic keyword‑based grading via Kaggle Benchmarks SDK.
• 5. Run benchmark on Gemini 3.7 Flash, Claude Sonnet 5, Claude Opus 4.7, GPT‑5.6 Luna.
• 6. Analyze results and identify strengths/weaknesses.
• 7. Publish benchmark and results publicly.

证据

I built DarijaBench, a 60-item benchmark that tests whether today's frontier models truly understand Moroccan Darija — and ran it on four of them.

来源

DarijaBench tests three practical skills, 20 items each: Darija → French translation, Sentiment analysis, Moroccan cultural knowledge.

来源

The whole thing was built with the Kaggle Benchmarks SDK, entirely on the free quota.

来源

Gemini 3.7 Flash 0.95, Claude Sonnet 5 0.95, Claude Opus 4.7 0.95, GPT-5.6 Luna 0.90

来源

Sentiment is the easiest skill. Gemini scored a perfect 20/20 on sentiment.

来源

Translation is the hardest. At 85%, Darija→French was the lowest category for Gemini.

来源

Keyword-based grading is brittle — a correct answer phrased unexpectedly can fail.

来源