DarijaBench: Do AI Models Actually Understand Moroccan Darija?

Description and use

DarijaBench was created to measure AI models’ understanding of Moroccan Darija. Four leading models scored 0.95, with translation being the hardest skill.

Problem

AI assistants perform poorly on Moroccan Darija, a low‑resource dialect, despite high performance on English and French.

Outcome

Top models achieve 0.95 DarijaBench score, indicating strong basic understanding; translation remains hardest, sentiment easiest. Benchmark highlights gaps in cultural knowledge and code‑switching handling.

Workflow

• 1. Identify need for Darija evaluation.
• 2. Design benchmark with 3 skill categories (translation, sentiment, cultural knowledge).
• 3. Create 60 items (20 per category).
• 4. Implement automatic keyword‑based grading via Kaggle Benchmarks SDK.
• 5. Run benchmark on Gemini 3.7 Flash, Claude Sonnet 5, Claude Opus 4.7, GPT‑5.6 Luna.
• 6. Analyze results and identify strengths/weaknesses.
• 7. Publish benchmark and results publicly.

Evidence

I built DarijaBench, a 60-item benchmark that tests whether today's frontier models truly understand Moroccan Darija — and ran it on four of them.

Source

DarijaBench tests three practical skills, 20 items each: Darija → French translation, Sentiment analysis, Moroccan cultural knowledge.

Source

The whole thing was built with the Kaggle Benchmarks SDK, entirely on the free quota.

Source

Gemini 3.7 Flash 0.95, Claude Sonnet 5 0.95, Claude Opus 4.7 0.95, GPT-5.6 Luna 0.90

Source

Sentiment is the easiest skill. Gemini scored a perfect 20/20 on sentiment.

Source

Translation is the hardest. At 85%, Darija→French was the lowest category for Gemini.

Source

Keyword-based grading is brittle — a correct answer phrased unexpectedly can fail.

Source