DarijaBench: Do AI Models Actually Understand Moroccan Darija?

Описание и применение

Создана бенчмарка DarijaBench для оценки понимания марокканского диалекта дариджи моделями ИИ. Четыре ведущие модели набрали 0,95 балла, при этом перевод на французский оказался самым сложным навыком.

Задача

AI assistants perform poorly on Moroccan Darija, a low‑resource dialect, despite high performance on English and French.

Результат

Top models achieve 0.95 DarijaBench score, indicating strong basic understanding; translation remains hardest, sentiment easiest. Benchmark highlights gaps in cultural knowledge and code‑switching handling.

Рабочий процесс

• 1. Identify need for Darija evaluation.
• 2. Design benchmark with 3 skill categories (translation, sentiment, cultural knowledge).
• 3. Create 60 items (20 per category).
• 4. Implement automatic keyword‑based grading via Kaggle Benchmarks SDK.
• 5. Run benchmark on Gemini 3.7 Flash, Claude Sonnet 5, Claude Opus 4.7, GPT‑5.6 Luna.
• 6. Analyze results and identify strengths/weaknesses.
• 7. Publish benchmark and results publicly.

Доказательства

I built DarijaBench, a 60-item benchmark that tests whether today's frontier models truly understand Moroccan Darija — and ran it on four of them.

Источник

DarijaBench tests three practical skills, 20 items each: Darija → French translation, Sentiment analysis, Moroccan cultural knowledge.

Источник

The whole thing was built with the Kaggle Benchmarks SDK, entirely on the free quota.

Источник

Gemini 3.7 Flash 0.95, Claude Sonnet 5 0.95, Claude Opus 4.7 0.95, GPT-5.6 Luna 0.90

Источник

Sentiment is the easiest skill. Gemini scored a perfect 20/20 on sentiment.

Источник

Translation is the hardest. At 85%, Darija→French was the lowest category for Gemini.

Источник

Keyword-based grading is brittle — a correct answer phrased unexpectedly can fail.

Источник