A RAG Pre-Production Checklist with Real Configuration

Описание и применение

Пошаговый чеклист для подготовки RAG к выпуску, включающий создание золотого набора данных, детерминированные проверки, настройку метрик и интеграцию с CI.

Задача

RAG launch checklists often lack concrete, actionable steps tied to actual settings, making it hard to implement and monitor retrieval and generation quality.

Результат

Teams can now gate RAG releases with a 20–50 row golden dataset, deterministic rule checks, and metric thresholds, reducing bugs and ensuring consistent quality before production.

Рабочий процесс

• Build a small golden dataset with canonical inputs, expected outputs, metadata, and a one-sentence reason.
• Add deterministic rule checks (Non-Empty, JSON Schema, Contains, Length Check, Regex Match) as a structural floor.
• Set metric thresholds for Answer Precision (0.8), Faithfulness (0.7), Contextual Recall (0.7) and adjust as needed.
• Create and configure a connector in EvaliQA, mapping actual output, retrieval context, and other paths.
• Run the test plan from CI by calling the EvaliQA API with the test_plan_id, dataset_id, and connector_id, handling error codes and interpreting results before merging.

Доказательства

Build a small golden dataset first

Источник

Add deterministic checks as a structural floor

Источник

Set metric thresholds deliberately

Источник

Connect your RAG endpoint and map the output

Источник

Run the plan from CI

Источник