Automating Data Quality Checks in GitHub Actions with DataGuard AI

Описание и применение

Автоматизация проверок качества данных в GitHub Actions с помощью DataGuard AI позволяет обнаруживать ошибки в данных до их попадания в отчёты, генерируя отчёты в JSON и HTML, которые можно просматривать в артефактах workflow.

Задача

Data quality issues can slip through ETL pipelines and affect downstream reports before detection.

Результат

Automated scans run on every push or pull request to main, producing reports that expose missing emails, null values, and out‑of‑range ages. The workflow does not fail automatically on findings; a separate policy step is needed to enforce quality gates.

Рабочий процесс

• Install DataGuard AI (python -m pip install dataguard-ai==0.2.1)
• Prepare sample data (customers.csv) and configuration (dataguard.yml)
• Run local scan (dataguard scan customers.csv --config dataguard.yml --json-out report.json --html-out report.html)
• Create GitHub Actions workflow (.github/workflows/data-quality.yml) that checks out code, sets up Python, installs DataGuard AI, runs the scan, and uploads reports as artifacts

Доказательства

DataGuard AI will inspect the dataset and write two reports: report.json — structured findings suitable for automation; report.html — a human-readable report for reviewing findings

Источник

Important limitation: DataGuard AI v0.2.1 generates findings, but the workflow above does not automatically fail merely because a quality issue is detected.

Источник

Create a file called customers.csv: ... This intentionally includes a missing email address and an invalid negative age.

Источник