I Broke My Own GitHub Trending Scraper 5 Ways on Purpose. All 5 Builds Stayed Green.

Описание и применение

Автоматический рецепт проверки целостности данных в CI‑пайплайне GitHub Trending, использующий простые проверки на количество строк, покрытие описаний, корректность имени репозитория и возраст снимка.

Задача

A CI pipeline that aggregates GitHub Trending data was only checking for build success, not data integrity, allowing corrupted or incomplete data to be published without detection.

Результат

All 201 runs succeeded in CI, but five intentional HTML corruptions caused silent failures; the new canary prevented publication of corrupted data and detected stale snapshots.

Рабочий процесс

• 1. Schedule GitHub Actions cron (0 */6 * * *) to run scraper.
• 2. Use urllib.request to GET six trending pages (weekly, python, javascript, typescript, rust, go).
• 3. Apply regex parser to extract repo name, description, language, stars, forks, weekly gain.
• 4. Write trending.json, summary.json, index.html and upload as Pages artifact.
• 5. Before publishing, run completeness canary: check row count ≥15, description coverage ≥80%, repo name contains exactly one '/', and snapshot age ≤8h.
• 6. If any assertion fails, block publish and flag the build as failed.

Доказательства

Every single test returned normally. No exception, no non-zero exit, no failed build.

Источник

Four assertions, none of which is "did it crash": Row floor, Description coverage ratio, Name shape, Snapshot age.

Источник