I Broke My Own GitHub Trending Scraper 5 Ways on Purpose. All 5 Builds Stayed Green.

Description and use

An automation recipe that adds a completeness canary to a GitHub Actions pipeline scraping GitHub Trending, ensuring data integrity by checking row count, description coverage, repo name shape, and snapshot age before publishing.

Problem

A CI pipeline that aggregates GitHub Trending data was only checking for build success, not data integrity, allowing corrupted or incomplete data to be published without detection.

Outcome

All 201 runs succeeded in CI, but five intentional HTML corruptions caused silent failures; the new canary prevented publication of corrupted data and detected stale snapshots.

Workflow

• 1. Schedule GitHub Actions cron (0 */6 * * *) to run scraper.
• 2. Use urllib.request to GET six trending pages (weekly, python, javascript, typescript, rust, go).
• 3. Apply regex parser to extract repo name, description, language, stars, forks, weekly gain.
• 4. Write trending.json, summary.json, index.html and upload as Pages artifact.
• 5. Before publishing, run completeness canary: check row count ≥15, description coverage ≥80%, repo name contains exactly one '/', and snapshot age ≤8h.
• 6. If any assertion fails, block publish and flag the build as failed.

Evidence

Every single test returned normally. No exception, no non-zero exit, no failed build.

Source

Four assertions, none of which is "did it crash": Row floor, Description coverage ratio, Name shape, Snapshot age.

Source