I wrote that their test harness had no denominator. Their own script header said it first.

Description and use

Missing fixtures in a regression harness caused silent drift. Updating the script and workflow now discovers all fixtures and treats missing ones as build errors, eliminating silent failures.

Problem

A regression harness silently ignored missing fixtures, causing undetected drift and inaccurate test coverage.

Outcome

The harness now fails early when fixtures are missing, ensuring all tests run as intended and preventing silent drift.

Workflow

• 1. Inspect regression harness for console warnings.
• 2. Locate the script that determines which fixtures run (plan-regression-shards.mjs).
• 3. Read the script header to understand fixture scheduling rules.
• 4. Verify the workflow that calls the script and generates the shard matrix.
• 5. Run the script locally to confirm it fails when a fixture is missing.
• 6. Update the fixture schedule (shard-schedule.json) to include all valid fixtures and exclude invalid ones with reasons.
• 7. Commit changes and run CI to ensure no missing fixtures cause failures.

Evidence

Silent drift. A fixture only ran if someone remembered to paste its name into the YAML. 25 fixtures that the harness can run were in no shard at all, some for months, and 3 more were rejected at load time for invalid meta.json with nothing louder than a console warning.

Source

Fixtures are now discovered from disk. Every one must be either scheduled (with a timing) or explicitly excluded with a reason, or this script fails.

Source