My 300 unit tests passed while my lead finder returned "Refund Policy" as a person

Description and use

OpenLeads unit tests passed, yet live searches returned junk data. A nightly benchmark was added, uncovering and fixing quality bugs, improving lead coverage and reducing junk names.

Problem

Unit tests passed but real searches returned irrelevant or missing results, revealing quality bugs not caught by mocked tests.

Outcome

Nightly live benchmarks uncovered and fixed quality bugs that unit tests missed, improving lead coverage, reducing junk names, and clarifying email confidence.

Workflow

• 1. Add "openleads bench" command to run 12 real searches against live sources.
• 2. Score each search on return presence, lead coverage, junk name rate, email evidence, and duration.
• 3. Commit the JSON scores to BENCHMARK.md via a GitHub Actions workflow nightly.
• 4. Update the project website to display the latest benchmark JSON.
• 5. Use benchmark results to refine name filtering, SMTP verification handling, and API fallback logic.

Evidence

All 300 tests were green, yet real searches returned junk names, missing leads, or timed out.

Source

I added a command, openleads bench, that runs 12 real searches against the live sources and scores the results.

Source

The first run was humbling. The latest one: 12 of 12 searches return results, 82% of requested leads are filled, 0% junk names, and 57% of emails are found or built from a pattern the company really uses.

Source

Your unit tests check that the code does what you wrote. They can't tell you whether what you wrote is any good against today's data.

Source