Custom AI Company Filter Mis‑validated: Zero False Positives Due to Flawed Test Logic

Description and use

A keyword‑based AI company filter claimed zero false positives, but the test logic was self‑contradictory. Redefining negative examples independently revealed realistic false positives, allowing proper tuning of the filter.

Problem

A keyword‑based filter claimed zero false positives, but the validation logic was self‑contradictory, allowing non‑AI companies to pass.

Outcome

The author realized that the test logic was inherently flawed because the definition of negative examples conflicted with the filter’s selection rule, leading to an impossible zero‑false‑positive result. After redefining negative examples independently, the filter’s performance was accurately assessed and could be tuned.

Workflow

• Define positive and negative keyword lists
• Score each company for keyword hits
• Set filter rule: select if positive_hits ≥ 2
• Create test set of positive (AI) and negative (non‑AI) examples
• Define negative examples as companies with 0 positive hits and ≥2 negative hits
• Count false positives by checking if a negative example is selected
• Discover that the test logic can never flag a false positive
• Redefine negative examples as an independent fixed list
• Re‑run test and observe realistic false‑positive counts
• Use results to tune filter threshold

Evidence

The test was too tightly coupled to the logic it was supposed to validate, preemptively deciding the outcome.

Source

I changed the definition of negative examples to be independent of keyword hit counts.

Source