Custom AI Company Filter Mis‑validated: Zero False Positives Due to Flawed Test Logic

说明与用途

一个基于关键词的 AI 公司过滤器声称零误报，但测试逻辑自相矛盾。重新定义负例为独立列表后，发现真实误报，从而可以正确调优过滤器。

问题

A keyword‑based filter claimed zero false positives, but the validation logic was self‑contradictory, allowing non‑AI companies to pass.

结果

The author realized that the test logic was inherently flawed because the definition of negative examples conflicted with the filter’s selection rule, leading to an impossible zero‑false‑positive result. After redefining negative examples independently, the filter’s performance was accurately assessed and could be tuned.

工作流

• Define positive and negative keyword lists
• Score each company for keyword hits
• Set filter rule: select if positive_hits ≥ 2
• Create test set of positive (AI) and negative (non‑AI) examples
• Define negative examples as companies with 0 positive hits and ≥2 negative hits
• Count false positives by checking if a negative example is selected
• Discover that the test logic can never flag a false positive
• Redefine negative examples as an independent fixed list
• Re‑run test and observe realistic false‑positive counts
• Use results to tune filter threshold

证据

The test was too tightly coupled to the logic it was supposed to validate, preemptively deciding the outcome.

来源

I changed the definition of negative examples to be independent of keyword hit counts.

来源