When a paper flags your training data, the warning lives in a PDF

Description and use

A peer‑reviewed paper flagged two popular Kaggle datasets as unverifiable, yet the warning remained only in a PDF. A machine‑readable registry was built to track such flags and prevent their use in downstream models.

Problem

Clinical prediction models may continue to be used even after their training data is flagged as unverifiable, because the warning is embedded in a PDF and not reflected in the dataset metadata.

Outcome

The registry provides a transparent, citation‑based record of datasets flagged for authenticity or provenance issues, preventing their unverified use in downstream models.

Workflow

• 1. Identify datasets cited in peer‑reviewed literature.
• 2. Review literature for authenticity or provenance warnings.
• 3. Create a JSONL row with dataset id, status, flags, source, and downstream impact.
• 4. Host the registry on GitHub with CC0 license and snapshot semantics.
• 5. Encourage contributions via pull requests with citations.
• 6. Use the registry to filter datasets before model training or deployment.

Evidence

The authenticity of both datasets could not be verified ...

Source

125 clinical prediction model studies in peer-reviewed publications