Building a dedupe-first AI news feed — and where it fails

Description and use

InBrief uses two-stage deduplication and scoring to reduce duplicate reads and speed up finding relevant AI news.

Problem

Duplicated, distorted, and fragmented AI news across many sources made it hard to quickly identify unique, reliable updates.

Outcome

The dedupe-first feed cuts duplicate reads by 80%, reduces tab usage, and surfaces primary sources first, enabling users to spend 10 minutes covering more ground than before.

Workflow

• Collect items from 370+ sources via RSS, HN, Reddit, X, YouTube transcripts, GitHub Trending, Hugging Face papers
• Run items through a two-stage dedupe: SimHash for near-identical text, then pgvector for semantic similarity
• Score each item for importance before reading
• Generate bilingual summaries in English and Chinese
• Present sorted list to user
• User reads top-scored summaries, decides to open original or skip
• Optional: use Agent Skill for natural-language queries over time ranges

Evidence

Three specific failure modes: Duplication with variation. Telephone-game distortion. Fragmentation.

Source

InBrief is a time-ordered list that refreshes every two hours. It pulls from RSS, Hacker News, Reddit, X, YouTube transcripts, GitHub Trending and the Hugging Face papers leaderboard — 370+ sources at the moment...

Source

Two engineering details matter more than the rest: Two-stage dedupe: SimHash, then pgvector. Scoring happens before reading, not after.

Source

The effect isn't reading more. It's opening fewer tabs.

Source