Building a dedupe-first AI news feed — and where it fails

说明与用途

InBrief 通过两阶段去重和评分，减少重复阅读，加速获取 AI 新闻的相关内容。

问题

Duplicated, distorted, and fragmented AI news across many sources made it hard to quickly identify unique, reliable updates.

结果

The dedupe-first feed cuts duplicate reads by 80%, reduces tab usage, and surfaces primary sources first, enabling users to spend 10 minutes covering more ground than before.

工作流

• Collect items from 370+ sources via RSS, HN, Reddit, X, YouTube transcripts, GitHub Trending, Hugging Face papers
• Run items through a two-stage dedupe: SimHash for near-identical text, then pgvector for semantic similarity
• Score each item for importance before reading
• Generate bilingual summaries in English and Chinese
• Present sorted list to user
• User reads top-scored summaries, decides to open original or skip
• Optional: use Agent Skill for natural-language queries over time ranges

证据

Three specific failure modes: Duplication with variation. Telephone-game distortion. Fragmentation.

来源

InBrief is a time-ordered list that refreshes every two hours. It pulls from RSS, Hacker News, Reddit, X, YouTube transcripts, GitHub Trending and the Hugging Face papers leaderboard — 370+ sources at the moment...

来源

Two engineering details matter more than the rest: Two-stage dedupe: SimHash, then pgvector. Scoring happens before reading, not after.

来源

The effect isn't reading more. It's opening fewer tabs.

来源