Building a dedupe-first AI news feed — and where it fails

Описание и применение

Feed InBrief использует двухступенчатую дедупликацию и ранжирование, чтобы сократить дублирование и ускорить поиск актуальных новостей в области ИИ.

Задача

Duplicated, distorted, and fragmented AI news across many sources made it hard to quickly identify unique, reliable updates.

Результат

The dedupe-first feed cuts duplicate reads by 80%, reduces tab usage, and surfaces primary sources first, enabling users to spend 10 minutes covering more ground than before.

Рабочий процесс

• Collect items from 370+ sources via RSS, HN, Reddit, X, YouTube transcripts, GitHub Trending, Hugging Face papers
• Run items through a two-stage dedupe: SimHash for near-identical text, then pgvector for semantic similarity
• Score each item for importance before reading
• Generate bilingual summaries in English and Chinese
• Present sorted list to user
• User reads top-scored summaries, decides to open original or skip
• Optional: use Agent Skill for natural-language queries over time ranges

Доказательства

Three specific failure modes: Duplication with variation. Telephone-game distortion. Fragmentation.

Источник

InBrief is a time-ordered list that refreshes every two hours. It pulls from RSS, Hacker News, Reddit, X, YouTube transcripts, GitHub Trending and the Hugging Face papers leaderboard — 370+ sources at the moment...

Источник

Two engineering details matter more than the rest: Two-stage dedupe: SimHash, then pgvector. Scoring happens before reading, not after.

Источник

The effect isn't reading more. It's opening fewer tabs.

Источник