I Built an Agent That Found Real Duplicate Content in My Production SEO Database

说明与用途

使用 GROQ 和 Node.js 的自动化代理能快速识别 SEO 内容重复，手动检查时间从 20 分钟降至 3 秒，但需人工核实误报。

问题

Manually hunting duplicate content in a large SEO database was tedious, error‑prone, and time‑consuming.

结果

The agent re‑discovered known duplicate clusters (e.g., "Gutter Cleaning Cost" vs "Gutter Cleaning Cost 2026") and surfaced a false positive due to shared words, demonstrating both effectiveness and limitations of word‑overlap similarity.

工作流

• 1. Migrate a curated sample of live content (5 articles per trade) from Supabase to Sanity Content Lake as structured article documents.
• 2. Write a GROQ query to fetch all articles ordered by trade: *[_type == "article"] { title, slug, trade } | order(trade asc).
• 3. In a Node.js agent, normalize each title (strip stopwords and punctuation).
• 4. Compute word‑overlap similarity between every pair of titles.
• 5. Flag pairs with similarity above a 40% threshold as potential duplicates.
• 6. Run the agent against the full 130‑article production table.
• 7. Review flagged clusters and decide which to merge or keep.

证据

The agent correctly re‑discovered several duplicate clusters I'd already manually identified and fixed back in September — like "Gutter Cleaning Cost" vs "Gutter Cleaning Cost 2026" (75% overlap) and "Commercial Roofing" vs "Commercial Roofing Company" (67% overlap).

来源

It also honestly surfaced a limitation: a Calgary‑localized article with a typo in its slug got flagged as similar to several generic "near me" articles — purely because of shared words like "roofing" and "near me".

来源