I Built an Agent That Found Real Duplicate Content in My Production SEO Database

Описание и применение

Автоматический агент, использующий GROQ и Node.js, быстро выявляет дублирующий контент в базе SEO, сокращая ручной поиск с 20 минут до 3 секунд, но требует проверки ложных срабатываний.

Задача

Manually hunting duplicate content in a large SEO database was tedious, error‑prone, and time‑consuming.

Результат

The agent re‑discovered known duplicate clusters (e.g., "Gutter Cleaning Cost" vs "Gutter Cleaning Cost 2026") and surfaced a false positive due to shared words, demonstrating both effectiveness and limitations of word‑overlap similarity.

Рабочий процесс

• 1. Migrate a curated sample of live content (5 articles per trade) from Supabase to Sanity Content Lake as structured article documents.
• 2. Write a GROQ query to fetch all articles ordered by trade: *[_type == "article"] { title, slug, trade } | order(trade asc).
• 3. In a Node.js agent, normalize each title (strip stopwords and punctuation).
• 4. Compute word‑overlap similarity between every pair of titles.
• 5. Flag pairs with similarity above a 40% threshold as potential duplicates.
• 6. Run the agent against the full 130‑article production table.
• 7. Review flagged clusters and decide which to merge or keep.

Доказательства

The agent correctly re‑discovered several duplicate clusters I'd already manually identified and fixed back in September — like "Gutter Cleaning Cost" vs "Gutter Cleaning Cost 2026" (75% overlap) and "Commercial Roofing" vs "Commercial Roofing Company" (67% overlap).

Источник

It also honestly surfaced a limitation: a Calgary‑localized article with a typo in its slug got flagged as similar to several generic "near me" articles — purely because of shared words like "roofing" and "near me".

Источник