I Built an Agent That Found Real Duplicate Content in My Production SEO Database

Description and use

An automated agent using GROQ and Node.js quickly identifies duplicate SEO content, cutting manual effort from 20 minutes to 3 seconds, though false positives need human review.

Problem

Manually hunting duplicate content in a large SEO database was tedious, error‑prone, and time‑consuming.

Outcome

The agent re‑discovered known duplicate clusters (e.g., "Gutter Cleaning Cost" vs "Gutter Cleaning Cost 2026") and surfaced a false positive due to shared words, demonstrating both effectiveness and limitations of word‑overlap similarity.

Workflow

• 1. Migrate a curated sample of live content (5 articles per trade) from Supabase to Sanity Content Lake as structured article documents.
• 2. Write a GROQ query to fetch all articles ordered by trade: *[_type == "article"] { title, slug, trade } | order(trade asc).
• 3. In a Node.js agent, normalize each title (strip stopwords and punctuation).
• 4. Compute word‑overlap similarity between every pair of titles.
• 5. Flag pairs with similarity above a 40% threshold as potential duplicates.
• 6. Run the agent against the full 130‑article production table.
• 7. Review flagged clusters and decide which to merge or keep.

Evidence

The agent correctly re‑discovered several duplicate clusters I'd already manually identified and fixed back in September — like "Gutter Cleaning Cost" vs "Gutter Cleaning Cost 2026" (75% overlap) and "Commercial Roofing" vs "Commercial Roofing Company" (67% overlap).

Source

It also honestly surfaced a limitation: a Calgary‑localized article with a typo in its slug got flagged as similar to several generic "near me" articles — purely because of shared words like "roofing" and "near me".

Source