My AI-Built Site's Internal Links Were Unmanageable — So I Mapped the Whole Graph Before Touching Anything

Description and use

An AI-built site struggled with internal linking because pages were generated in isolation. A custom crawler built a raw link graph, identified issues, and enabled a structured consolidation workflow that improved site architecture.

Problem

Internal linking was unmanageable because AI-generated pages were created in isolation without a global link ledger, leading to competing pages, orphans, dead ends, and unclear link strategy.

Outcome

The tool revealed 4,868 pages with 3,931 product URLs, reconstructed 268 category groups covering 92% of products, and enabled a systematic link consolidation workflow that reduced orphan pages and concentrated link equity on stronger pages.

Workflow

• 1. Crawl site and separate navigation from content links using a 90% threshold rule.
• 2. Export graph as CSV, GEXF, GraphML, JSON, and a single offline HTML report.
• 3. Join pages with Search Console data to assess traffic and competition.
• 4. Identify merge candidates and redirect weaker pages.
• 5. Re-point links toward ranking pages and fix orphans/dead ends.
• 6. Re-crawl and diff edge lists to verify consolidation.

Evidence

internal linking is the one lever AI never managed, because it's site-wide state, not page state.

Source

I built the instrument I needed, and open-sourced it: internal-links-export.

Source

The tool recovered the de-facto taxonomy from the link structure itself: every non-hub page grouped under its strongest content in-link source.

Source

With the graph exported (pages CSV + edge list CSV), the workflow I'm running is: Join pages against Search Console, Find merge candidates, Re-point links, Fix the floor, Re-crawl and diff.

Source