SiteReadyForSi: Building an AI Search Readiness Scanner with Open-Source AI (Side Hustle)

Description and use

SiteReadyForSi is an automated AI‑readiness diagnostic platform that checks crawler permissions, HTML content, structured data, and generates remediation steps to improve visibility in AI search engines.

Problem

Traditional SEO is insufficient for AI search engines because many modern client‑side rendered sites expose only an empty HTML shell, preventing AI crawlers from parsing content within their rendering budgets.

Outcome

SiteReadyForSi provides an automated, AI‑ready diagnostic platform that identifies crawler permission issues, empty shells, and structured data gaps, delivering actionable remediation steps and improving AI search visibility for SaaS founders, e‑commerce owners, and digital agencies.

Workflow

• Enter homepage URL in the web interface
• SiteReadyForSi performs SSR audit via React/Express
• Crawler permissions are checked against robots.txt for AI bots
• Initial HTML is inspected for empty shells
• JSON‑LD structured data is validated
• llms.txt index is generated
• Audit results are stored in MongoDB Atlas and presented to the user
• User receives a readiness score and fix instructions

Evidence

SiteReadyForSi audits websites to ensure they are machine‑retrievable and technically ready for autonomous AI scrapers.

Source

It analyzes: Crawler Permissions, Initial HTML Inspection, Structured Data (JSON‑LD), llms.txt Generator.

Source

The production platform is engineered as a resilient, full‑stack micro‑SaaS: Frontend & Server: Built with React 19 and React Router 7 over Express, providing blazing‑fast SSR for SEO diagnostics.

Source

Uses BullMQ backed by Redis to orchestrate asynchronous background crawls and audit queues for large websites (>15 pages) without degrading main app latency.

Source