Monitoring YouTube without the Data API (and without the quota headache)

Description and use

A scraper parses public YouTube pages to extract all video components without API keys or quota limits, deduplicates over a 7‑day memory, and emits only new videos.

Problem

Relying on the YouTube Data API for frequent monitoring causes quota limits and slow approval for increases, making it impractical for real‑time alerts on new videos.

Outcome

The scraper reliably retrieves all public videos for a keyword, including Shorts, without API keys or quota limits, and deduplicates results to provide true "new" alerts.

Workflow

• 1. Send HTTP GET to youtube.com/results?search_query={keyword} without authentication.
• 2. Parse the embedded ytInitialData JSON blob from the page source.
• 3. Iterate over known renderer components (videoRenderer, gridVideoRenderer, compactVideoRenderer, lockupViewModel, shortsLockupViewModel).
• 4. Extract video ID, title, channel, view count, approximate publish time, thumbnail URL, and Shorts duration if available.
• 5. Detect blocked responses by checking if response size < 50,000 bytes and rotate IP if necessary.
• 6. Store video IDs in a 7‑day memory keyed by keyword set.
• 7. On each scheduled run (e.g., every 15 minutes), emit only videos not seen in the memory.
• 8. Send new video data to the chosen alerting channel (e.g., webhook).

Evidence

Polling five keywords every fifteen minutes would need 480 searches a day. You would be rate‑limited before lunch.

Source

YouTube does not use one component. It uses several, and which one you get depends on the page and on whatever A/B test you have been bucketed into:

Source

When YouTube does not like your IP it returns HTTP 200 with a much smaller page, not a 4xx.

Source