Scrape YouTube Video Metadata and Transcripts with Python and Crawlee

Описание и применение

Руководство по созданию скрейпера для YouTube на Python с использованием Crawlee и Playwright. Решает проблему лимитов API YouTube, обрабатывая бесконечную прокрутку, GDPR-согласие и извлечение транскриптов в формате XML.

Задача

YouTube's official API has strict quotas (10,000 units/day), limiting large-scale data extraction. Direct HTTP scraping is difficult due to complex JSON structures and infinite scrolling. There is a need for a reliable method to extract video metadata, statistics, and transcripts at scale without hitting API limits.

Результат

A robust Python-based scraper that bypasses YouTube API limits by using browser automation. It successfully handles infinite scrolling, GDPR compliance, and transcript extraction, outputting structured data suitable for analysis or ML training.

Рабочий процесс

• Set up the project using uv and crawlee['cli'] with Playwright and Httpx.
• Analyze YouTube's network traffic to identify JSON data structures and transcript endpoints.
• Configure PlaywrightCrawler with concurrency limits (max_tasks_per_minute) and increased timeouts for scrolling.
• Implement a pre_navigation_hook to block unnecessary media requests and manage GDPR consent cookies via use_state.
• Create a router to handle channel pages, using infinite_scroll as a background task to load videos.
• Implement a request transform function to redirect consent.youtube URLs to www.youtube.
• Intercept transcript API requests using page.route to capture the URL and fulfill the request.
• Extract video metadata (title, stats) and transcript content from the page and intercepted requests.
• Deploy the scraper as an Actor on the Apify platform.

Доказательства

If your project requires more data than the API allows, you'll need to use crawling.

Источник

YouTube uses infinite scrolling to load new elements on the page... Therefore, we'll use Playwright for crawling.

Источник

Removing the fmt parameter is necessary to get the data in a convenient XML format instead of the complex JSON3 format.

Источник