Scrape YouTube Video Metadata and Transcripts with Python and Crawlee

说明与用途

使用 Python、Crawlee 和 Playwright 构建 YouTube 爬虫的指南。通过处理无限滚动、GDPR 同意和提取 XML 格式的转录文本来解决 YouTube API 的限制。

问题

YouTube's official API has strict quotas (10,000 units/day), limiting large-scale data extraction. Direct HTTP scraping is difficult due to complex JSON structures and infinite scrolling. There is a need for a reliable method to extract video metadata, statistics, and transcripts at scale without hitting API limits.

结果

A robust Python-based scraper that bypasses YouTube API limits by using browser automation. It successfully handles infinite scrolling, GDPR compliance, and transcript extraction, outputting structured data suitable for analysis or ML training.

工作流

• Set up the project using uv and crawlee['cli'] with Playwright and Httpx.
• Analyze YouTube's network traffic to identify JSON data structures and transcript endpoints.
• Configure PlaywrightCrawler with concurrency limits (max_tasks_per_minute) and increased timeouts for scrolling.
• Implement a pre_navigation_hook to block unnecessary media requests and manage GDPR consent cookies via use_state.
• Create a router to handle channel pages, using infinite_scroll as a background task to load videos.
• Implement a request transform function to redirect consent.youtube URLs to www.youtube.
• Intercept transcript API requests using page.route to capture the URL and fulfill the request.
• Extract video metadata (title, stats) and transcript content from the page and intercepted requests.
• Deploy the scraper as an Actor on the Apify platform.

证据

If your project requires more data than the API allows, you'll need to use crawling.

来源

YouTube uses infinite scrolling to load new elements on the page... Therefore, we'll use Playwright for crawling.

来源

Removing the fmt parameter is necessary to get the data in a convenient XML format instead of the complex JSON3 format.

来源