Is It OK to Scrape? Working Out Where I Stand

说明与用途

Clusterflick 在尊重网站技术限制的前提下，合理绕过禁止抓取的控制性限制，聚合公开的电影场次数据。

问题

Determining the legal and ethical boundaries of scraping public cinema listings for an aggregator service.

结果

Clusterflick now operates within a framework that respects site‑specific harm limits while justifiably overriding control restrictions for public, non‑harmful data, thereby reducing legal risk and improving transparency.

工作流

• 1. Identify target venues and their data sources.
• 2. Retrieve robots.txt and terms of service for each source.
• 3. Classify restrictions as harm‑based (rate limits) or control‑based (scraping bans).
• 4. Honor harm‑based restrictions by limiting request rate and using polite user agents.
• 5. Override control‑based restrictions when the data is public and the use is non‑harmful, employing stealth or browser automation as needed.
• 6. Document compliance decisions and maintain a report of all sources and their rules.
• 7. Publish aggregated data under an open license where appropriate.

证据

Clusterflick scrapes listings for 488 London venues twice a day.

来源

I used to describe this as complying with the technical restrictions while ignoring the terms: the check asks for a browser, so I send one.

来源

The act of collecting is mostly neutral, and what gets judged is what you do with what you've collected.

来源

The BFI run (Southbank, IMAX and festivals) takes around two hours because it goes one page at a time to stay inside them.

来源