Is It OK to Scrape? Working Out Where I Stand

Описание и применение

Сервис Clusterflick агрегирует расписания кинотеатров, соблюдая технические ограничения сайтов и обосновывая обхождение правил, запрещающих скрапинг, когда данные публичны и использование не наносит вреда.

Задача

Determining the legal and ethical boundaries of scraping public cinema listings for an aggregator service.

Результат

Clusterflick now operates within a framework that respects site‑specific harm limits while justifiably overriding control restrictions for public, non‑harmful data, thereby reducing legal risk and improving transparency.

Рабочий процесс

• 1. Identify target venues and their data sources.
• 2. Retrieve robots.txt and terms of service for each source.
• 3. Classify restrictions as harm‑based (rate limits) or control‑based (scraping bans).
• 4. Honor harm‑based restrictions by limiting request rate and using polite user agents.
• 5. Override control‑based restrictions when the data is public and the use is non‑harmful, employing stealth or browser automation as needed.
• 6. Document compliance decisions and maintain a report of all sources and their rules.
• 7. Publish aggregated data under an open license where appropriate.

Доказательства

Clusterflick scrapes listings for 488 London venues twice a day.

Источник

I used to describe this as complying with the technical restrictions while ignoring the terms: the check asks for a browser, so I send one.

Источник

The act of collecting is mostly neutral, and what gets judged is what you do with what you've collected.

Источник

The BFI run (Southbank, IMAX and festivals) takes around two hours because it goes one page at a time to stay inside them.

Источник