Is It OK to Scrape? Working Out Where I Stand

Description and use

Clusterflick aggregates cinema listings while respecting harm‑based restrictions and justifying overrides of control‑based bans for public, non‑harmful data.

Problem

Determining the legal and ethical boundaries of scraping public cinema listings for an aggregator service.

Outcome

Clusterflick now operates within a framework that respects site‑specific harm limits while justifiably overriding control restrictions for public, non‑harmful data, thereby reducing legal risk and improving transparency.

Workflow

• 1. Identify target venues and their data sources.
• 2. Retrieve robots.txt and terms of service for each source.
• 3. Classify restrictions as harm‑based (rate limits) or control‑based (scraping bans).
• 4. Honor harm‑based restrictions by limiting request rate and using polite user agents.
• 5. Override control‑based restrictions when the data is public and the use is non‑harmful, employing stealth or browser automation as needed.
• 6. Document compliance decisions and maintain a report of all sources and their rules.
• 7. Publish aggregated data under an open license where appropriate.

Evidence

Clusterflick scrapes listings for 488 London venues twice a day.

Source

I used to describe this as complying with the technical restrictions while ignoring the terms: the check asks for a browser, so I send one.

Source

The act of collecting is mostly neutral, and what gets judged is what you do with what you've collected.

Source

The BFI run (Southbank, IMAX and festivals) takes around two hours because it goes one page at a time to stay inside them.

Source