Ten full catalogues in a day: what the arr ecosystem's own backends will and will not let a directory carry

Описание и применение

Создан каталог из 90 000+ телешоу, 350 000+ игр и других медиа, собранных из открытых источников, разрешающих коммерческое использование с атрибуцией.

Задача

Determine which public metadata backends allow a commercial directory to store full data with attribution.

Результат

Successfully built a commercial catalog of 90,000+ TV shows, 350,000+ games, and additional media, all sourced from licenses permitting commercial use with attribution.

Рабочий процесс

• 1. Reviewed each backend’s terms to assess commercial use and attribution requirements.
• 2. Identified compliant sources (TVmaze, IGDB, Wikidata, TheSportsDB, Kitsu, Project Gutenberg, LibriVox, Steam).
• 3. Built one agent per source to fetch data, store responses as test fixtures, and probe live APIs.
• 4. Created a second agent per source to validate the first: rerun tests, compare consecutive pulls, verify license compliance.
• 5. Corrected seven of eight adapters based on manual review of license text and API behavior.
• 6. Implemented capped, resumable pagination with cursors and scheduled re‑runs.
• 7. Published catalogues at nichedb.dev and documented the pattern at logicsrc.com/asdlc.

Доказательства

Every show TVmaze knows, close to 90,000, CC BY-SA, walked at 250 a page and then kept current from its updates feed.

Источник

Every game IGDB knows, about 350,000; IGDB says in its own words that it prefers the data stored and served.

Источник

Every film and every video game on Wikidata, CC0, walked in numeric id windows over SPARQL.

Источник