Building a no-signup REST API for 25 daily-updated public data sites

Описание и применение

Создано REST‑API без регистрации, агрегирующее данные из 25 ежедневных публичных источников, обеспечивая целостность и консистентность перед публикацией.

Задача

Archiving daily public records from multiple sources requires a reliable way to determine when all data streams are complete and consistent before publishing.

Результат

The API delivers a consistent, validated, and easily consumable dataset from 25 daily‑updated public sources, enabling archival work, integration testing, data cleaning, automation, and descriptive statistics without requiring user authentication.

Рабочий процесс

• 1. Fetch JSON from each of the 25 public data sites daily.
• 2. Validate each source for freshness, matching issue/date, and duplicate signatures.
• 3. If any source fails validation, reject the aggregate and return HTTP 502.
• 4. If all sources pass, publish the aggregate via the API.
• 5. Serve the data through a stable response envelope with OpenAPI 3.1, CORS, pagination, rate limiting, and provenance metadata.
• 6. Clients cache successful responses and retry on 502 errors.

Доказательства

I built a read-only REST API that normalizes the archive and aggregates 25 daily-updated series.

Источник

For example, I reject the latest aggregate rather than returning a partial success when any of the 25 series is stale, has a mismatched issue/date, or duplicates another signature.

Источник