44 days of AI signals as a dataset — and 3 silent API traps I hit publishing it
Description and use
Publishing an AI signals dataset revealed three silent failures: Cloudflare blocking due to the default User‑Agent, a deprecated Hugging Face upload endpoint, and a hardcoded namespace mismatch. Fixes involved adding a real User‑Agent, switching to the commit endpoint with NDJSON, and deriving the namespace from the token.
Problem
Publishing a public API or dataset can fail silently, misleading developers into thinking the service works when it does not.
Outcome
By addressing silent failure modes—adding proper User‑Agent headers, updating to the new commit endpoint, and deriving the namespace from the token—the dataset was successfully published and daily refreshes now work reliably.
Workflow
- 1. Add a real User‑Agent header to stdlib requests to avoid Cloudflare 403s.
- 2. Replace the deprecated POST /api/datasets/{repo}/upload/main/{filename} endpoint with the commit endpoint, constructing NDJSON payloads and using LFS for files >10 MB.
- 3. Retrieve the token’s namespace with GET /api/whoami‑v2, verify the role is "write", and use that namespace for all uploads.
- 4. Test the surface anonymously after each deploy to catch silent failures.
Evidence
If your JSON endpoints sit behind Cloudflare, a request with the default Python-urllib/3.x User-Agent (or an empty UA) can get a 403 at the edge.
I built an uploader against POST /api/datasets/{repo}/upload/main/{filename}. It used to work. Now it returns: HTTP 410: This endpoint is deprecated and has been retired, use the commit endpoint instead.
My publisher had owner = "my-org-name" written into it, but the API token I was handed belonged to a different account. The result: the repository got created under the token's account, and every file upload was sent to the other namespace.
- Record date: 2026-10-14T22:56:15.428Z