44 days of AI signals as a dataset — and 3 silent API traps I hit publishing it

说明与用途

发布 AI 信号数据集时发现三种无声错误：Cloudflare 阻止默认 User‑Agent、Hugging Face 上传端点已弃用以及硬编码命名空间不匹配。解决方案包括添加真实 User‑Agent、切换到使用 NDJSON 的 commit 端点，并从令牌中获取命名空间。

问题

Publishing a public API or dataset can fail silently, misleading developers into thinking the service works when it does not.

结果

By addressing silent failure modes—adding proper User‑Agent headers, updating to the new commit endpoint, and deriving the namespace from the token—the dataset was successfully published and daily refreshes now work reliably.

工作流

• 1. Add a real User‑Agent header to stdlib requests to avoid Cloudflare 403s.
• 2. Replace the deprecated POST /api/datasets/{repo}/upload/main/{filename} endpoint with the commit endpoint, constructing NDJSON payloads and using LFS for files >10 MB.
• 3. Retrieve the token’s namespace with GET /api/whoami‑v2, verify the role is "write", and use that namespace for all uploads.
• 4. Test the surface anonymously after each deploy to catch silent failures.

证据

If your JSON endpoints sit behind Cloudflare, a request with the default Python-urllib/3.x User-Agent (or an empty UA) can get a 403 at the edge.

来源

I built an uploader against POST /api/datasets/{repo}/upload/main/{filename}. It used to work. Now it returns: HTTP 410: This endpoint is deprecated and has been retired, use the commit endpoint instead.

来源

My publisher had owner = "my-org-name" written into it, but the API token I was handed belonged to a different account. The result: the repository got created under the token's account, and every file upload was sent to the other namespace.

来源