44 days of AI signals as a dataset — and 3 silent API traps I hit publishing it

Описание и применение

Публикация датасета AI‑сигналов выявила три скрытых ошибки: блокировка Cloudflare из‑за стандартного User‑Agent, устаревший эндпоинт загрузки файлов Hugging Face и жестко закодированное пространство имён. Исправления включают добавление реального User‑Agent, переход на commit‑эндпоинт с NDJSON и определение пространства имён из токена.

Задача

Publishing a public API or dataset can fail silently, misleading developers into thinking the service works when it does not.

Результат

By addressing silent failure modes—adding proper User‑Agent headers, updating to the new commit endpoint, and deriving the namespace from the token—the dataset was successfully published and daily refreshes now work reliably.

Рабочий процесс

• 1. Add a real User‑Agent header to stdlib requests to avoid Cloudflare 403s.
• 2. Replace the deprecated POST /api/datasets/{repo}/upload/main/{filename} endpoint with the commit endpoint, constructing NDJSON payloads and using LFS for files >10 MB.
• 3. Retrieve the token’s namespace with GET /api/whoami‑v2, verify the role is "write", and use that namespace for all uploads.
• 4. Test the surface anonymously after each deploy to catch silent failures.

Доказательства

If your JSON endpoints sit behind Cloudflare, a request with the default Python-urllib/3.x User-Agent (or an empty UA) can get a 403 at the edge.

Источник

I built an uploader against POST /api/datasets/{repo}/upload/main/{filename}. It used to work. Now it returns: HTTP 410: This endpoint is deprecated and has been retired, use the commit endpoint instead.

Источник

My publisher had owner = "my-org-name" written into it, but the API token I was handed belonged to a different account. The result: the repository got created under the token's account, and every file upload was sent to the other namespace.

Источник