Triggering Data Pipelines from CI/CD via the REST API

Описание и применение

В CI‑процессах теперь корректно обрабатываются ошибки запуска пайплайнов через REST‑API Datanika: при использовании параметра ?wait=true и флага --fail-with-body HTTP‑статус 422 приводит к падению сборки и выводу сообщения об ошибке.

Задача

CI jobs often trigger data pipelines but fail to detect pipeline failures because the API returns 200 even when the run fails, causing silent failures in CI.

Результат

CI jobs now correctly fail on pipeline errors, preventing silent failures and ensuring downstream dashboards reflect real data state.

Рабочий процесс

• 1. Set DATANIKA_API_KEY environment variable in CI.
• 2. Trigger pipeline with curl: curl -sS --fail-with-body -X POST "https://app.datanika.io/api/v1/pipelines/1/run?wait=true&timeout=300" -H "Authorization: Bearer $DATANIKA_API_KEY"
• 3. The request blocks until the run is terminal; 200 indicates success, 422 indicates failure, 408 indicates timeout.
• 4. If 422, CI exits non‑zero and logs the error_message from the JSON body.
• 5. For long runs (>5min), trigger without wait, capture run_id, then poll /api/v1/runs/{id} until terminal or timeout.
• 6. Handle 408 by extending timeout or allowing async flow.
• 7. Use set -e in CI script to propagate non‑zero exit.
• 8. Optionally parse JSON with jq to extract status or error_message for custom logging.

Доказательства

One call. Blocks until the pipeline finishes, returns the run as JSON, and exits non-zero if the pipeline failed.

Источник

With ?wait=true, the endpoint polls the run until it reaches a terminal state, then answers with a status code that describes the run, not just the request.

Источник