Triggering Data Pipelines from CI/CD via the REST API

说明与用途

CI 作业现在通过 Datanika 的 REST API 正确处理管道错误：使用 ?wait=true 和 --fail-with-body，422 状态导致作业失败并记录错误信息。

问题

CI jobs often trigger data pipelines but fail to detect pipeline failures because the API returns 200 even when the run fails, causing silent failures in CI.

结果

CI jobs now correctly fail on pipeline errors, preventing silent failures and ensuring downstream dashboards reflect real data state.

工作流

• 1. Set DATANIKA_API_KEY environment variable in CI.
• 2. Trigger pipeline with curl: curl -sS --fail-with-body -X POST "https://app.datanika.io/api/v1/pipelines/1/run?wait=true&timeout=300" -H "Authorization: Bearer $DATANIKA_API_KEY"
• 3. The request blocks until the run is terminal; 200 indicates success, 422 indicates failure, 408 indicates timeout.
• 4. If 422, CI exits non‑zero and logs the error_message from the JSON body.
• 5. For long runs (>5min), trigger without wait, capture run_id, then poll /api/v1/runs/{id} until terminal or timeout.
• 6. Handle 408 by extending timeout or allowing async flow.
• 7. Use set -e in CI script to propagate non‑zero exit.
• 8. Optionally parse JSON with jq to extract status or error_message for custom logging.

证据

One call. Blocks until the pipeline finishes, returns the run as JSON, and exits non-zero if the pipeline failed.

来源

With ?wait=true, the endpoint polls the run until it reaches a terminal state, then answers with a status code that describes the run, not just the request.

来源