Triggering Data Pipelines from CI/CD via the REST API

Description and use

CI jobs now correctly fail on pipeline errors via Datanika’s REST API: using ?wait=true and --fail-with-body, a 422 status causes the job to fail and logs the error message.

Problem

CI jobs often trigger data pipelines but fail to detect pipeline failures because the API returns 200 even when the run fails, causing silent failures in CI.

Outcome

CI jobs now correctly fail on pipeline errors, preventing silent failures and ensuring downstream dashboards reflect real data state.

Workflow

• 1. Set DATANIKA_API_KEY environment variable in CI.
• 2. Trigger pipeline with curl: curl -sS --fail-with-body -X POST "https://app.datanika.io/api/v1/pipelines/1/run?wait=true&timeout=300" -H "Authorization: Bearer $DATANIKA_API_KEY"
• 3. The request blocks until the run is terminal; 200 indicates success, 422 indicates failure, 408 indicates timeout.
• 4. If 422, CI exits non‑zero and logs the error_message from the JSON body.
• 5. For long runs (>5min), trigger without wait, capture run_id, then poll /api/v1/runs/{id} until terminal or timeout.
• 6. Handle 408 by extending timeout or allowing async flow.
• 7. Use set -e in CI script to propagate non‑zero exit.
• 8. Optionally parse JSON with jq to extract status or error_message for custom logging.

Evidence

One call. Blocks until the pipeline finishes, returns the run as JSON, and exits non-zero if the pipeline failed.

Source

With ?wait=true, the endpoint polls the run until it reaches a terminal state, then answers with a status code that describes the run, not just the request.

Source