The Task Failed, but Where Did the Logs Go? Troubleshooting an Apache DolphinScheduler Failed Task

Description and use

In Apache DolphinScheduler, a task is marked failed with empty runtime metadata. Metadata and Master logs reveal the failure happened during task submission due to an invalid tenant configuration. Alert delivery failure is separate.

Problem

A scheduled task in Apache DolphinScheduler is marked as failed in the UI and alerts are triggered, yet all runtime metadata (host, start_time, log_path, execute_path) is NULL, making it unclear why the task failed and where to find logs.

Outcome

By inspecting metadata and Master logs, the root cause was identified as an invalid tenant configuration causing the Master to mark the task as failed before Worker assignment. Alert routing failure was a separate issue.

Workflow

• 1. Retrieve alert record from t_ds_alert to get process_instance_id and taskCode.
• 2. Query t_ds_task_instance with these identifiers to inspect task lifecycle fields.
• 3. Observe that state=6 (FAILURE) but start_time, host, log_path, execute_path, app_link are NULL.
• 4. Conclude task never reached Worker; stop searching for Worker logs.
• 5. Verify Master IP addresses and connectivity from Worker nodes to rule out network issues.
• 6. Search Master logs using workflow instance ID, task instance ID, task code, and task name to locate failure cause.
• 7. Find "Tenant does not exists" error indicating invalid tenant configuration.
• 8. Separate alert delivery failure (no bind plugin instance) from task failure.
• 9. Resolve tenant configuration issue to prevent future failures.

Evidence

The task instance was created and its state was already 6 (FAILURE). However, except for submit_time, all other runtime-related fields were empty.

Source

"Tenant does not exists" in Master logs

Source