The Task Failed, but Where Did the Logs Go? Troubleshooting an Apache DolphinScheduler Failed Task

Описание и применение

В задаче Apache DolphinScheduler задача помечена как неудачная, но все метаданные о выполнении пусты. Анализ метаданных и логов мастера показал, что ошибка возникла на этапе отправки задачи из-за неверной конфигурации арендатора. Отдельно выявлена проблема с доставкой оповещений.

Задача

A scheduled task in Apache DolphinScheduler is marked as failed in the UI and alerts are triggered, yet all runtime metadata (host, start_time, log_path, execute_path) is NULL, making it unclear why the task failed and where to find logs.

Результат

By inspecting metadata and Master logs, the root cause was identified as an invalid tenant configuration causing the Master to mark the task as failed before Worker assignment. Alert routing failure was a separate issue.

Рабочий процесс

• 1. Retrieve alert record from t_ds_alert to get process_instance_id and taskCode.
• 2. Query t_ds_task_instance with these identifiers to inspect task lifecycle fields.
• 3. Observe that state=6 (FAILURE) but start_time, host, log_path, execute_path, app_link are NULL.
• 4. Conclude task never reached Worker; stop searching for Worker logs.
• 5. Verify Master IP addresses and connectivity from Worker nodes to rule out network issues.
• 6. Search Master logs using workflow instance ID, task instance ID, task code, and task name to locate failure cause.
• 7. Find "Tenant does not exists" error indicating invalid tenant configuration.
• 8. Separate alert delivery failure (no bind plugin instance) from task failure.
• 9. Resolve tenant configuration issue to prevent future failures.

Доказательства

The task instance was created and its state was already 6 (FAILURE). However, except for submit_time, all other runtime-related fields were empty.

Источник

"Tenant does not exists" in Master logs

Источник