When Workflows Get Stuck: Troubleshooting and Preventing Task Deadlocks in Apache DolphinScheduler

说明与用途

在 DolphinScheduler MySQL 数据库中删除或更新被阻塞的记录解决了死锁问题，使 backfill 成功完成。为防止再次发生，建议使用 Serial Discard 而非 Serial Wait。

问题

A ClickHouse outage caused a surge of 200+ backfill tasks in DolphinScheduler, overwhelming ClickHouse and leading to a deadlock where tasks entered a Serial Wait state and could not progress.

结果

Clearing the blocked task records from the DolphinScheduler MySQL tables resolved the deadlock, allowing the backfill to complete. Using Serial Discard instead of Serial Wait prevents similar issues.

工作流

• 1. Identify the issue: ClickHouse outage → DolphinScheduler tasks stopped. 2. Restart DolphinScheduler → 200+ backfill tasks triggered, overloading ClickHouse. 3. Observe deadlock: tasks in Serial Wait/Ready to Stop, UI red. 4. Access DolphinScheduler MySQL database. 5. Locate key tables: t_ds_process_instance, t_ds_task_instance, t_ds_process_definition. 6. Query t_ds_process_instance for non‑success states. 7. Delete or update blocked records (DELETE or UPDATE state to 6). 8. Verify changes. 9. Run backfill again via UI; tasks complete successfully. 10. Implement prevention: use Serial Discard instead of Serial Wait, monitor task states, and back up DB before manual edits.

证据

There is already a task in progress.

来源

Delete blocked workflow instances (use with caution; back up the database first)

来源