When Workflows Get Stuck: Troubleshooting and Preventing Task Deadlocks in Apache DolphinScheduler

Описание и применение

Удаление или обновление заблокированных записей в базе данных DolphinScheduler решило проблему deadlock, позволив успешно выполнить backfill. Для предотвращения подобных ситуаций рекомендуется использовать Serial Discard вместо Serial Wait.

Задача

A ClickHouse outage caused a surge of 200+ backfill tasks in DolphinScheduler, overwhelming ClickHouse and leading to a deadlock where tasks entered a Serial Wait state and could not progress.

Результат

Clearing the blocked task records from the DolphinScheduler MySQL tables resolved the deadlock, allowing the backfill to complete. Using Serial Discard instead of Serial Wait prevents similar issues.

Рабочий процесс

• 1. Identify the issue: ClickHouse outage → DolphinScheduler tasks stopped. 2. Restart DolphinScheduler → 200+ backfill tasks triggered, overloading ClickHouse. 3. Observe deadlock: tasks in Serial Wait/Ready to Stop, UI red. 4. Access DolphinScheduler MySQL database. 5. Locate key tables: t_ds_process_instance, t_ds_task_instance, t_ds_process_definition. 6. Query t_ds_process_instance for non‑success states. 7. Delete or update blocked records (DELETE or UPDATE state to 6). 8. Verify changes. 9. Run backfill again via UI; tasks complete successfully. 10. Implement prevention: use Serial Discard instead of Serial Wait, monitor task states, and back up DB before manual edits.

Доказательства

There is already a task in progress.

Источник

Delete blocked workflow instances (use with caution; back up the database first)

Источник