My Cron Job Double-Fired and Placed the Same Order Twice — Postmortem

Описание и применение

Скрипт, запускаемый по расписанию, дважды выполнил заказ из-за одновременного запуска и отсутствия блокировки. Решение: добавить lockfile, улучшить логирование, использовать небуферизованный вывод и сверять локальное состояние с API перед действием.

Задача

A scheduled Python script ran twice concurrently, placing duplicate orders via an exchange API, causing state drift and duplicate fees.

Результат

Implemented a lockfile, enhanced logging, ensured unbuffered output, and added pre-action reconciliation, eliminating duplicate orders and state drift.

Рабочий процесс

• 1. Detect duplicate orders via fee log and state file mismatch.
• 2. Identify root causes: silent-but-alive job and scheduler-human race.
• 3. Implement a file lock at script start to prevent concurrent runs.
• 4. Add start/end logging with PID for visibility.
• 5. Enable unbuffered output or use flush=True to avoid silent jobs.
• 6. Reconcile local state with exchange API before acting.
• 7. Adjust manual run timing to avoid scheduler overlap.

Доказательства

The core fix is about fifteen lines at the top of main(): ... lock_fd = open(LOCK, 'w') ... fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) ...

Источник

If the job runs at :02, I run manual checks at :15. It feels comically simple.

Источник

Before the job opens anything, it now pulls actual positions from the exchange API and treats the local state file as a hint, not as truth.

Источник