My Cron Job Double-Fired and Placed the Same Order Twice — Postmortem

说明与用途

一个按计划运行的Python脚本因并发执行和静默作业导致订单被重复下单。通过添加锁文件、改进日志、使用无缓冲输出以及在执行前对状态进行对账来解决问题。

问题

A scheduled Python script ran twice concurrently, placing duplicate orders via an exchange API, causing state drift and duplicate fees.

结果

Implemented a lockfile, enhanced logging, ensured unbuffered output, and added pre-action reconciliation, eliminating duplicate orders and state drift.

工作流

• 1. Detect duplicate orders via fee log and state file mismatch.
• 2. Identify root causes: silent-but-alive job and scheduler-human race.
• 3. Implement a file lock at script start to prevent concurrent runs.
• 4. Add start/end logging with PID for visibility.
• 5. Enable unbuffered output or use flush=True to avoid silent jobs.
• 6. Reconcile local state with exchange API before acting.
• 7. Adjust manual run timing to avoid scheduler overlap.

证据

The core fix is about fifteen lines at the top of main(): ... lock_fd = open(LOCK, 'w') ... fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) ...

来源

If the job runs at :02, I run manual checks at :15. It feels comically simple.

来源

Before the job opens anything, it now pulls actual positions from the exchange API and treats the local state file as a hint, not as truth.

来源