My Cron Job Double-Fired and Placed the Same Order Twice — Postmortem

Description and use

A scheduled Python script double-executed an order due to concurrent runs and silent jobs. Fixes included a lockfile, better logging, unbuffered output, and pre-action state reconciliation.

Problem

A scheduled Python script ran twice concurrently, placing duplicate orders via an exchange API, causing state drift and duplicate fees.

Outcome

Implemented a lockfile, enhanced logging, ensured unbuffered output, and added pre-action reconciliation, eliminating duplicate orders and state drift.

Workflow

• 1. Detect duplicate orders via fee log and state file mismatch.
• 2. Identify root causes: silent-but-alive job and scheduler-human race.
• 3. Implement a file lock at script start to prevent concurrent runs.
• 4. Add start/end logging with PID for visibility.
• 5. Enable unbuffered output or use flush=True to avoid silent jobs.
• 6. Reconcile local state with exchange API before acting.
• 7. Adjust manual run timing to avoid scheduler overlap.

Evidence

The core fix is about fifteen lines at the top of main(): ... lock_fd = open(LOCK, 'w') ... fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) ...

Source

If the job runs at :02, I run manual checks at :15. It feels comically simple.

Source

Before the job opens anything, it now pulls actual positions from the exchange API and treats the local state file as a hint, not as truth.

Source