Unmasking the Silent Failures of Distributed Transactions: Detecting In-Doubt States and Validating Compensations with Python

Description and use

The recipe detects silent in‑doubt states in distributed transactions by parsing logs with orjson, validating timestamps with Pydantic, and reporting inconsistencies.

Problem

Silent data inconsistency in microservices caused by network timeouts leaves distributed state in an "In-Doubt" state without explicit errors.

Outcome

The tool processes millions of log lines, normalizes timestamps, validates schema, and flags transactions stuck in an indeterminate state, enabling operators to investigate and remediate silent failures.

Workflow

• Collect log entries from all services
• Parse each log line with orjson for speed
• Validate ISO 8601 timestamps via Pydantic field validator
• Group entries by correlation_id
• Detect state mismatches (COMMITTED vs IN_DOUBT/STARTED)
• Generate DiscrepancyReport JSON
• Output report for manual review or alerting

Evidence

Network timeouts cause transactions to enter an "In-Doubt" state across service boundaries

Source

The following tool processes multi-service log entries, groups states by transaction identifier (correlation_id), validates ISO 8601 timestamps, and identifies transactions trapped in incomplete or indeterminate states.

Source