My Tests Agreed With My Code. Neither of Them Checked Reality

Description and use

Tests and reviews passed, yet the code failed against a real API due to incorrect assumptions about timestamp format and header handling. Live testing uncovered two defects: timestamp mismatch and API key leakage on redirects. Fixes involved updating the parser, securing headers, and setting a custom User‑Agent.

Problem

Unit tests and code reviews passed, but the code failed against a real external API due to incorrect assumptions about data format and header handling.

Outcome

Two critical defects were uncovered only through live integration: a timestamp format mismatch and improper header forwarding on redirects. Fixing these required updating the parser, adding redirect‑safe headers, and setting a custom User‑Agent. The experience underscored the limitations of internal tests and the value of testing against real external systems.

Workflow

• 1. Ran local test suite – all green.
• 2. Pointed code at real API – first live read failed due to timestamp mismatch.
• 3. Reviewed parser and fixtures – both assumed ISO 8601.
• 4. Identified redirect issue – urllib forwarded X-API-Key on redirects.
• 5. Engaged external collaborator (FIPSign) – obtained disposable API keys and performed live tests.
• 6. Determined User‑Agent header was missing, causing bot‑flagging.
• 7. Added custom User‑Agent and used add_unredirected_header for X-API-Key.
• 8. Re‑ran live integration – all stages succeeded (201, 200, 200, 200).
• 9. Documented findings and lessons learned.
• 10. Reflected on testing practices – highlighted need for live system tests and independent fixtures.

Evidence

My tests used ISO because my code used ISO, so they agreed with each other and never checked reality.

Source

No live FIPSign credential was ever shown to have crossed an origin.

Source

If you are hitting an endpoint from a script and getting bot-flagged for no visible reason, check what user agent your HTTP library is sending by default.

Source