My Tests Agreed With My Code. Neither of Them Checked Reality

Описание и применение

Тесты и ревью прошли, но код не работал с реальным API из‑за неверных предположений о формате даты и заголовках. После реального тестирования выявлены две ошибки: неверный формат времени и передача ключа API при редиректах. Исправления включали обновление парсера, безопасную обработку заголовков и установку пользовательского User‑Agent.

Задача

Unit tests and code reviews passed, but the code failed against a real external API due to incorrect assumptions about data format and header handling.

Результат

Two critical defects were uncovered only through live integration: a timestamp format mismatch and improper header forwarding on redirects. Fixing these required updating the parser, adding redirect‑safe headers, and setting a custom User‑Agent. The experience underscored the limitations of internal tests and the value of testing against real external systems.

Рабочий процесс

• 1. Ran local test suite – all green.
• 2. Pointed code at real API – first live read failed due to timestamp mismatch.
• 3. Reviewed parser and fixtures – both assumed ISO 8601.
• 4. Identified redirect issue – urllib forwarded X-API-Key on redirects.
• 5. Engaged external collaborator (FIPSign) – obtained disposable API keys and performed live tests.
• 6. Determined User‑Agent header was missing, causing bot‑flagging.
• 7. Added custom User‑Agent and used add_unredirected_header for X-API-Key.
• 8. Re‑ran live integration – all stages succeeded (201, 200, 200, 200).
• 9. Documented findings and lessons learned.
• 10. Reflected on testing practices – highlighted need for live system tests and independent fixtures.

Доказательства

My tests used ISO because my code used ISO, so they agreed with each other and never checked reality.

Источник

No live FIPSign credential was ever shown to have crossed an origin.

Источник

If you are hitting an endpoint from a script and getting bot-flagged for no visible reason, check what user agent your HTTP library is sending by default.

Источник