My AWS AI Agent Passed Its Tests. It Still Missed a Check.

Description and use

The AI agent produced correct counts and durations, but its tests omitted explicit timestamp assertions, revealing a coverage gap.

Problem

Determine how many deployment checks were unhealthy and the duration of each completed unhealthy period from a CSV history, ensuring the AI-generated tests cover all required assertions.

Outcome

The AI-generated code correctly computed 5 unhealthy samples out of 12 (41.7%) and identified two completed unhealthy periods of 15 and 10 minutes. However, the tests only asserted period counts and durations, not individual timestamps, exposing a coverage gap.

Workflow

• Provide CSV and calculation rules to Amazon Nova 2 Lite via managed agent
• Model generates Python script
• Script sent to Amazon Bedrock AgentCore Code Interpreter for execution
• Receive results and generated tests
• Compare results to expected values
• Review generated tests for completeness

Evidence

I gave my AI agent a small deployment history and asked how often the checks were unhealthy. It wrote the Python, ran its own tests, and came back with the right answer.

Source

The program also printed MODEL_GENERATED_TESTS=PASS.

Source

But my prompt asked for assertions, the checks that stop the program when an expectation is wrong, for every stated test expectation. The generated tests covered period counts and durations. They didn't explicitly check every expected start and end timestamp.

Source