My AWS AI Agent Passed Its Tests. It Still Missed a Check.

说明与用途

AI 代理生成的代码正确计算了不健康检查次数和持续时间，但测试未检查所有时间戳，暴露了覆盖缺口。

问题

Determine how many deployment checks were unhealthy and the duration of each completed unhealthy period from a CSV history, ensuring the AI-generated tests cover all required assertions.

结果

The AI-generated code correctly computed 5 unhealthy samples out of 12 (41.7%) and identified two completed unhealthy periods of 15 and 10 minutes. However, the tests only asserted period counts and durations, not individual timestamps, exposing a coverage gap.

工作流

• Provide CSV and calculation rules to Amazon Nova 2 Lite via managed agent
• Model generates Python script
• Script sent to Amazon Bedrock AgentCore Code Interpreter for execution
• Receive results and generated tests
• Compare results to expected values
• Review generated tests for completeness

证据

I gave my AI agent a small deployment history and asked how often the checks were unhealthy. It wrote the Python, ran its own tests, and came back with the right answer.

来源

The program also printed MODEL_GENERATED_TESTS=PASS.

来源

But my prompt asked for assertions, the checks that stop the program when an expectation is wrong, for every stated test expectation. The generated tests covered period counts and durations. They didn't explicitly check every expected start and end timestamp.

来源