I Built an Open-Source QA Agent Because AI Shouldn’t Grade Its Own Homework

Description and use

MaruCheck is an independent verification layer that compares AI‑generated code against a persistent Quality Contract, detects semantic drift, assesses risk, and recalls historical regressions, ensuring the product meets its original intent.

Problem

AI coding agents can generate both code and tests, risking semantic drift where the same misunderstanding propagates into passing tests that mask incorrect implementations.

Outcome

MaruCheck prevents semantic drift by enforcing a Quality Contract, performs risk‑based test selection, and recalls historical regressions, leading to more reliable AI‑generated code that truly meets product requirements.

Workflow

• Define a Quality Contract for the feature
• Coding agent implements the feature
• MaruCheck scans the repo and analyzes the git diff
• Risk-based verification selects relevant tests
• MaruCheck runs tests and compares results to the contract
• If a semantic drift is detected, MaruCheck blocks the PR and reports the issue
• Developer fixes the implementation and re‑runs MaruCheck
• Repeat until all checks pass

Evidence

If the model misunderstands the requirement once, it can easily carry that same misunderstanding into the tests.

Source

MaruCheck verifies. If it finds something: BLOCKING ISSUE SUB-004 violated.

Source