AI as a Fluency Machine: What Software Engineering Taught Me About Checking a Model's Answer

说明与用途

团队为LLM生成的代码实施了基于规则的验证系统，填补了检查空白并提高了透明度。

问题

Generated code from language models bypasses existing software engineering checks, creating a blind spot where the model’s output is assumed correct without human scrutiny.

结果

The team achieved a 84% rule coverage with honest gaps, reduced false positives, and gained visibility into system performance decay. Compliance checks now explicitly state what was verified, mitigating automation bias.

工作流

• 1. Define coding rules in JSON with IDs, severity, examples, and regex patterns.
• 2. Generate system prompts and retrieval indexes from the rule corpus using nexa build.
• 3. Run the model to generate code based on the prompt.
• 4. Validate generated code against the same rule set with nexa validate.
• 5. Cross‑check each rule against other rules’ correct samples using nexa crosscheck.
• 6. Diagnose performance issues with nexa doctor and adjust model configuration.
• 7. Iterate rule definitions to reduce false positives and improve coverage.

证据

"Citation existence is not rule compliance."

来源

"An unmeasured number decays into a claim."

来源