AI as a Fluency Machine: What Software Engineering Taught Me About Checking a Model's Answer

Описание и применение

Команда внедрила систему правил для проверки кода, генерируемого языковой моделью, устранив пробелы в проверке и повысив прозрачность.

Задача

Generated code from language models bypasses existing software engineering checks, creating a blind spot where the model’s output is assumed correct without human scrutiny.

Результат

The team achieved a 84% rule coverage with honest gaps, reduced false positives, and gained visibility into system performance decay. Compliance checks now explicitly state what was verified, mitigating automation bias.

Рабочий процесс

• 1. Define coding rules in JSON with IDs, severity, examples, and regex patterns.
• 2. Generate system prompts and retrieval indexes from the rule corpus using nexa build.
• 3. Run the model to generate code based on the prompt.
• 4. Validate generated code against the same rule set with nexa validate.
• 5. Cross‑check each rule against other rules’ correct samples using nexa crosscheck.
• 6. Diagnose performance issues with nexa doctor and adjust model configuration.
• 7. Iterate rule definitions to reduce false positives and improve coverage.

Доказательства

"Citation existence is not rule compliance."

Источник

"An unmeasured number decays into a claim."

Источник