auditsift: security-audit every PR with an AI agent, without drowning in false positives

Description and use

auditsift audits PRs by first running offline static checks, then sending only candidate snippets to two LLM passes that require quoted code evidence, producing SARIF reports that CI can enforce.

Problem

AI‑based security reviews of PRs often return noisy, untrustworthy findings and incur high costs when run on every pull request.

Outcome

The workflow reduces false positives by requiring quoted evidence, limits LLM usage to a few snippets, and keeps most checks offline, enabling free audits with optional paid explanations.

Workflow

• Run offline "map" phase to detect languages, entry points, routes, dependencies and redact secrets.
• Run offline "scan" phase applying hardcoded rules for secrets, SQL interpolation, shell usage, eval, pickle, yaml.load, weak hashes, verify=False, path traversal, SSRF, permissive CORS.
• Send only candidate snippets to a triage LLM for impact and fix suggestions.
• Send the same snippets to a second, independent verify LLM that must quote existing code before confirming a finding.
• Generate JSON, SARIF 2.1.0, and Markdown reports; use "--fail-on high" to block CI if high‑severity findings remain.
• Filter findings to the PR diff with "--diff" and run the audit in a GitHub Actions job that uploads SARIF to the Security tab.
• Optionally enable LLM passes by setting API keys and model names; use AtmoRouter for cost‑effective access.
• Integrate with agents via "auditsift serve" MCP server to call map/scan/audit only when "use_llm:true".

Evidence

Ask a coding agent to "review this PR for security issues" and you usually get back a long, confident list. Some of it is real. A lot of it is plausible-sounding noise

Source

auditsift runs an audit in phases. The first two need no API key and never touch the network.

Source

The verify step is the part I care about most. It doesn't see the triage write-up, so it can't just agree with it. A "confirmed" verdict only counts when the quoted evidence is found in the code.

Source

Exit code 1 means a kept finding is at or above the gate, so CI blocks the merge.

Source