We Built an AI Auditor. Here's What It Missed

TL;DR — Quick Summary

• An AI auditor that comes back clean has told you what it found, not what is there. "No findings" means "not found", never "safe".
• Our own AI auditor, Krait (v8), ran blind on five public audit contests. Every finding it raised was real: 10 true, 0 false. The judges had confirmed 66 High and Medium bugs. That is 15.2% recall.
• On 18 bugs we planted ourselves, a later version (v8.2, run as a Claude Code skill) found 13 by our strict count, two of them only at Low. It missed four and half-found a fifth, and the misses mostly needed to know what the code was supposed to do.
• Attackers have AI too. On EVMbench, the best agent wrote working exploits for about seven in ten real audit bugs, and patched about four in ten. In our demo, an AI agent wrote a working exploit test for one of our planted bugs: one signature for 5,000 paid out 25,000.
• Use AI before a human audit, not instead of one: AI pass, fix, human audit on a frozen commit, then AI on every pull request after.

Introduction

Two different runs, two different versions

1. Reading "no findings" as "no bugs"

The demo

The fix

1// workflow/RegressionTest.t.sol
2// Every bug you fix gets a test that fails if it comes back.
3contract RegressionTest is Test, Setup {
4    function test_bug_P14_staleNav() public {
5        vm.warp(block.timestamp + 3 days);           // the NAV feed stops updating
6        vm.expectRevert(RedemptionVault.StaleNAV.selector);
7        vault.redeem(1_000_000e18);                  // must refuse the old price
8    }
9}

2. Trusting precision as coverage

The scorecard

The fix

3. Using AI where it's weakest

What the misses have in common

• P01: the spend limit reads a badge NFT, and anyone can send an NFT. The bug is that the input to the limit is something an attacker controls. Krait flagged the same function for missing access control, a different bug.
• P12: a batch checks only the first root entry against the checkpoint, then applies them all. A second entry replaces the trusted root.
• P15: the vault stores a reference price feed and never reads it. The bug is the check that isn't there.
• P17: one agent key can move every holder's balance, instantly and without a cap. The code does exactly what it says. Whether it should is a design question.

The fix

4. Forgetting attackers have AI too

EVMbench

The demo

1// corpus/src/bridge/ReplayAdapter.sol
2function release(address to, uint256 amount, bytes[] calldata sigs) external {
3    bytes32 digest = keccak256(abi.encode(to, amount));
4    _checkSigs(digest, sigs);
5    token.transfer(to, amount);
6}

1// exploit/Exploit.t.sol (written by the agent, excerpt)
2function test_exploit_replay() public {
3    // The verifiers sign exactly ONE release of 5,000 tokens to RECIPIENT.
4    bytes[] memory sigs = _signOneRelease(RECIPIENT, RELEASE_AMOUNT);
5
6    adapter.release(RECIPIENT, RELEASE_AMOUNT, sigs);
7
8    // REPLAY: resubmit the SAME signatures over and over.
9    uint256 extraReplays = 4;
10    for (uint256 i; i < extraReplays; i++) {
11        adapter.release(RECIPIENT, RELEASE_AMOUNT, sigs);
12    }
13    // ...asserts the recipient was paid amount * number of replays
14}

1$ forge test --match-test test_exploit_replay --fork-url http://127.0.0.1:8547 -vv
2[PASS] test_exploit_replay() (gas: 175621)
3Logs:
4  tokens authorized by the single signature set: 5000000000000000000000
5  times the one signature set was accepted: 5
6  tokens actually paid to recipient: 25000000000000000000000
7  tokens drained from bridge beyond authorization: 20000000000000000000000
8
9Suite result: ok. 1 passed; 0 failed; 0 skipped

The fix

1# workflow/Makefile
2# A finding is closed only when its exploit test exists,
3# passes against the vulnerable commit and fails after the fix.
4close-finding:
5	git stash && forge test --match-test 'test_exploit_$(ID)'          # must PASS: the bug is real
6	git stash pop && ! forge test --match-test 'test_exploit_$(ID)'    # must FAIL: the fix holds

5. AI instead of review, when it should be AI before review

1. Run AI first, and fix the easy bugs cheaply.
2. Freeze the code and bring in a human audit on that commit, so expensive hours go to the hard bugs.
3. Run AI again on every change after the audit.

The fix

1# workflow/ai-review.yml
2# An AI review on every pull request, posted as comments.
3# A lead list for the reviewer, never an approval.
4name: ai-review
5on: pull_request
6jobs:
7  ai-review:
8    runs-on: ubuntu-latest
9    permissions: { contents: read, pull-requests: write }
10    steps:
11      - uses: actions/checkout@v4
12      - run: npx claude -p "/krait-review $(git diff --name-only origin/main -- '*.sol')" > leads.md
13      - run: gh pr comment ${{ github.event.number }} --body-file leads.md
14        env: { GH_TOKEN: "${{ github.token }}" }

Launching soon? Send us your code and tell us what your AI pass found. We will tell you where it breaks, and what it would take to make it safe. Request an audit quote →

The checklist

1. AI "clean" means "not found". Never "safe".
2. Measure recall on bugs you already know, not only false alarms.
3. AI on its strong classes. Humans on custom math, design and integrations.
4. Every finding needs a PoC that runs before it is closed.
5. AI on every pull request, and a human audit before launch.

FAQ

Glossary

Build it safely

Launching soon? Send us your code and tell us what your AI pass found. We will tell you where it breaks, and what it would take to make it safe. Request an audit quote →