On the Ammplify security contest benchmark, Solidity-auditor v3 caught 5 of the 13 Highs. v4 catches 11.

Everything is the same except one thing. ↓

What one run finds

Solidity-auditor v3 already scored 80%+ recall on security contests like DODO. In one run, its twelve specialized AI agents found 14 of 17 findings in under 20 minutes.

But run v3 on the same codebase twice and you do not get the same report - you get two good reports that found different bugs.

Each run can find bugs the others miss. So we measured it with three runs over Ammplify in one scan. For each finding, we recorded which run found it.

• 27 + 18 + 13 = 58. Together, the three runs produced 58 findings: 27 in run 1, 18 in run 2, and 13 in run 3. Runs 2 and 3 mostly found new issues rather than repeating what run 1 had already found.
• 54 of those 58 findings appeared in one run only. Four appeared in two. None in all three. The reason is simple: AI is non-deterministic.

Introducing Loop Mode

If runs barely overlap, the obvious move is to run more of them. v4 does that with two pieces:

Loop mode. One scan, N runs. In each run, the same twelve specialists analyze the same scope. At the end of the scan, v4 combines all findings into one report instead of making you merge N reports yourself.

The memory ledger. A shared list of everything found so far. Before each run, the agents check the list to avoid repeating earlier findings and focus on areas they have not explored yet. New findings are added to the list.

On the Ammplify scan, the ledger went from empty to 37 records after run 1, 65 after run 2, and 92 after run 3. It was still climbing when we stopped.

The ledger persists between scans, too - run the skill again next week after a refactor, and it starts from what it already knew, rather than from nothing.

What it’s worth

From the 36 accepted H/M findings in the Ammplify contest, here is what each run contributed:

Ten of the twenty-three real bugs - including six of the eleven Highs - were found for the first time in run 2 or run 3. A single-run scan does not almost find those. It does not see them at all.

And here is where that lands against the alternatives on the same 36 findings, scored the same way:

Compared with Claude Code running bare-bones, solidity-auditor v4 achieved 3.7× the recall on Highs and 2.3× the overall recall.

Installation & usage of solidity-auditor v4

Follow the instructions in the README here: