Benchmarking Trident

Trident is a pentest harness, the system a model works inside when it attacks a web app. We wanted to know how much the harness itself changes what an agent finds. We ran current models on Trident and on the agent harnesses the labs ship, Claude Code and Codex, against the same 20 black-box web apps, and counted the runs in which the agent confirmed a real flaw on the live target.

Changing only the harness changed the result. Each of the three models we ran on both Trident and Claude Code confirmed a flaw in more runs on Trident.

• DeepSeek V4 Flash on Trident confirmed a flaw in 38 of 44 runs, at $1.29 per confirmed flaw on average. GPT-5.6 Sol on Codex confirmed one in 13 and Claude Sonnet 4.6 on Claude Code in 10.
• With the model held fixed, Claude Sonnet 4.6 had 26 more runs with a confirmed flaw on Trident than on Claude Code. DeepSeek V4 Flash had 23 more and Grok 4.7 had 17 more.
• GPT-6.1 Sol on Codex refused all 44 runs.
• We publish the results but keep the apps and the findings private, so they stay out of training data.

Each setup is one model on one harness. From October 12 to 16, 2026, every setup worked against the same 20 black-box web apps, with up to 44 runs each. The agents had no source code, only the running apps.

A run counts only when the agent confirms a real flaw on the live target. We ran Claude Code and Codex as their labs ship them, with Claude Sonnet 4.6 on Claude Code and the GPT models on Codex. Three models (DeepSeek V4 Flash, Claude Sonnet 4.6 and Grok 4.7) ran on both Trident and Claude Code, so we could compare the same model across harnesses.

To see what the harness alone changes, we held the model fixed and swapped only the harness, on the same apps. On Trident, Claude Sonnet 4.6 confirmed a flaw in 36 runs against 10 on Claude Code, DeepSeek V4 Flash in 38 against 15 and Grok 4.7 in 26 against 9 (on the 32 runs both of its setups ran).

DeepSeek V4 Flash on Trident confirmed flaws in 10 of the 16 vulnerability classes we tested, among them broken access control, business logic, injection and template injection.

We publish the results, but we keep the apps, the details of each target and the findings private. Once a benchmark’s tasks and answers are public, they end up in training data and in public write-ups. A model can then learn the answers instead of finding them, and the score stops measuring anything.

Keeping the benchmark held out keeps the next comparison honest.

The same model had up to 26 more runs with a confirmed flaw on Trident than on Claude Code, and DeepSeek V4 Flash on Trident confirmed a flaw in more runs than GPT-5.6 Sol on Codex or Claude Sonnet 4.6 on Claude Code. A pentest agent’s score belongs to the model and the harness together, and we think the harness deserves as much scrutiny as the model inside it.