PentestBench
PentestBench evaluates how well different models and harnesses perform on finding web vulnerabilities in real life scenarios in modern frameworks. We evaluated 17 apps with 761 planted vulnerabilities. The corpus includes vulnerability-chain scenarios, where one weakness can enable another or increase its impact.
The corpus spans Next.js, Express, Fastify, SvelteKit, Django, Flask, FastAPI, Ruby on Rails, Laravel, Spring Boot, and Gin. Each tool has access to the application source code; the vulnerability catalog stays sealed. Results compare detection coverage, report precision, and catalog F1.
White box pentest benchmarks only. Black-box pentesting benchmarks are coming soon.
Performance overview
How the scanners compare
Catalog F1 · Higher is better
- OpenHack v2 HarnessMythos 5.2Catalog F1: 43.21%
- Claude Code /security-reviewMythos 5.2 + HaikuCatalog F1: 37.04%
- Claude Code /security-reviewOpus 5.5Catalog F1: 35.51%
- Claude SecurityOpus 5.5 + SonnetCatalog F1: 34.91%
- Claude SecurityMythos 5.2 + SonnetCatalog F1: 32.59%
- Semgrep CE 1.185.0security-audit rulesCatalog F1: 4.69%
Leaderboard
Ranked by catalog F1. Best values in each metric are bold.
Results from the first full pass across all 17 applications. Each scanner configuration is ranked separately.
| # | Scanner / model | Matched (reports) | RECALLPlanted bugs found, out of 761.matched ÷ 761 | PRECISIONShare of reports that matched a planted vulnerability.matched ÷ reports | CATALOG F1 ↓Harmonic mean of recall and precision; the ranking metric.2 × matched ÷ (761 + reports) | Evidence |
|---|---|---|---|---|---|---|
| 1 | OpenHack v2 HarnessMythos 5.2 | 277 (521) | 36.40% | 53.17% | 43.21% | Runtime proof + review |
| 2 | Claude Code /security-reviewMythos 5.2 + Haiku | 200 (319) | 26.28% | 62.70% | 37.04% | Unverified reports |
| 3 | Claude Code /security-reviewOpus 5.5 | 182 (264) | 23.92% | 68.94% | 35.51% | Unverified reports |
| 4 | Claude SecurityOpus 5.5 + Sonnet | 310 (1,015) | 40.74% | 30.54% | 34.91% | Code review |
| 5 | Claude SecurityMythos 5.2 + Sonnet | 327 (1,246) | 42.97% | 26.24% | 32.59% | Code review |
| 6 | Semgrep CE 1.185.0security-audit rules | 19 (50) | 2.50% | 38.00% | 4.69% | Unverified static hits |
Matched = planted vulnerabilities found. Reports = total reported findings. Scroll horizontally on smaller screens.
Dataset
Application corpus
We plan to open-source the 17-application corpus used in PentestBench. It contains 761 planted vulnerabilities, and every evaluated tool receives the same application source code.
PentestBench v1
- Applications
- 17
- Planted vulnerabilities
- 761
- Evaluation
- White-box pentesting
- Open-source release
- Application corpus (planned)
17 applications · planted labels per app
- ArtVault45
- BoardSync45
- BudgetBee45
- Casa50
- ClinicADB45
- DataPulse45
- Devils Advocate34
- EduStream46
- FreightFlow45
- MedFlow45
- MediVault49
- Proplist43
- RentEase44
- ShipDash45
- TeamForge46
- TrailGuard45
- VaultPay44
- Total761
FAQ
Frequently asked questions
About the benchmark, what it measures, and what’s next.
Why create this when others exist?
We created PentestBench to accurately benchmark OpenHack, the always-on AI security engineer we’re building. We wanted an evaluation centered on finding vulnerabilities in complete, modern web applications, including vulnerability chains, rather than only isolated snippets or individual functions.
This is the first version of the PentestBench corpus. Future versions will include applications deployed in cloud environments and evaluate vulnerability reachability and the ability to identify false positives—capabilities a real AI security engineer needs.
What does the benchmark evaluate?
PentestBench measures vulnerability discovery across 17 applications with 761 planted vulnerabilities. Recall measures how many planted vulnerabilities a tool finds; precision measures how many of its reports match the catalog. Catalog F1 balances the two. Results reflect each model and harness configuration working together, rather than model capability in isolation.
Does the benchmark cover vulnerability chaining?
Yes. The corpus includes documented scenarios where vulnerabilities combine to create greater impact, such as a weak session-signing key enabling an authorization bypass, or cross-site scripting combined with readable session cookies enabling session theft. The leaderboard scores individual vulnerability matches through recall, precision, and catalog F1.
Why white-box pentesting?
We want to assess how well tools use application source code to identify security weaknesses in modern web applications. Every evaluated tool receives the source code, while the reference vulnerability catalog stays sealed. This provides a common setting for comparing their findings.
Will there be black-box pentesting benchmarks?
Black-box pentesting results are coming soon.
Which harness performed best?
In our tests, OpenHack v2 Harness achieved the highest catalog F1 in the first full pass. The leaderboard shows first-pass results for every evaluated configuration. OpenHack also maintains PentestBench.