Benchmark v1Published NOV 16, 2026

PentestBench

PentestBench evaluates how well different models and harnesses perform on finding web vulnerabilities in real life scenarios in modern frameworks. We evaluated 17 apps with 761 planted vulnerabilities. The corpus includes vulnerability-chain scenarios, where one weakness can enable another or increase its impact.

The corpus spans Next.js, Express, Fastify, SvelteKit, Django, Flask, FastAPI, Ruby on Rails, Laravel, Spring Boot, and Gin. Each tool has access to the application source code; the vulnerability catalog stays sealed. Results compare detection coverage, report precision, and catalog F1.

White box pentest benchmarks only. Black-box pentesting benchmarks are coming soon.

Performance overview

How the scanners compare

Catalog F1 · Higher is better

  1. OpenHack v2 HarnessMythos 5.2
    Catalog F1: 43.21%
  2. Claude Code /security-reviewMythos 5.2 + Haiku
    Catalog F1: 37.04%
  3. Claude Code /security-reviewOpus 5.5
    Catalog F1: 35.51%
  4. Claude SecurityOpus 5.5 + Sonnet
    Catalog F1: 34.91%
  5. Claude SecurityMythos 5.2 + Sonnet
    Catalog F1: 32.59%
  6. Semgrep CE 1.185.0security-audit rules
    Catalog F1: 4.69%

Leaderboard

Ranked by catalog F1. Best values in each metric are bold.

First full pass

Results from the first full pass across all 17 applications. Each scanner configuration is ranked separately.

First full pass: scanner configurations ranked by catalog F1
#Scanner / modelMatched (reports)RECALLPRECISIONCATALOG F1 ↓Evidence
1OpenHack v2 HarnessMythos 5.2277 (521)36.40%53.17%
43.21%
Runtime proof + review
2Claude Code /security-reviewMythos 5.2 + Haiku200 (319)26.28%62.70%
37.04%
Unverified reports
3Claude Code /security-reviewOpus 5.5182 (264)23.92%68.94%
35.51%
Unverified reports
4Claude SecurityOpus 5.5 + Sonnet310 (1,015)40.74%30.54%
34.91%
Code review
5Claude SecurityMythos 5.2 + Sonnet327 (1,246)42.97%26.24%
32.59%
Code review
6Semgrep CE 1.185.0security-audit rules19 (50)2.50%38.00%
4.69%
Unverified static hits

Matched = planted vulnerabilities found. Reports = total reported findings. Scroll horizontally on smaller screens.

Dataset

Application corpus

We plan to open-source the 17-application corpus used in PentestBench. It contains 761 planted vulnerabilities, and every evaluated tool receives the same application source code.

PentestBench v1

Applications
17
Planted vulnerabilities
761
Evaluation
White-box pentesting
Open-source release
Application corpus (planned)

17 applications · planted labels per app

  • ArtVault45
  • BoardSync45
  • BudgetBee45
  • Casa50
  • ClinicADB45
  • DataPulse45
  • Devils Advocate34
  • EduStream46
  • FreightFlow45
  • MedFlow45
  • MediVault49
  • Proplist43
  • RentEase44
  • ShipDash45
  • TeamForge46
  • TrailGuard45
  • VaultPay44
  • Total761

FAQ

Frequently asked questions

About the benchmark, what it measures, and what’s next.

Why create this when others exist?

We created PentestBench to accurately benchmark OpenHack, the always-on AI security engineer we’re building. We wanted an evaluation centered on finding vulnerabilities in complete, modern web applications, including vulnerability chains, rather than only isolated snippets or individual functions.

This is the first version of the PentestBench corpus. Future versions will include applications deployed in cloud environments and evaluate vulnerability reachability and the ability to identify false positives—capabilities a real AI security engineer needs.

What does the benchmark evaluate?

PentestBench measures vulnerability discovery across 17 applications with 761 planted vulnerabilities. Recall measures how many planted vulnerabilities a tool finds; precision measures how many of its reports match the catalog. Catalog F1 balances the two. Results reflect each model and harness configuration working together, rather than model capability in isolation.

Does the benchmark cover vulnerability chaining?

Yes. The corpus includes documented scenarios where vulnerabilities combine to create greater impact, such as a weak session-signing key enabling an authorization bypass, or cross-site scripting combined with readable session cookies enabling session theft. The leaderboard scores individual vulnerability matches through recall, precision, and catalog F1.

Why white-box pentesting?

We want to assess how well tools use application source code to identify security weaknesses in modern web applications. Every evaluated tool receives the source code, while the reference vulnerability catalog stays sealed. This provides a common setting for comparing their findings.

Will there be black-box pentesting benchmarks?

Black-box pentesting results are coming soon.

Which harness performed best?

In our tests, OpenHack v2 Harness achieved the highest catalog F1 in the first full pass. The leaderboard shows first-pass results for every evaluated configuration. OpenHack also maintains PentestBench.