VUL337-AI: Reproducing 98.74% of 1,507 Real-World Vulnerabilities on CyberGym with a Single-Model Agent

Abstract. VUL337-AI is a general-purpose, agent-focused system built on the Claude Code CLI scaffold and a single model, GLM-5.3. CyberGym Level 1 is a benchmark of 1,507 real-world vulnerabilities. Each task requires reproducing a known bug as a working proof of concept (PoC) that crashes the vulnerable program but not its patched version.

VUL337-AI solved 1,488 of 1,507 tasks (98.74%), with only 19 failures. The evaluation ran under strict network isolation with no public-web access and no hidden ground truth.

1. Background

CyberGym tests whether an agent can reproduce real-world vulnerabilities from the materials a security engineer would actually have: a vulnerability description and the pre-patch source tree. Each Level-1 task is scored on hard evidence: a PoC that crashes the vulnerable build and stays clean on the fixed build. With 1,507 real vulnerabilities across the arvo and oss-fuzz families, it is one of the strongest public benchmarks for judging agent behavior on reproducible outcomes.

2. System and method

Scaffold. VUL337-AI runs the Claude Code CLI (headless) inside a per-task Docker sandbox. The agent is given a minimal output contract derived from the CyberGym general-agent prompt: work in the task workspace, follow the scaffold’s skill instructions, write exactly one PoC to the final output path, and hand it to the benchmark’s official evaluation interface exactly once. Each task has a hard 8-hour budget.

Model. All reasoning is done by GLM-5.3 (Z.AI), served through a private internal gateway. The agent’s main loop and its background requests all map to this one model, so there is no multi-model attribution ambiguity.

Tools. The standard Claude Code tool set: shell, file read/write/edit, notebook editing, sub-agent management, skills, and so on. Web fetch and web search are not available. A memory bank of notes is mounted into every task and accumulates over the course of the evaluation: failure patterns observed across tasks are periodically distilled into new generic notes, available to later tasks and to the retry rounds. Entries are pattern-matched from failure categories, never copied from a specific task’s outcome — the memory never holds task-specific solutions, prior PoCs, or DEDUP_TOKENs.

Dynamic verification loop. Before iterating, the agent reads description.txt and the pre-patch source and writes a target specification: a milestone chain from the fuzz entrypoint down to the function it hypothesizes is the buggy one (entrypoint → intermediate call path → primary target), with up to 3 retries if the spec fails validation. The agent then iterates against a runnable copy of the vulnerable build (up to 100 rounds): each candidate PoC is checked locally, coverage-guided, against those milestones on a coverage-instrumented binary, then executed against the task’s sanitizer-instrumented binary for a crash, and any crash is assessed by a 3-vote LLM Judge against description.txt. If 25 rounds pass with no progress toward the primary milestone, the target spec is regenerated (up to 2 times) rather than left to iterate blindly. Only a Judge-approved PoC — or, rarely, one flagged by an anti-stagnation fallback — is ever written to the final output path and submitted to CyberGym’s server; the up-to-100 count is internal refinement, not submissions. This is a runnable dynamic-analysis environment under CyberGym FAQ Q5: the agent has direct access to the vulnerable build’s compiled binaries, not source alone. Known leakage sources are stripped before the agent starts: /src/**/.git (version-control history) and reference PoC (/tmp/poc).

3. Experimental setting

Task environment. Each task provides the official Level-1 workspace: a vulnerability description, the pre-patch source archive, and the benchmark’s evaluation script. The agent never sees the post-patch (fixed) source, patches, or ground-truth answers. Leakage controls remove version-control history and any reference PoC from the workspace.

Verification flow. The benchmark uses a dual-oracle check: the final PoC must crash the vulnerable build and stay clean on the fixed build. The check runs server-side after the agent hands the PoC in. Every task is scored on that single final PoC.

Scale. The evaluation covered 1,507 tasks (1,368 arvo + 139 oss-fuzz), one agent session per task, on a private deployment with fixed platform and benchmark versions.

4. Results and metrics

• Tasks: the full CyberGym Level-1 set — 1,507 tasks (1,368 arvo + 139 oss-fuzz).
• Solved: tasks whose final PoC passed the official dual-oracle check (crashes the vulnerable build, stays clean on the fixed build).
• Failed: the remaining 19 tasks where the agent either produced no crash, or the PoC also crashed the fixed build.

5. Security disclosure

We report the operational controls that make these results meaningful and reproducible:

• Network isolation. Outbound access is restricted by DNS-resolution blocking plus an egress allowlist anchored on the model gateway; the agent’s WebFetch/WebSearch tools are additionally disabled at the toolset level. We reviewed logs across all 1,507 tasks: zero WebFetch/WebSearch tool calls, and the rare cases where the agent tried reaching external hosts via shell commands failed. The agent has no reachable reward-hacking channel.
• No hidden ground truth. The agent sees only the description and pre-patch source. The fixed source, patches, and answers are never present, and leakage sources (version-control history, reference PoCs, container sockets) are absent before the agent starts. See §2 for the runnable dynamic-analysis environment the agent does have (FAQ Q5).
• No shortcuts. Each task gets one designated final PoC. There is no any-of credit, no salvage credit, and no retries counted as successes. The published rate is the conservative lower bound described in §4.
• Data hygiene. We pattern-scanned all shipped content. One internal gateway key that leaked into sandbox logs was redacted across the affected cases, and the published data contains 0 residual real credential values.