Conference In 5 weeks
December
Monday
to Tue 8 December 2026 · Sydney time
You sign up with the host. foundercal only links to it.
Speakers: Simon Harloff (Dam Secure)
Track: Research / Deep Tech
In this presentation, I’ll show how we set out to give developers useful security feedback on every pull request in under three minutes. The benchmark results and methodology are published here: https://docs.damsecure.ai/blog/pr-review-security-benchmark-update/. It has become our most-cited research to date. We initially expected to compare models using established security benchmarks, but discovered that those tests made agents look more capable than they were. In our testing, frontier models recognised canonical vulnerable repositories such as OWASP Juice Shop, while markers intended for traditional scanners gave them clues that real pull requests would not contain. We needed private fixtures with known ground truth, realistic code and no familiar answers. We also needed to evaluate the complete review system, including the model, prompts, tools, context and stopping rules.
We first built synthetic replicas of open-source applications. They gave us controlled tests, but producing realistic codebases was too slow. We switched to reversing real CVE patches and replaying the vulnerabilities into matching open-source commit histories, then built a model- and harness-independent eval suite that records findings, files opened, searches, tool calls, runtime and cost. I’ll demonstrate the approach by running a synthetic pull request containing an IDOR through an open-source harness, revealing the hidden ground truth and inspecting the agent’s trace. Our benchmark covered ten planted access-control bugs, with five runs per model. Recall ranged from 30 to 100 percent and cost ranged from about $0.04 to $4.25 per pull request. I’ll close with what we would change: start with replayable workloads, capture traces from the first run, and choose metrics based on the failures developers care about.
https://data.webdirections.org/ai-engineer-sydney/sessions.json#55fb0cbc-f032-4c0a-840a-f53cc92b6372
Full text on webdirections.org .
488 George Street, 2000, Sydney
Open in Maps ↗Founders pitch
Investors in the room
One of the year's big ones