Conference In 6 weeks
December
Monday
to Tue 8 December 2026 · Sydney time
You sign up with the host. foundercal only links to it.
Speakers: Jay Yuan (Commonwealth Bank of Australia)
Track: Software Engineering
AI applications still need regular security patches and dependency updates, but those software changes can also affect the data processing, model interaction, and evaluation paths around an LLM. In our releases, traditional tests and pass/fail evaluation outcomes could remain broadly stable while the underlying evaluation score distributions shifted. That raised a practical question: is the movement a real behavioural regression, normal LLM variability, or variation in the evaluator itself—and when is there enough evidence to stop a release?
This session shares how I approached that problem by bringing behavioural regression testing into the pull request workflow for AI-impacting changes. I’ll cover repeated evaluation runs, behavioural ranges and score distributions, green/amber/red signals, and release-level historical tracking, as well as what worked, what didn’t, and where engineering judgment is still required when deciding whether probabilistic software is safe to ship.
https://data.webdirections.org/ai-engineer-sydney/sessions.json#771b3d00-4c7d-4490-a019-434dd5eeafe5
Full text on webdirections.org .
488 George Street, 2000, Sydney
Open in Maps ↗Founders pitch
One of the year's big ones