Speakers: Jon Shen (Suncorp)
Track: AI Engineering
In March I onboarded our first team member dedicated to AI Red Teaming. That was a month before the announcement of Claude Mythos Preview, and the open letters that followed from APRA in April and ASIC in May (Australia’s prudential and conduct regulators) warning organisations to act by shoring up their AI risk management and security practices. We moved before these events because we could see the impending and growing risks, and that specialised skills are needed to keep our customers safe.
Six months and eight deployments later we have broken in repeatedly. We have great data scientists and AI engineers, but the reality is that the skills and mindset needed to build robust AI systems are different from what's necessary to defend them. The surprising thing was the multitude of failure modes we observed, across prompts and architecture, demonstrating the many gaps developers must intentionally think about.
This talk covers what we tested and how, including the process failures we didn't expect, and what we've built to prepare for the future. Leaders will leave with practical steps to start building their own AI red team and defending their AI systems.
https://data.webdirections.org/ai-engineer-sydney/sessions.json#a5764637-5374-4a78-933d-db1a88c06255
Full text on webdirections.org
.