Join a Claude Community Event, organized by local enthusiasts for everyone who loves building with Claude!

Anyone can build an AI agent that works in a demo. The hard part is knowing it still behaves when a real customer is vague, angry, or changes their mind halfway through.

In this hands-on workshop you'll learn eval-driven development: turning business rules and observed failures into tests that tell you how often your agent gets it right, and whether your fixes actually helped.

You'll work on a customer refund agent for a fictional retailer, built with Claude Code on the Anthropic API. It runs, and it has flaws. Your job is to find them, measure them, and fix them.

What you'll do

  • Turn a vague business policy into a testable eval spec

  • Write test cases in plain English and let Claude Code turn them into a dataset

  • Run the agent, read its transcripts, and group its failures

  • Build graders: code checks for actions, plus an LLM judge you calibrate against your own labels

  • Fix the agent's prompt, tools, and design, then rerun and watch pass rates move

You'll leave with a working eval suite and a workflow you can apply to your own agents.

Who it's for

Engineers and engineering students. Mixed experience is fine: checkpoint branches catch you up if you fall behind, and stretch goals keep fast finishers busy. All data is synthetic.

What to bring

  • A laptop with Claude Code and Python 3 (detailed instructions to follow before the event)/

  • Basic git (clone and switch branches)

  • API credits provided for approved attendees. Register with the email you'll use for your Anthropic Console account.

You'll write and run Python from the terminal. Setup steps will be sent before the event.

Agenda

5:15 PM Doors open5:30 PM Setup check5:45 PM Why agents need evals, not just unit tests5:55 PM From business policy to an eval spec6:20 PM Building the test dataset, in plain English6:45 PM Break6:55 PM Baseline run and error analysis7:25 PM Graders: code checks and an LLM judge7:50 PM Fix, rerun, and harden8:15 PM Wrap-up: shipping with online evals8:30 PM Event ends. Hang around for Q&A until 9:00.

Times are approximate. Checkpoint branches keep everyone on track if you fall behind.

About Claude Community

Claude Community events are hosted by community members worldwide. Meet other people building with Claude and learn something hands-on. Hosted by Dom Roselli, Claude Community Leader, Las Vegas.