Note: For Government Officers (Technical Level 200 and above)
Building an AI application is just the first step. Equally important is knowing whether it performs as expected, produces reliable results, and can be used safely in real-world settings.
Evaluating AI agents brings another layer of complexity. Their behaviour unfolds across multi-step trajectories, where decisions depend on intermediate outputs, tool calls, changing state, and interactions with external systems. A system may produce a reasonable final answer while still taking an incorrect, unsafe, or inefficient path to get there.
Meaningful agent Evals therefore need to look beyond final outputs to the trajectory as a whole: how the agent makes decisions, uses tools, responds to intermediate results, and whether errors compound along the way. This becomes especially important as agents take more autonomous actions and interact with external systems.
This hands-on technical workshop introduces METIS, a specifications-driven approach to functionality evaluations that brings together best practices into a guided testing workflow, including assessing functional performance within agentic traces.
Participants will leave with a practical framework and recommendations for designing repeatable evaluations for their own agentic systems, including how to build testing specifications, when to evaluate final outcomes, individual actions, or the session as a whole.
Who Should Attend
This workshop is primarily intended for developers, engineers, and technical product teams building or evaluating agentic AI systems. It will also be accessible to technology enthusiasts, researchers, and innovators interested in understanding how agent evaluations can be designed and carried out in practice.
Some familiarity with LLM applications or AI agents would be useful, but no prior experience with evaluation frameworks is required.
Prerequisites
This is a hands-on, code-along workshop. To get the most out of the session, participants should:
Be comfortable with the command line: Running basic Bash CLI commands and navigating terminal environments.
Have Python proficiency: Practical experience with Python fundamentals and package management tooling (e.g., uv, pip, or virtual environments).
Understand AI basics: Foundational knowledge of LLM applications, prompt flows, or chatbots.
Bring a laptop for hands-on exercises:Web: You will require a web browser to read the walkthrough documentation and interact with METIS' web UI
Local Setup (Optional): If you prefer to code along locally on your own machine, follow the METIS Installation Guide and ensure your local development environment has the necessary dependencies installed prior to the session.
Important Notes
Please bring a laptop for this hands-on session.
Registration begins at 2.00pm, with the workshop starting at 2.30pm.
This workshop is strictly open to Singapore Government officers only.
Please register using your government email address.
This workshop is part of GovTech's AI Practice division's efforts to uplift Whole of Government capabilities. Read more about how GovTech is building AI for public good.
Get involved: Learn more about Lorong AI | Speaker Sign-up | WhatsApp Community | LinkedIn | X