Join us for Agentic + AI Observability meetup on Tuesday, October 13th from 5pm - 8pm PST at the Databricks SF office,  an evening focused on evaluating, debugging, and optimizing agentic architectures. We'll explore how to leverage observed agentic traces to continuously iterate on and improve complex AI systems in production.

This meetup is built for software engineers, ML/AI developers, system architects, and AI startup founders who are building or deploying autonomous agents and want to go deeper into the tech. We’ll dive into real-world patterns, failure modes, and engineering solutions for composing multi-step agent workflows, extracting actionable insights from production traces, and establishing robust evaluation loops to refine agent behavior over time.

Whether you’re at an early-stage startup or an established company, if you care about getting AI agents into production and keeping multi-agent systems healthy, this meetup is for you.

Why you should attend

  • Debug & Iterate on Production Agents: Move past generic logs to structured agentic traces, evaluation metrics, and real-time observability pipelines that isolate system failures and track complex model reasoning.

  • Engineer Multi-Step Agent Workflows: Discover practical architectural patterns for composing, running, and managing multi-step agentic systems safely at scale.

  • Turn Observed Traces Into Model Improvements: Learn how to analyze execution traces to fine-tune prompts, refine tool selection, and continuously iterate on your AI workflows post-deployment.

  • Connect with Other Builders: Meet engineers, founders, and practitioners to discuss production observability challenges, swap agent architectural patterns, and network with builders.

Agenda

  • 5:00pm: Registration/Mingling

  • 6:00pm: Welcome Remarks

  • 6:15pm: Talk #1: Hidden Gems in Agent Traces: Automatically Improving Your Agent with Open Source MLflow Observability (Yuki Watanabe, Sr. Software Engineer at Databricks)

  • 6:45pm: Talk #2 with Mandeep Singh, AI Deployment Engineer at OpenAI

  • 7:15pm: Mingling with bites + dessert

  • 8:00pm: Night Ends

Session Descriptions:

  • Hidden Gems in Agent Traces: Automatically Improving Your Agent with Open Source MLflow Observability- Agent traces capture where your agent gets stuck, which mistakes repeat, and where users get frustrated. In this talk, we'll walk through how to use those signals to improve agent quality. We'll show how automatic trace analysis surfaces patterns and quality issues, and how to turn those findings into improvements with repeatable verification. We'll demonstrate the workflow with MLflow, an open-source observability and evaluation tool.