How do you know if an AI answer is actually good and how do you catch the cases where it gets things wrong?
Join us on November 2 at 6 PM at CDT Hub (6th floor) for a special Heapcon WarmUp meetup with our sponsor Perplexity. Two speakers from Perplexity will share practical insights into evaluating AI-generated answers and search quality, from using LLMs to judge other LLMs to building adversarial evaluations that uncover weaknesses in a product.
≫≫ Please register here ≪≪
Doors open at 5:30 PM. Talks start at 6 PM. Please note that this event will be held in English.
---------- Not perfect LLM-as-a-judge for not perfect answers
speaker: Leonid Mednikov, Senior Data Scientist @Perplexity
How do we measure the quality of an answer from an LLM or a search engine? Traditionally, we've relied on human raters to do this work. Now LLMs can do much of it for us. But how reliable are their judgments? In this talk, I'll share some practical experience with evaluating search relevance: why we're moving towards LLM-as-a-judge, where it works and where it fails, and why we still need humans.
Adversarial Evals for Search Quality speaker: Aleksei Primak, Member of Technical Staff, Search Quality @Perplexity
How to identify your product's weaknesses, build evals, set up continuous feedback loops and what is possible with latest AI agents.
