Speakers: Charli Posner (Stile Education)
Track: Research / Deep Tech
Training a classifier used to mean collecting labelled data, choosing an architecture, training a model, evaluating it, and deploying it. Today, for a surprising number of problems, you can replace most of that with a prompt.
At Stile, we’ve been doing exactly that: using multimodal generative models to classify handwritten marks on scanned worksheets. In this talk, I’ll explore where generative models make surprisingly good classifiers, and where the abstraction starts to break down. We’ll look at why asking a model to make a binary decision performed badly on messy student handwriting, while asking it to describe what it saw and applying deterministic rules afterwards worked far better.
The problem is that some inputs are genuinely ambiguous. A cross might mean “selected” when every other box is empty, but lose to a tick elsewhere. A crossed-out tick means “not selected”... unless the student then circles the box to select it again. Sometimes the right answer is simply to ask the teacher who knows that student’s handwriting. To safely automate the obvious cases, we therefore need to identify the ones near the decision boundary. And that’s where generative classifiers get awkward.
Traditional classifiers give us scores over a fixed set of classes that we can evaluate and calibrate. Generative models give us text, and sometimes token log probabilities (if the provider exposes them at all). Are those meaningful classification probabilities? What happens with structured outputs, multi-token labels and constrained decoding? And if they aren’t, how do we decide when to trust the model and when to ask a human?
The goal isn’t to build an LLM classifier that never gets things wrong. It’s to build one that knows when its answer is uncertain enough to hand back to a human.
https://data.webdirections.org/ai-engineer-sydney/sessions.json#b1d25aeb-2ffc-4740-ad61-e72177c8e0ed
Full text on webdirections.org
.