The Benchmark Ends. The World Doesn’t: Building Persistent Engineering Environments for Continual Learning
Picture: Web Directions

Conference In 5 weeks

December
Monday
to Tue 8 December 2026 · Sydney time

The Benchmark Ends. The World Doesn’t Building Persistent Engineering Environments for Continual Learning

Where
Research / Deep Tech room 488 George Street, 2000, Sydney
Host
Web Directions
Entry
Not stated
Sign-up
On the host's page
Open on webdirections.org

All Sydney events

You sign up with the host. foundercal only links to it.

Who you meetThe people on the listing, and what the host tells us about the room.

  • Theodoros Galanos Speaker
Investors Not stated The listing does not say
The format Talks and halls Conference
Sign-up On the host's page Price not stated

AboutIn the host's words.

Speakers: Theodoros Galanos (Nomic)

Track: Research / Deep Tech

Large language models can complete sequences of tasks—but continual learning is not just doing more tasks. Most agent environments reset after every episode: state disappears, required follow-up vanishes, delayed effects are cut off, and the next task arrives as though the previous one never happened.

Real engineering projects do not reset. They deal in atoms, not bits: decisions end up in steel and concrete, and a missed comment cannot be patched after the pour. Evidence arrives late, decisions expire, interventions require verification, and a new agent may inherit the consequences of decisions it did not make. A project is one continuing, long-horizon world, and agents must carry work forward across tasks, revisions, and handovers.

This talk presents Persistent Task Worlds: executable environments for evaluating and training agents across continuing engineering project histories. They separate world state, observable evidence, institutional records, conversation state, and learner state, because a record is not the world: a stored session lists which file an agent read, not what it saw, so learning from history means replaying it. Time advances independently of agent turns; documents and requests arrive mid-stream; actions create requirements that persist; fresh agents inherit the same world, including the project memory earlier agents wrote; and evaluation includes effects beyond the visible scoring window, such as whether later revisions preserve earlier work.

I'll present results and early evidence from two studies, both starting from single-episode evaluation (one instruction in, one answer out) as the baseline. First, fixed models and agent harnesses traverse matched environments with different ways of carrying the past: continuous context, raw history, structured state, project memory written by the agent and harness, or only the current snapshot. We measure decision quality, missed follow-up requirements, revision behaviour, how memory is written and reused, downstream outcomes, and interaction cost.

Second, we compare frozen models with models post-trained on real agent trajectories, contrasting trajectories flattened into single episodes with continuing histories. We measure forward transfer, sample efficiency, and performance on untouched histories. External records remain available as a control, letting us separate improvement in the learner from improvement in record-keeping.

The same problem appears in coding agents, scientific workflows, infrastructure, and any system where today's actions change tomorrow's work. Persistent memory is not a persistent world—and a persistent world is not proof of continual learning.

https://data.webdirections.org/ai-engineer-sydney/sessions.json#5b215820-4e4d-46e3-b714-7d6fc03945ad

Full text on webdirections.org .

Where

Research / Deep Tech room

488 George Street, 2000, Sydney

Open in Maps ↗

Same week in Sydney All 151 events

Dec
to 5 Dec
Hackathon

AI Engineer Hackathon

Founders pitch

Web Directions Free
Dec
Fri 08:00
Dinner

Startup Coffee Social #39

Investors in the room

Tank Stream Suites
Startup Coffee Social Events
A$5
Dec
to 13 Dec
Conference

AI Week

Web Directions
Dec
to 12 Dec
Conference

NeurIPS 2026

One of the year's big ones

NeurIPS Foundation
Dec
to 8 Dec
Meetup

Lessons from Economics for the Human / Agent Software Workforce

Leadership room
Web Directions
Dec
to 8 Dec
Meetup

Ontologies

AI Engineering room
Web Directions
Dec
to 8 Dec
Meetup

Shipping together in an AI native team

Software Engineering room
Web Directions
Dec
to 8 Dec
Meetup

Dependency hell is back. This time it's your agent's config.

Software Engineering room
Web Directions
Dec
to 8 Dec
Meetup

Your website is a terrible API

AI Engineering room
Web Directions
Dec
to 8 Dec
Meetup

Sovereignty Is an Inference Problem

Leadership room
Web Directions
Open on webdirections.org Mon 7 Dec ↗