Autonomous AI Agents Need a Budget, Not a Prompt

For thirty years we engineered systems to stay up. The interesting failure mode of autonomous AI agents is the opposite: they do not fall over, they keep going. An agent that has stalled on a task does not crash your monitoring. It quietly loops, taking one more action, consulting one more tool, refining its own work one more time, while the bill climbs and every health indicator stays a reassuring green. The outage you know how to handle. The agent that will not stop is a newer kind of problem, and many institutions are meeting it for the first time.

The reason this matters to a CIO rather than only to an engineer is that autonomous AI agents change the risk shape of the systems they sit in. An agent that decides how many steps to take is, underneath the marketing, a program whose stopping is data-dependent and partly delegated to a component that is unreliable by design. Unbounded runtime and unbounded spend are the same property read on two meters, and neither is governed by good intentions.

So the position is this. The control that keeps an agent bounded cannot live in the instructions you give it. It has to live in the code that runs it, enforced whether or not anyone is watching, because the thing you would otherwise be trusting to stop is the very thing you cannot trust.

Why a prompt cannot govern autonomous AI agents

It is tempting to believe this is a prompting problem, that a sufficiently forceful instruction to be efficient and to stop when confident will do the job. It will not, and the reason is structural rather than a matter of prompt craft. The agent's judgement about when it is finished is one of the specific things models are bad at. An over-confident agent stops early and returns half an answer with full conviction; an anxious one loops; a confused one cycles between the same few tools. You cannot bound an unreliable component using a decision made by that same component. The limit has to be exogenous, imposed from outside by code the model does not control and measured against real resources it cannot argue away.

This is the distinction that separates governance from wishful thinking. A prompt shapes the distribution of an agent's behaviour, and shaping is useful. It does not enforce a ceiling. Treating an instruction as if it were a control is how an institution ends up explaining a large and unexpected bill, or an agent that spent a weekend calling an external service, as though these were surprises rather than the predictable result of leaving the off-switch inside the thing being switched off.

What an enforced budget actually looks like

I built a multi-agent system where stopping was treated as a contract rather than a hope, and this is real, working, unit-tested code I designed, not a sketch. I set the ceilings deliberately tight, on the principle that a budget you occasionally have to raise is a far better failure than one you discover you needed after the fact. The shape is what transfers. Every action an agent took passed through a single checkpoint that enforced hard ceilings: a cap on total steps, a cap on tool calls, a wall-clock limit, a spend limit in real currency, a circuit breaker that halted the run after a short series of consecutive failures, and a cap on how many times an agent could loop trying to improve its own output.

Two design choices did the real work. The enforcement sat at one chokepoint that every action had to pass through, because a limit scattered across many call sites is a limit that some later change will forget to apply, whereas a single gate cannot be quietly bypassed. And when a ceiling was hit, the run stopped legibly, naming the limit it had breached, rather than hanging or trailing off with a truncated result nobody could interpret. That legibility is not a nicety. In a regulated setting it is the difference between an incident you can explain and one you cannot.

The self-improvement cap deserves singling out, because it is the ceiling teams most often leave off and the one most likely to run away. The loop where an agent critiques its own work and tries again is seductive precisely because every extra round looks locally justified; it is, after all, still improving. Capping the rounds converts good enough from a judgement you are trusting the agent to make about its own work into a fact the system guarantees. That is the general move behind all of this: take a decision the model would otherwise make about itself, and turn it into a constraint the model does not get a vote on.

This is why agentic AI is as much a platform-control problem as it is a model-capability problem.

Degrade slowly instead of running unbounded

There is a second control that separates a resilient agent platform from a brittle one, and it concerns how the system behaves when an external limit bites. Providers cap request rates. A burst of parallel agent activity can hit that cap and, handled naively, take the whole run down partway through, after you have already paid for the steps that succeeded. In the system I built, every model call across every agent flowed through one shared limiter that kept the combined rate under the ceiling, paced itself when it needed to, and, when it was told to slow down, waited exactly as long as it was asked to before trying again. The result was a run that slowed under pressure and still finished, rather than one that hit the wall and died with money already spent.

For an operational-resilience function this is the property that matters, and it maps onto obligations institutions already carry. Bursty workloads that share a provider quota either pace through the pressure or drop work at the worst possible moment. A system that degrades to slower rather than to broken is doing exactly what resilience regulation asks of critical processes, and it is doing it against the newest and least predictable dependency in the estate.

The steelman, and where it holds

The serious counter-argument comes in two parts, and both deserve a fair hearing. First, hard ceilings abort legitimate work. A genuinely difficult task that needed one more round dies at the cap, and if your limits are set badly you will spend your time explaining why the system gave up on something it could have finished. That is true, and it is the price of the guarantee. Ceilings trade a slice of completion for the certainty that the process terminates, and that trade is obviously right when the cost of not stopping exceeds the cost of an occasional premature halt. For a supervised, high-consequence process, that is not a close call. For a low-stakes assistant a person is watching in real time, an aggressive wall-clock might simply be irritating, and the honest answer is to set the ceilings to match the stakes rather than to abandon them.

Second, some will argue this belongs in infrastructure, in a gateway or a service mesh that meters calls, rather than in application code. For raw, organisation-wide spend caps, that is right, and you should have those too. But the ceilings that actually prevent runaway behaviour are semantic. They need to know what counts as a step, what counts as a failure, and that the agent is on its fourth round of self-critique. That knowledge lives in the application, not in the plumbing. Infrastructure can meter tokens; only the application knows it is in a loop. The two are complementary, and treating the platform layer as sufficient leaves the most dangerous behaviour ungoverned.

What to do differently on Monday

Before any autonomous agent goes near a production process, require that it runs behind an enforced budget with a named owner, and treat a missing budget the way you would treat a missing access control: as a blocker, not a backlog item. Insist the enforcement lives at a single chokepoint every action passes through, and that hitting a ceiling produces a clear, logged, attributable stop rather than a silent truncation. Ask specifically about the self-improvement loop, because it is the one usually left uncapped. And satisfy yourself that the system fails slow under provider pressure rather than dropping work mid-task.

Then close the accountability loop, because this is where the CIO's interest becomes the institution's. Under individual-accountability regimes such as the FCA Senior Managers Regime, someone senior owns the outcomes of an automated process, and “the agent decided to keep going” is not a defence anyone wants to offer a regulator. The useful part is that the instrumentation you build for control is the same instrumentation you need for evidence: the count of steps, tools, spend, and time behind every run is at once a cost lever and an audit trail. Build the accounting once, and it serves both.

That control model belongs in the wider AI-native governance discipline: authority, limits and evidence should be designed into the runtime rather than attached after deployment.

Put the stopping control outside the agent

Give an autonomous agent room to be clever, by all means. Just make sure the room has walls, a clock, and a door that locks from the outside, and that a named person holds the key. The institutions that get burned by agents will not be the ones whose agents were insufficiently intelligent. They will be the ones who mistook a well-phrased instruction for a control, and learned the difference from an invoice.