When AI Agents Fail Halfway Through: Designing for Recovery, Compensation, and Safe Re-Execution
AI reasoning can be non-deterministic. Enterprise execution cannot be careless. The architecture between the two is what makes agent autonomy safe.
Imagine an AI agent responsible for fulfilling a customer order. It validates the customer, checks availability, reserves inventory, creates the ERP order, and requests payment authorization. The payment API times out before returning a response.
What should the agent do now? Retrying might charge the customer twice. Restarting the workflow might reserve the inventory twice. Continuing might leave the ERP and payment systems in disagreement. Rolling everything back may not even be possible because some of those actions have already changed the outside world.
Asking the model what to do next does not solve the fundamental problem. The model may not know whether the payment succeeded before the connection failed.
This is where agent engineering stops being primarily an AI problem and becomes a distributed-systems problem.
An Agent Is More Than a Model Call
A traditional application request is usually short-lived and operates within well-understood transactional boundaries. A database transaction either commits or rolls back, and the application generally knows which occurred.
An AI agent can behave very differently. It may run for minutes or hours, invoke multiple external systems, pause for human input, replan based on new information, and perform actions with real side effects. Those actions may include reserving inventory, modifying an ERP record, approving a refund, sending an email, creating a purchase order, or publishing information externally.
The reasoning process that determines those actions is also non-deterministic. The same objective and similar context can lead the agent down different execution paths.
That combination creates an important architectural challenge: a non-deterministic decision-maker is controlling a distributed process that changes deterministic enterprise state.
This is not simply a theoretical concern. Cemri and colleagues’ MAST research, Why Do Multi-Agent LLM Systems Fail?, analyzed execution traces across multiple agent frameworks and identified 14 recurring failure modes spanning system design and specification, inter-agent coordination, and task verification and termination.
The broader lesson is that agent failures are not simply failures of model intelligence. They can emerge from how the surrounding system is structured, how work is coordinated, and how completion is verified.
This article focuses on one particularly consequential subset of that problem: what happens when an agent has already begun changing external enterprise state and execution fails, stalls, or becomes ambiguous.
Failure Is Not One Thing
Before deciding how to recover an agent, the architecture first has to understand what kind of failure occurred. The most useful distinction is often not simply why the agent failed, but when it failed relative to a real-world side effect.
If the agent fails before an external action executes, recovery may be straightforward: resume from the previous checkpoint. If the request was sent but the response never arrived, the outcome is ambiguous. Did the payment fail, or did the payment succeed and only the response get lost?
A third case is even more subtle. The action may have succeeded, but the agent crashes before recording that success. When execution resumes, its internal state says the action has not happened even though the outside world says otherwise.
Finally, several actions may have completed before a later one fails. Inventory has been reserved and an ERP order created, but payment has not been captured. The enterprise is now in a partially completed state.
These situations require different responses. Some can be retried. Some must first be verified. Some require compensation. Others should stop and wait for a human.
The first rule of agent recovery is therefore simple: classify before you react.
Separate Reasoning From Durable Execution
One of the most useful architectural patterns emerging in agent systems is the separation of reasoning from durable execution.
The reasoning layer answers questions such as: What should happen next? Which tool should I use? Has the objective been achieved? Should I replan? Frameworks such as LangGraph and CrewAI are oriented toward this kind of agent reasoning, graph traversal, coordination, and tool selection.
The durability layer answers a different set of questions: What already happened? What state must survive a crash? Is this operation safe to retry? How long should we wait? What should happen if the worker disappears? Where should execution resume?
Durable workflow technologies such as Temporal or AWS Step Functions can provide capabilities at this layer, while some agent frameworks provide their own checkpointing and persistence mechanisms. The point is not that every architecture requires a particular combination of products. The important point is the separation of responsibilities.
Let the reasoning layer decide what should happen next. Let the durability layer ensure that execution survives what happens next.
This composition matters because the agent’s conversational memory is not an execution ledger. A model remembering that it called a tool is not equivalent to having durable evidence that the resulting business transaction committed.
The Checkpoint Tells You What the Agent Believed. The System of Record Tells You What Happened.
Checkpointing is one of the foundational recovery mechanisms. After meaningful execution steps, the system persists enough state to reconstruct the workflow: messages, relevant context, tool results, variables, control position, and other information required to resume execution.
This allows a crashed agent to resume from a known point instead of repeating an entire process. Durable execution can go further by preserving an event history that lets the runtime reconstruct state and replay only what needs to be replayed.
But checkpointing alone is not enough.
Suppose the agent sends a payment request. The payment succeeds, but the agent crashes before saving the successful response. The checkpoint says payment has not completed. The payment provider says that it has.
That distinction leads to a principle that becomes increasingly important as agents gain autonomy:
Reasoning state is not enterprise state.
An agent may believe inventory was reserved, but the inventory system owns that fact. It may believe a customer was charged, but the payment platform owns that fact. It may believe an ERP transaction completed, but the ERP system owns that fact.
Recovery therefore cannot rely solely on replaying what the agent remembers. It must sometimes reconcile the agent’s execution state against the authoritative systems where the side effects actually occurred.
The checkpoint tells you what the agent believed happened. The system of record tells you what actually happened. Recovery requires both.
Idempotency Must Come Before Retry
Retries are one of the easiest recovery mechanisms to implement and one of the easiest to implement dangerously.
Imagine that a tool call times out. The agent concludes that the call failed and tries again. If the original request actually succeeded, the retry may create a second order, make a second reservation, send another notification, or initiate another payment.
The architecture therefore needs a durable way to recognize that two execution attempts represent the same intended business action.
A common technique is an idempotency key derived from durable workflow identity:
{workflow_run_id}:{step_id}
Before executing a mutating operation, middleware checks an idempotency ledger. If the operation has already completed, the previous result can be returned without performing the side effect again. If it has not completed, execution can proceed and the result can be recorded.
The key must remain the same across retries. Generating a new identifier every time the model retries defeats the entire purpose because the downstream system can no longer distinguish a retry from a genuinely new action.
For ambiguous outcomes, an even safer pattern is verify before retry. Instead of assuming a timeout means failure, query the authoritative system to determine whether the intended action actually occurred.
Idempotency precedes retries. When the outcome is ambiguous, verification precedes both.
Even then, not every failure deserves another attempt. A transient network timeout may justify exponential backoff, jitter, and a bounded retry budget. A malformed request that violates the same contract on every attempt will not improve by being submitted five more times. Repeated failures against the same dependency may instead trigger a circuit breaker, fallback path, or escalation.
These decisions should not be left entirely to the agent’s own reasoning. Retry budgets, termination conditions, and failure classifications belong in the execution architecture so an agent cannot consume resources indefinitely while repeatedly pursuing the same unsuccessful strategy.
Not every failure is worth retrying. The architecture—not the model—should decide when to try again, change course, or stop.
When Retrying Cannot Put the World Back Together
Checkpointing helps execution resume. Idempotency makes repeated operations safe. Neither solves the problem when several successful actions must be undone because a later step fails.
Consider:
Reserve inventory → Create order → Authorize payment → Arrange shipment
If shipment cannot be arranged, simply marking the workflow as failed may leave inventory unavailable and a payment authorization outstanding.
This is where an old distributed-systems pattern becomes useful again: the saga.
Instead of pretending a long-running process across many systems is one atomic transaction, a saga treats each business action as its own transaction and associates it, where possible, with a compensating action.
Reserve inventory can be paired with release inventory. A temporary booking can be paired with cancellation. A captured payment may be paired with a refund.
A compensation is not a database rollback. It is another business operation intended to reverse the effect of an earlier operation. That distinction matters because compensation can itself fail, require retries, or produce new side effects.
In an agent architecture, this suggests something more explicit than simply giving the model a collection of tools. Side-effecting tools should carry execution semantics: whether the action is idempotent, whether it can be compensated, what the compensating operation is, whether it expires, how its result can be verified, and what authority is required to execute it.
A simple execution contract might look conceptually like this:
Action: reserve_inventory
Idempotent: yes
Compensates: release_inventory
Expires: 30 minutes
Authority: autonomous
Verify: inventory_systemThis does not have to be the literal implementation format. The architectural point is that these properties should be explicit and machine-enforceable rather than existing only in documentation or the model’s prompt.
Compensation is easiest when a central orchestrator knows which steps completed and can invoke compensating actions in the appropriate order. In event-driven or multi-agent choreography, recovery becomes harder because no single participant may know the state of the entire business transaction. The architecture then needs correlation, progress tracking, timeouts, and saga-level monitoring capable of recognizing that a distributed process has stalled and deciding when compensation should begin.
The model can reason about the workflow. The architecture should own the rules for safely changing the world.
Not Everything Can Be Compensated
The saga pattern has an obvious boundary: some actions cannot meaningfully be undone.
An inventory reservation can be released. A payment can often be refunded. A provisional record can usually be cancelled.
An email that has been read cannot be unsent. A public statement cannot reliably be recalled. A legal commitment may have consequences that no API call can reverse.
That does not mean every irreversible action requires human approval. A routine order-confirmation email may be irreversible but low-risk enough to automate. The more useful architectural decision considers two dimensions: reversibility and consequence.
Low-risk irreversible actions may be handled through strong validation, policy controls, and postcondition verification. High-consequence irreversible actions should generally cross a stronger authorization boundary. Depending on the context, that may mean explicit business authorization, a human approval gate, or both.
This suggests a broader architectural boundary between reasoning and execution. Before a proposed action is allowed to change enterprise state, it should cross a control boundary that evaluates identity, delegated authority, policy, risk, and, where required, human approval.
Human-in-the-loop is therefore one form of execution authorization, not the control boundary itself.
Compensate when an action is reversible. Verify aggressively when it is irreversible but low-risk. Require stronger authorization or human approval when irreversibility and consequence intersect.
Human-in-the-Loop Is an Execution State
Human review is often drawn in architecture diagrams as a box sitting outside the normal workflow, something the system falls back to when automation fails.
That understates its architectural importance.
A production agent system should be capable of entering an explicit state such as:
PAUSED_AWAITING_APPROVAL
The agent checkpoints its execution state, generates an approval request, and stops. It should not need to keep a process running in memory while waiting for someone to respond, and it should not have to restart the workflow when approval eventually arrives.
When the response comes back, the workflow resumes from its durable checkpoint. But resuming execution safely requires more than restoring the agent’s state because the enterprise state on which the original decision was based may have changed while approval was pending.
A robust implementation can generate the idempotency key before the pause and persist it with the checkpoint. At the same boundary, it can create a cryptographic hash of the proposed action and the parameters the human is actually approving.
When approval arrives, the runtime revalidates the relevant enterprise state and reconstructs the action that would now be executed. It compares the resulting action hash with the checkpointed hash. If the action is materially different, execution stops rather than allowing an old approval to authorize a changed transaction.
The approval request itself should also have a policy-defined time-to-live. An approval that has been waiting beyond its valid window should expire or escalate rather than remain indefinitely executable.
The human approves a specific action, not a vague intention that the agent can reinterpret later.
Human approval therefore becomes more than a user-interface feature. It is an asynchronous, durable execution primitive protected by idempotency, state validation, and expiration policy.
The Failures Your APM Dashboard May Never See
There is another class of failure that makes agent systems especially difficult to operate: nothing technically fails.
The model responds successfully. The tool endpoint returns HTTP 200. Latency looks normal. CPU and memory are healthy. No exception appears in the application log.
Yet the agent may have called the wrong tool, used stale context, stopped prematurely, or fallen into an infinite reasoning loop—repeatedly choosing the same unsuccessful tool strategy while consuming model tokens and compute. Step repetition and failure to recognize termination conditions are explicit failure modes identified in the MAST taxonomy.
From an infrastructure perspective, the system can look perfectly healthy throughout such a loop. Model calls succeed, APIs respond, requests remain within latency thresholds, and no exception is necessarily thrown. The agent is technically running while operationally going nowhere.
Traditional application performance monitoring remains essential, but infrastructure health is not the same as agent correctness. An HTTP success code tells us that a request was processed; it does not tell us that the agent made the right decision, made progress toward the objective, or knew when to stop.
Agent observability therefore has to connect a longer chain:
Intent → Plan → Decision → Tool Call → Side Effect → Verification → Business Outcome
It also needs progress signals that conventional application monitoring rarely requires: repeated tool/action combinations, repeated reasoning states, unusually high step counts, token consumption without corresponding state advancement, and failure to satisfy a termination condition.
This is the difference between asking “Is the software running?” and asking “Is the agent making meaningful progress toward the intended outcome?”
A healthy-looking agent can still be wrong—and a busy-looking agent can still be stuck.
A Failed Agent Run Should Become an Operational Object
When automatic recovery is no longer safe, the run should not disappear into an error log or retry indefinitely.
A dead-letter mechanism can turn the failed execution into something operationally manageable. But for agent systems, the dead-letter record needs considerably more context than a conventional failed message.
Operators may need the original intent, workflow identity, failed step, completed actions, partial side effects, retry history, checkpoint, compensation status, approval history, and trace lineage. That information makes it possible to inspect the state, correct the underlying problem, compensate where necessary, and safely replay the workflow.
This also turns failure data into an operational signal. A rising dead-letter rate for one customer cohort, document type, tool, schema version, or agent release may reveal systemic degradation before conventional infrastructure metrics show anything unusual.
The dead-letter queue should not be designed as a graveyard. It should be designed as a recovery surface.
Recovery Paths Need to Fail in Testing Before They Fail in Production
Recovery architecture should not be trusted simply because the code exists. Teams need to deliberately exercise the failure conditions the architecture is supposed to survive.
Terminate a worker immediately after a side effect but before checkpointing. Inject a timeout after a downstream system has accepted a request. Deliver the same event twice. Force a compensation to fail. Allow an approval to expire. Change enterprise state while an action is waiting for human approval.
These tests answer questions ordinary happy-path testing cannot. Does the idempotency ledger actually prevent duplication? Does verify-before-retry distinguish an ambiguous outcome from a genuine failure? Can a compensation itself be retried safely? Does stale approval detection actually stop execution? Can an operator replay a dead-lettered run without creating another side effect?
The same discipline should apply to the tool catalog. Every side-effecting tool should be auditable for its execution semantics: whether it is idempotent, how its outcome is verified, whether it has a compensating action, whether that action expires, and which authorization boundary applies.
This is where fault injection and chaos-style testing become particularly valuable. The goal is not merely to prove that the happy path works, but to deliberately create the awkward states that production will eventually create on its own.
A recovery mechanism that has never been forced to recover is still an assumption.
What the Recovery Architecture Looks Like
The individual patterns become easier to understand when they are viewed as parts of one architecture. The agent is responsible for reasoning, but every proposed real-world action passes through architectural controls before durable execution changes enterprise state.
The reasoning layer is responsible for planning, context, tool selection, evaluation, and adaptive decision-making. Frameworks such as LangGraph or CrewAI may participate here, depending on the implementation.
The control boundary determines whether a proposed action should be allowed to proceed. Identity, delegated authority, policy, consequence, and human approval belong here rather than being left solely to the model’s judgment.
The durability layer makes authorized execution recoverable. Durable workflow runtimes such as Temporal or AWS Step Functions can contribute here, alongside persistent checkpoint stores, idempotency middleware, compensation handlers, retry policies, reconciliation, and dead-letter processing.
The enterprise tool layer is where reasoning becomes reality. These systems remain authoritative for the business state they own, which is why recovery sometimes requires reconciling the agent’s checkpoint against those systems rather than simply replaying the agent’s memory.
Finally, observability crosses every layer. A useful execution trace should connect the original intent to reasoning decisions, authorization, tool execution, side effects, retries, human intervention, compensation, and the final business outcome.
No single framework needs to own all of these responsibilities. A practical architecture may combine an agent framework for reasoning, a durable workflow runtime for execution, persistent stores for checkpoints and idempotency, middleware around side-effecting tools, and distributed tracing for observability.
Reasoning and durability are different architectural problems and should be allowed to evolve independently.
The Architecture Around the Agent Is What Makes Autonomy Possible
The temptation with agentic systems is to concentrate on the intelligence in the middle: better models, better prompts, better reasoning, better planning, and increasingly sophisticated tool use.
Those capabilities matter, but production trust is often determined somewhere else. It is determined when the network times out after a payment request, when an agent crashes after modifying three systems, when the same event arrives twice, when an approval sits unanswered, when a compensation fails, when an agent enters a perfectly healthy-looking loop, or when every dashboard is green while the business outcome is wrong.
The architectural principles are surprisingly familiar. Separate reasoning from durable execution. Make idempotency a prerequisite for retries. Reconcile ambiguous outcomes against authoritative systems. Define compensation before granting autonomy. Put consequential irreversible actions behind appropriate authorization boundaries. Treat human intervention as durable workflow state. Test the recovery paths rather than assuming they work. Preserve enough evidence to reconstruct what the agent intended, attempted, changed, and verified.
These ideas come from decades of distributed systems, workflow orchestration, transaction processing, integration, and operational resilience. AI agents do not make those disciplines obsolete. They make them more important because the decision-maker controlling the workflow is now probabilistic and adaptive.
Once an agent begins changing the real world, durability, authority, state, and recovery must belong to the architecture—not to the model.