Classify the failure before restarting
A failed run may have done no work, saved a partial draft, or completed an external action before losing its response. Those states require different recovery steps. Start by checking the logs and the destination system.
A timeout after sending an email does not prove the email was never sent. Blindly restarting can duplicate messages, charges, records, or publications. If you cannot establish the action state, pause for human review.
- Input failure: validate or replace the missing input.
- Reasoning or output failure: retain the draft and record why it failed acceptance.
- Tool failure before action: retry within a documented limit.
- Uncertain external effect: inspect the destination before attempting again.
Give each run a stable identity
Record a run ID, input revision, instruction revision, start time, and output path. Where an API supports idempotency keys, follow that API’s documented semantics and reuse the same key for a retry of the same logical action.
Do not assume a local run ID makes an external action idempotent. If the destination has no duplicate protection, check its records and keep the final action behind a review gate.
run_id: briefing-2026-09-07
input_revision: approved-sources-v3
checkpoint: draft-saved
output: drafts/briefing-2026-09-07.md
external_action: not-started
retry_limit: 2
next_action: editor reviews draft before deliverySave useful checkpoints
A checkpoint is a durable record of a completed stage, not just a sentence in a chat. Save collected sources, intermediate data, and the final draft separately where appropriate. Validate the checkpoint before resuming; a corrupt intermediate file is not a reliable recovery point.
Keep retries bounded by both attempts and time or spend. Repeating the same prompt against the same broken input is unlikely to repair the problem. Escalate with the error, last safe checkpoint, and proposed next action.
Turn incidents into regression cases
After recovery, add a small test that reproduces the failure condition. Examples include an unavailable source, a duplicate trigger, an expired credential, or a malformed tool response. Remove real secrets and personal data from fixtures.
Rerun the case when you change scheduling, tools, or permission policy. Track recovery time and duplicated actions separately from normal run completion. A workflow that completes after extensive manual rescue still has an operational cost.
Sources and further reading
These guides combine linked documentation with our suggested operating practices. Examples are illustrative; they are not customer results or harness benchmarks.