Back to blog

· By Sajeevan (Saj) Veeriah

Automation · 4 min read

When an automation times out, did the action happen?

Designing retries, operation records and human review for the uncertain gap between sending a request and seeing its result.

Illustrative lost-response sequence: request operation A creates one job; the reply is lost. Retrying operation A returns the existing result under an idempotent API contract, rather than creating another job.
Illustrative lost-response sequence: request operation A creates one job; the reply is lost. Retrying operation A returns the existing result under an idempotent API contract, rather than creating another job. View full-size diagram

An automation sends a request to create a job. The connection drops before a response arrives. The screen says the request timed out, but the receiving system may already have created the job.

This is an awkward state because both immediate retry and immediate abandonment can be wrong. The following design uses a fictional service-request workflow to explain how to make that uncertainty visible and recoverable. It is not a claim about a particular deployed client system.

A timeout describes what the caller knows

There are at least three possibilities: the receiver never got the request, it received the request but did not finish, or it finished and the response was lost. The caller's timeout does not distinguish them.

AWS's discussion of idempotent APIs uses this ambiguity to explain why retrying a request can create duplicate effects. An idempotent operation permits a repeat of the same intended request without repeating its effect. Whether an API provides that contract must be checked in its documentation.

Record an unknown outcome explicitly. If the interface only has success and failure, it encourages someone to turn uncertainty into a guess.

Sources: [1]

Give the intended action a stable identity

For the fictional job workflow, assign an operation identifier before the first attempt. Keep it for retries of that same intended action. A genuinely new request gets a new identifier even if its description happens to be identical.

The receiving service needs a defined duplicate-handling contract. Reusing an identifier with changed parameters should produce a conflict or another explicit outcome, not quietly reinterpret the earlier request. Also establish how long duplicate protection lasts.

Disabling the submit button helps prevent some accidental clicks. It does not handle a page reload, a second device, an automatic retry or a lost response. Those cases require the operation's identity to survive beyond one button's state.

Sources: [1]

Use states that tell the next person what to do

This proposed state model separates preparation, execution and uncertainty. An unknown outcome remains open until authoritative evidence resolves it.

Use states that tell the next person what to do
Recorded statePermitted next step
PreparedSubmit the validated request with its operation identifier.
In progressObserve or query the existing operation; avoid starting a duplicate.
SucceededDisplay the authoritative result and its reference.
Rejected before effectCorrect the stated input problem under the API's contract.
Outcome unknownReconcile by operation identifier or hold for review.

A status lookup should identify the operation, not merely search for a similar description. Two customers can request the same work. Two requests from one customer can also be legitimate.

Know where atomicity stops

SQLite documents atomic commit as all changes within a transaction occurring together or not occurring. That is useful when a local service records an operation and its associated local state change.

It does not make a database transaction and a remote side effect one indivisible event. If a workflow writes a local record and then calls a separate service, a crash can occur between those steps. Treat that boundary explicitly rather than assuming the word transaction covers both systems.

For a remote API with documented idempotency support, reuse its supported request identifier and reconcile its result. If no safe retry or status mechanism exists, a human review state may be the correct recovery path. An uncertain physical action needs a separate hazard assessment; retrying machinery is not equivalent to retrying a database lookup.

Sources: [2]

Test the lost-response case deliberately

In a controlled test double, let the receiver create the fictional job but drop the response. Retry using the same operation identifier. Inspect the receiver's records: exactly one intended job should exist, and the caller should recover its reference.

Next, use the same identifier with changed content. Check that the system reports the defined conflict. Then use a new identifier with identical content and verify that a genuinely new action is possible. These cases distinguish intent from superficial similarity.

Include restart and delayed-response cases. A late response from an old attempt should not overwrite the displayed result of a different operation. Keep attempt numbers separate from operation identity so logs explain both.

The acceptance evidence is the receiving system's resulting state and the caller's recovery behaviour. A log line saying retry successful is useful only when those observations agree.

Sources and further reading

Sources checked on 11 September 2026. The job workflow and fault-injection cases are proposed software examples, not measured production results.

  1. Amazon Builders' Library, Malcolm Featonby: Making retries safe with idempotent APIs; undated web article
  2. SQLite: Atomic Commit In SQLite; live documentation
Back to all posts