# Retries, Duplicates and Idempotency: Evidence Dossier for Outcome-First Recovery Design

**Research snapshot:** 10 October 2026  
**Perspective:** Decision-maker — data/operations lead choosing control depth, ownership and staged rollout  
**Scope:** Small team operating an event/webhook-driven workflow into a cloud analytical warehouse and downstream business automation  
**Evidence boundary:** Public primary sources and synthetic fixtures only. No private systems, customer results or invented operational experience are used.

## Executive answer and uncertainties

### Decision

**Recovery should be designed and reported in terms of verified business outcomes, not “green” workflow runs.**

A successful webhook invocation, queue acknowledgment, workflow completion or warehouse write is useful operational evidence, but none of those facts alone establishes that the intended business action happened correctly. The classic end-to-end argument is unusually direct on this point: lower-level delivery acknowledgments cannot necessarily establish whether the target application actually performed the requested action; the required acknowledgement may have to come from the application that knows the intended result. HTTP makes a related distinction: `202 Accepted` explicitly means a request was accepted for processing, not that processing completed. [1](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf)

That distinction matters because current provider guarantees deliberately stop at different boundaries. Stripe documents duplicate and unordered webhook delivery; Google Pub/Sub's “exactly-once” guarantee has subscription, client-mode and regional boundaries; BigQuery's exactly-once semantics concern writes to an application-created Storage Write API stream; and Amazon SQS FIFO deduplication applies to queue submissions within its deduplication interval. None of these claims, individually, proves that an order was canceled, a customer was charged only once, a message was sent exactly once, or downstream systems agree on the final business state. [2](https://docs.stripe.com/webhooks)

For the scoped order workflow, I would therefore use this hierarchy:

| Evidence level | What it proves | What it does **not** prove |
|---|---|---|
| Transport accepted | The webhook/queue/API accepted something | The business operation happened |
| Durable work accepted | The event is committed to storage/queue from which it can be recovered | The event is current or should cause an action |
| Business state applied | The authoritative revision was reflected in the local order state | An external side effect succeeded |
| External effect confirmed | The provider reports the intended payment/message/etc. operation | All participating systems agree |
| Reconciled business outcome | Source state, warehouse state and required external effects satisfy the business invariant | Nothing stronger is implied than the invariant actually checked |

This hierarchy is an **editorial design model**, not a vendor standard. Its rationale follows the end-to-end principle and the documented limits of provider acknowledgments. [1](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf)

The recommended control pattern is:

> **Accept durably → classify identity and version → apply only meaningful current transitions → execute side effects with operation-level idempotency where available → verify the effect → reconcile end-to-end → quarantine ambiguity for a named human owner.**

A “green run” remains useful for debugging and availability. It should not be the terminal success metric.

### What is strongly supported

Several parts of this recommendation rest on unusually consistent primary evidence.

Providers explicitly document retries, duplicates and ambiguous failure. Stripe says webhook events can arrive more than once and not in generated order; its live-mode webhook delivery can retry for up to three days. AWS's design guidance points out that after a timeout the caller can be unable to tell whether a resource was created, making blind retries potentially duplicative. [2](https://docs.stripe.com/webhooks)

Idempotency also has boundaries. AWS recommends caller-provided request identifiers because identical parameters do not necessarily mean identical intent, and says recording the idempotency token plus associated mutations must form an atomic operation. Stripe API v1 retains the result of an idempotent operation once endpoint execution begins and documents that keys can be pruned after at least 24 hours; reusing a pruned key can start a new operation. [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/)

Provider “exactly-once” capabilities are useful but scoped. Pub/Sub limits exactly-once delivery to pull subscriptions and says the guarantee is regional; it also warns that publish-side duplicate publications can still yield multiple copies with different message IDs. BigQuery explicitly advises teams to consider whether exactly-once stream-offset management is necessary when upstream is at-least-once or post-ingestion duplicates are easy to detect. [4](https://cloud.google.com/pubsub/docs/exactly-once-delivery)

### What remains uncertain

The evidence base is much stronger on **mechanisms and semantics** than on **business effect sizes**. I did not find a primary empirical study establishing a general ROI, duplicate-error reduction percentage, recovery-time improvement or staffing threshold for small teams adopting idempotent event-processing architecture. The primary material is dominated by standards, provider documentation, provider-reported engineering practice and foundational systems-design research.

There is also no universal answer to how much control is worthwhile. AWS explicitly acknowledges that the stronger idempotency contract it describes adds cost and complexity and is not appropriate for every solution. BigQuery similarly documents a simpler default stream when exact offsets are unnecessary. [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/)

That makes the decision conditional:

| Situation | Default choice |
|---|---|
| Low volume, visible exceptions, cheap manual correction | Manual/native baseline |
| Repeated process, clear state model, important or expensive side effects | Deterministic automation with durable ledger and reconciliation |
| High exception-review burden after deterministic controls are working | Add bounded AI triage |
| No reliable business identity or authoritative source version | Do **not** increase autonomy yet |
| Irreversible/high-consequence effect with weak provider idempotency or observability | Human approval or stronger deterministic control |
| “Exactly-once” feature adds complexity but duplicate repair is cheap | Prefer simpler at-least-once + dedup/reconciliation |

For Czech versus US operation, the retry/idempotency mechanics in scope are properties of the protocols and providers, not country-specific architecture. This dossier therefore finds no reason to design different technical retry logic merely because a team is in Czechia versus the US. Privacy, retention, financial and consumer obligations can differ and may change how long audit data is retained or who may execute recovery, but those legal questions are outside this technical evidence scope.

## Bounded process, ownership and baseline

### The recognizable process

The bounded synthetic process is an e-commerce order feed whose source system emits order creation, update and cancellation events. Those events feed a cloud analytical warehouse; selected meaningful transitions can also trigger downstream business automation.

The process intentionally contains the failure patterns in the research scope.

| Element | Bounded definition |
|---|---|
| Trigger | HTTP webhook/event delivery from the order source |
| Business entity | `order_id`, for example `O-1042` |
| Transport/message identity | `source + event_id` |
| Source version | Monotonic `revision`, for example `1`, `2`, `3` |
| Operation intent | Separate value such as `set_order_state`, `send_cancel_notice`, `create_refund` |
| Inputs | Event ID, order ID, revision, event type/state, source timestamp, payload, origin/causation metadata |
| Business owner | Order Operations/Product owner — defines what “correctly completed” means |
| Technical owner | Data/Automation owner — owns ingestion, recovery controls, replay and observability |
| Recovery owner | Named on-call/operator who can quarantine, reconcile, retry or escalate |
| Warehouse outcome | Latest materialized order state equals the authoritative source revision |
| Downstream outcome | Every required external action is confirmed once at the **business-operation** level, or is explicitly compensated/waived |
| Manual fallback | Freeze outbound action, inspect source + ledger + provider reference, repair/replay under an audit record |

CloudEvents is useful for illustrating why these identities should remain separate. Its specification defines event identity using `source` plus `id`; an occurrence can result in more than one event, while `subject` may identify the business object the event concerns. Event identity is therefore not synonymous with order identity. [5](https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md)

Stripe's duplicate guidance exposes the same distinction in practical provider terms: logging an event ID handles repeated delivery of one Event object, but Stripe notes that separate Event objects can in some cases represent duplicate underlying activity and recommends looking at the object ID together with event type. [2](https://docs.stripe.com/webhooks)

### Proposed state model

The following state machine is an **editorial design**, not a provider-defined protocol:

```text
WEBHOOK_RECEIVED
      |
      v
DURABLY_ACCEPTED
      |
      v
VALIDATED
      |
      +---- invalid / contradictory ----> QUARANTINED
      |
      v
BUSINESS_STATE_APPLIED
      |
      v
WAREHOUSE_COMMITTED
      |
      +---- no downstream action required ---> RECONCILE
      |
      v
OUTBOUND_OPERATION_PENDING
      |
      +---- known failure -------------> RETRYABLE / QUARANTINED
      |
      +---- unknown outcome -----------> VERIFY_EXTERNAL_EFFECT
      |
      v
EXTERNAL_EFFECT_CONFIRMED
      |
      v
RECONCILED
      |
      v
VERIFIED_COMPLETE
```

`DURABLY_ACCEPTED` deserves to be a separate state from `WEBHOOK_RECEIVED`. Stripe recommends returning a successful HTTP response quickly, before complex downstream logic, so webhook endpoints do not unnecessarily hold delivery open. But a transport response is not itself durable storage. The safe editorial interpretation is: perform the minimum durable inbox/queue commit first, then return `2xx`, and process complex work asynchronously. Otherwise, an application can acknowledge receipt and crash before preserving the work, turning a superficially successful webhook into a lost business operation. Stripe's guidance supports asynchronous processing but does not prescribe this particular internal transaction boundary. [2](https://docs.stripe.com/webhooks)

### Observable completion

For synthetic order `O-1042`, suppose the source's authoritative latest revision is:

```json
{
  "order_id": "O-1042",
  "revision": 3,
  "state": "CANCELED"
}
```

The business outcome is **not** “workflow run 91842 is green.”

A suitable outcome invariant might instead be:

```text
source(O-1042).revision = 3
AND source(O-1042).state = CANCELED
AND warehouse(O-1042).revision = 3
AND warehouse(O-1042).state = CANCELED
AND no downstream operation exists whose intent conflicts with cancellation
AND every required cancellation-related operation is either:
    CONFIRMED,
    COMPENSATED,
    or HUMAN_WAIVED with actor + reason
```

This is synthetic, but the architectural principle is supported by the end-to-end paper's distinction between intermediate delivery acknowledgement and confirmation that the target application actually performed the requested operation. [1](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf)

The useful top-level metrics then become things such as `verified_complete / accepted_business_operations`, age of the oldest unreconciled accepted item, number of unresolved external-outcome ambiguities, number of stale-event suppressions, number of pending compensations and time from detection to verified repair. Thresholds for those measures are business choices, not universal reliability constants.

### Simplest manual/native baseline

The baseline should be deliberately boring:

The source system remains the order system of record. A native export, scheduled warehouse load or source connector populates analytical data. An operator uses the source administration UI and provider dashboards/logs to inspect cancellations or discrepancies. Any customer message, payment repair or correction requiring judgment is performed manually. A recurring source-versus-warehouse report exposes missing or inconsistent orders.

This approach has weak latency and can consume human time, but it avoids a custom retry engine, deduplication store, operation ledger, loop controller and AI layer. For low volume or rapidly changing business rules, that can be the rational design. AWS explicitly cautions that stronger idempotency contracts add implementation complexity and are not right for all solutions, while BigQuery advises that exactly-once offsets need not be used where simpler duplicate detection is adequate. [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/)

The key point is that the manual baseline must still define the same outcome. Comparing a manual process's verified business completion to an automated workflow's “green runs” would be an apples-to-oranges evaluation.

## Evidence synthesis around recovery design

**Transport delivery and business completion are different claims.** RFC 9110 defines `202 Accepted` as acceptance for processing rather than completed processing. Saltzer, Reed and Clark's original end-to-end design paper goes further: for a requested application action, the meaningful acknowledgment may be one generated at the application endpoint indicating whether it actually did the work. Their paper explicitly covers duplicate message detection, sequencing, guaranteed delivery and recovery, while warning that lower-layer mechanisms can be useful performance enhancements without replacing end-to-end correctness checks. [6](https://www.rfc-editor.org/rfc/rfc9110.html)

This is the strongest conceptual evidence behind the topic decision. **“Green run” is an intermediate acknowledgment unless its success condition itself includes verification of the business invariant.**

**Retries are safe only relative to an operation's semantics.** AWS describes the classic timeout ambiguity: a resource-creation request may have succeeded even when the caller received no response, so blindly creating again can produce two resources. Its recommended API pattern is a caller-generated request identifier that states intent. AWS specifically rejects the idea that equal request parameters always imply duplicate intent, because two identical EC2 creation requests might legitimately mean “create two instances.” [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/)

That is why deduplication keys should not be casually derived from an entire payload hash. A payload hash can detect byte-level repetition, but business intent should be represented explicitly. In this dossier the useful distinction is:

```text
message identity      = source + event_id
business identity     = order_id
source version        = revision
operation intent      = send_cancel_notice
business operation ID = order_id + revision + operation_intent
provider retry key    = idempotency key for that business operation
```

The first protects against transport redelivery. The fifth protects against repeating the same business side effect. They solve different problems. AWS's warning about identical parameters with different intent directly supports that separation. [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/)

**Idempotency keys are not eternal memory.** Stripe API v1 documents that an idempotency result is stored once endpoint execution begins, including a resulting `500`, and that a key can be removed after it is at least 24 hours old; a reused key after pruning can initiate a new request. Stripe also compares parameters and rejects a reused key whose parameters differ. AWS says retention requirements vary by service and describes an EC2 strategy based on resource lifetime plus an interval covering plausible late requests. [7](https://docs.stripe.com/api/idempotent_requests)

The practical implication is important: **the application's own business-operation ledger should live for the full replay and reconciliation horizon, which may be substantially longer than a provider's idempotency-key cache**. That is an editorial inference from the documented finite provider windows.

**The idempotency boundary must be atomic where possible.** AWS explicitly says the operation combining the recording of the idempotency token and its associated server-side mutations needs ACID properties; otherwise one can record the token without performing the work, or perform the work without recording the token. [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/)

For a local database transaction, that can often be done directly. For a transaction spanning the local database and an external message/payment API, a single ACID transaction is normally not available. The practical substitute is a durable operation ledger/outbox, stable operation identity, provider-supported idempotency where available, external-effect verification and compensation/reconciliation for ambiguity. This is an architectural interpretation, not an AWS guarantee.

**“Exactly-once” must always be followed by “exactly once where?”** SQS standard queues document at-least-once delivery and tell consumers to be idempotent. SQS FIFO suppresses duplicate sends inside its documented five-minute deduplication interval. Google Pub/Sub's exactly-once delivery concerns deliveries on supported pull subscriptions, with a same-region requirement; the documentation explicitly notes publish-side duplicates and additional latency/quota considerations. BigQuery can provide exactly-once *write semantics* using offsets on application-created streams, but its documentation recommends considering the simpler default stream when duplicates can be detected elsewhere. [8](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html)

So an architecture can simultaneously have:

```text
exactly-once Pub/Sub delivery
+ exactly-once BigQuery stream writes
+ a duplicate customer email
```

without either provider violating its documented contract. The email call lies beyond both guarantees. That is precisely why end-to-end business verification is required. [4](https://cloud.google.com/pubsub/docs/exactly-once-delivery)

**Out-of-order events require business versions or reconciliation, not arrival timestamps.** Stripe explicitly says webhook event ordering is not guaranteed. When required related objects have not yet arrived, Stripe advises retrieving them rather than assuming webhook arrival order expresses business order. [2](https://docs.stripe.com/webhooks)

Accordingly, “cancellation arrived before creation” should not be solved with “whichever webhook arrived last wins.” If the source provides an authoritative monotonic revision, the higher revision should win for that entity. If it does not, an irreversible action should be based on a fresh source-of-truth read or held for reconciliation. That latter rule is an editorial safety design, conditional on the source API's semantics.

**Rate limiting is both a retry problem and a business-priority problem.** RFC 6585 defines `429 Too Many Requests` and allows the server to include `Retry-After`; the RFC intentionally does not define how a server counts requests. RFC 9110 defines the `Retry-After` field as a time or delay after which a subsequent request should be attempted. [9](https://www.rfc-editor.org/rfc/rfc6585.html)

Backoff therefore answers “when should I try again?” It does not answer “which accepted business operation is allowed to wait?” Google Pub/Sub's flow-control documentation gives the lower-level mechanism: bound outstanding messages/bytes so publishing does not outrun available throughput. It explicitly distinguishes transient spikes from persistent capacity problems. [10](https://cloud.google.com/pubsub/docs/flow-control-messages)

The business layer must add deadlines and priority. A warehouse enrichment job might tolerate an hour. A cancellation intended to prevent fulfillment might not. Those deadlines are process requirements, not properties of HTTP `429`.

**Retries can worsen dependency overload.** AWS's Builders' Library warns that retries consume additional downstream capacity and can amplify overload; it recommends bounded retries, retrying at an appropriate layer, and jitter to spread synchronized retries. AWS describes token-bucket retry throttling in its SDKs and notes that side-effecting operations need idempotency if they are to be safely retried. This is provider-reported engineering practice, not a universal experimental law. [11](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/)

**A dead-letter queue is a recovery inbox, not a business-completion state.** Google Pub/Sub retries unacknowledged work and can forward messages to a dead-letter topic after an approximately configured number of attempts. Google explicitly says that maximum attempt count is best effort and may be exceeded or undershot; the dead-letter subscription exists for analysis and debugging. [12](https://cloud.google.com/pubsub/docs/dead-letter-topics)

A dashboard showing “main queue empty” while the dead-letter queue contains a canceled order is therefore not green in business terms.

**Compensation is not magical rollback.** Microsoft's current compensating-transaction guidance says a failed multi-step eventually consistent operation may require recorded undo actions, but compensation is application-specific, can itself fail and sometimes requires manual intervention. It also cautions that compensation need not simply reverse operations in exact reverse order. [13](https://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction?utm_source=chatgpt.com)

That distinction is especially important for messages. A duplicate monetary reservation may have a refund or release operation. A duplicate customer email cannot literally be unsent; “compensation” might mean a corrective message or human contact. Recovery design must model the actual business effect.

**Feedback-loop prevention is a separate control from duplicate delivery.** GitHub provides a concrete provider-specific example: workflow events generated using the repository's `GITHUB_TOKEN` normally do not create new workflow runs, with documented exceptions, specifically to prevent recursive workflow runs. That proves that platform-level recursion suppression is a real design mechanism, but it does **not** establish a general solution for arbitrary business automations. [14](https://docs.github.com/en/actions/concepts/security/github_token)

For arbitrary workflows, provenance such as `origin`, `causation_id`, `root_correlation_id`, semantic no-op detection and a finite hop budget are safer than assuming “event ID deduplication” will stop a loop: each iteration in a loop can be a perfectly new event.

## Failure modes, worked timelines and safe repair

The following examples are **synthetic and reasoned**. They are not reports of customer incidents or provider tests.

### Synthetic end-to-end order

Assume the authoritative source history is:

```text
revision 1  CREATED
revision 2  UPDATED
revision 3  CANCELED
```

The observable desired final state is:

```text
O-1042 = revision 3 / CANCELED
warehouse agrees
no fulfillment-start operation remains pending
required cancellation message is confirmed or explicitly waived
all accepted events are processed, quarantined or superseded
```

### Duplicate delivery

```text
10:00:00  evt-100 / O-1042 / rev=1 / CREATED arrives
10:00:01  inbox insert commits
10:00:02  state becomes CREATED r1
10:00:03  webhook delivery acknowledged

10:00:15  evt-100 / O-1042 / rev=1 / CREATED arrives again
10:00:15  UNIQUE(source,event_id) finds existing record
10:00:15  return success; no state transition; no side effect
```

Stripe explicitly says webhook endpoints can receive the same Event more than once and recommends remembering processed event IDs. Standard SQS similarly documents possible redelivery and recommends idempotent consumers. [2](https://docs.stripe.com/webhooks)

**Safe repair criterion:** there is one accepted business transition and zero additional side effects. A duplicate-delivery counter increments; the order outcome is unchanged.

### Different messages representing the same business operation

```text
10:00  event A / O-1042 / rev=3 / CANCELED
10:01  event B / O-1042 / rev=3 / CANCELED
```

`event A != event B`, so transport-ID deduplication alone does not suppress B.

Business operation identity:

```text
(order_id=O-1042, revision=3, intent=set_order_state)
```

does suppress the repeated semantic transition.

This distinction is consistent with Stripe's warning that distinct Event objects can sometimes describe the same underlying object activity and AWS's warning that request semantics—not superficial parameter identity—should determine idempotency. [2](https://docs.stripe.com/webhooks)

**Safe repair criterion:** the second event remains auditable but produces no second business transition.

### Cancellation arrives before creation

```text
10:00:00  evt-cancel / O-1042 / revision=3 / CANCELED arrives
10:00:01  local latest := CANCELED r3

10:02:00  evt-create / O-1042 / revision=1 / CREATED arrives late
10:02:01  revision 1 < current revision 3
10:02:01  record event as STALE
10:02:01  do not regress order to CREATED
```

Stripe establishes the relevant premise: webhook delivery order is not guaranteed. [2](https://docs.stripe.com/webhooks)

**Safe repair criterion:** the materialized business state remains revision 3. The late event is retained for audit/replay evidence but causes no outbound action.

If the source has **no trustworthy monotonic revision**, the recommendation weakens. Before an irreversible effect, query the source-of-truth object. If that lookup is unavailable or ambiguous, quarantine instead of guessing.

This directly answers the proposed article claim **“The cancellation arrived first, then the confirmation. Which state should the automation use?”**: use the authoritative business version/current source state, not arrival order. The advice is conditional on having trustworthy source-version semantics.

### Acknowledged but lost

Unsafe ordering:

```text
10:00:00.000  source POSTs event
10:00:00.020  receiver returns HTTP 200
10:00:00.025  process crashes
10:00:00.030  inbox write never happened
```

The provider may reasonably treat the delivery as complete because it received a successful response. The business work disappeared.

Safer ordering:

```text
10:00:00.000  source POSTs event
10:00:00.020  INSERT inbox(...); COMMIT
10:00:00.030  receiver returns HTTP 200
10:00:01.000  asynchronous worker starts processing
```

Stripe recommends quick successful webhook responses before complex work; Saltzer et al. show why intermediate delivery acknowledgment is not equivalent to application completion. The “commit before `200`” sequencing here is an editorial design derived from those constraints. [2](https://docs.stripe.com/webhooks)

**Safe repair criterion:** every successful webhook acknowledgement corresponds to either a durable accepted-work record or a deliberately rejected event. There is no `2xx` path that can silently discard accepted work.

### Timeout after an external side effect

Consider a synthetic payment operation:

```text
operation_id = O-1042:r2:create_payment
idempotency_key = O-1042:r2:create_payment
amount = EUR 100.00
```

Timeline:

```text
10:00:00  local operation ledger -> PENDING
10:00:01  POST create-payment with idempotency key
10:00:02  provider performs side effect
10:00:03  response is lost / client times out
10:00:04  local system knows neither SUCCESS nor FAILURE
```

The wrong response is:

```text
"Timeout -> send a new create-payment request with a new key."
```

That creates new intent.

Where provider semantics support it, the correct path is to retry/lookup using the original operation identity, recover the provider reference and confirm outcome. AWS's retry-safety discussion centers on exactly this “request may have succeeded although the response was lost” ambiguity. Stripe supports idempotent POSTs, though its API v1 semantics include the important detail that once execution begins it returns the saved result for the same key, including saved `500` responses. [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/)

If the provider cannot prove whether the effect occurred, the state is not `FAILED`; it is **UNKNOWN_OUTCOME**.

**Safe repair criterion:**

```text
provider confirms action exists -> mark CONFIRMED, do not resend
provider confirms action absent -> retry same business operation safely
provider remains ambiguous -> human/reconciliation path
duplicate effect discovered -> compensate according to business rules
```

For a non-idempotent message API, the operation ledger can prevent your own worker from intentionally sending twice, but a timeout after the provider accepted the message may still leave irreducible ambiguity. That should be exposed explicitly rather than hidden behind a green retry.

### Rate-limited queue

Synthetic timeline:

```text
12:00:00  cancellation job C accepted; business deadline 12:05
12:00:01  analytics enrichment A accepted; deadline 18:00
12:00:02  external API returns 429 Retry-After: 60
12:00:03  both jobs remain in durable queue

12:01:02  retry window opens + jitter
           C dispatched first because its deadline is nearer
           A remains queued

12:04:00  C still cannot complete
           alert/escalate before business deadline

12:15:00  A can still safely wait
```

RFC 6585 establishes the meaning of `429`; RFC 9110 establishes `Retry-After`. Google Pub/Sub documents flow control for bounding outstanding requests/messages rather than allowing publication pressure to grow without bound. [9](https://www.rfc-editor.org/rfc/rfc6585.html)

**Safe repair criterion:** accepted work remains durable while throttled; retry traffic does not exceed the chosen backpressure policy; jobs whose business deadlines cannot be met are escalated rather than silently aging.

This answers the proposed article claim **“The system limits the number of requests. What can wait and what must not be lost?”**: transport retry policy tells the system *when* it may try again; the business process must separately define durability, priority and deadline.

Useful measurements are:

```text
oldest_accepted_work_age
accepted_work_past_deadline
queue_depth by priority
retry_attempt_rate
429 rate
time_to_verified_completion
```

A universal “maximum queue age = X minutes” would be invented. The limit has to come from the business consequence.

### Two automations trigger each other

Synthetic bad loop:

```text
A: order.updated  -> writes CRM.status
B: CRM.status     -> writes order.tag
A: order.updated  -> writes CRM.status
B: CRM.status     -> writes order.tag
...
```

Every event can have a unique message ID. Pure transport deduplication therefore does nothing.

Proposed loop metadata:

```json
{
  "origin": "automation-B",
  "causation_id": "event-7821",
  "root_correlation_id": "root-551",
  "hop_count": 3
}
```

Proposed controls, in priority order:

1. **Semantic no-op:** do not write a value that already has the desired business meaning.
2. **Origin filter:** an automation ignores changes produced solely by itself where appropriate.
3. **Causation/root tracking:** identify that apparently new events belong to the same chain.
4. **Hop/time budget:** quarantine a chain that still oscillates rather than running indefinitely.
5. **Human resolution:** decide which system owns the contested field.

GitHub demonstrates one provider-specific implementation of recursion suppression by generally preventing events produced using `GITHUB_TOKEN` from triggering another workflow run. It should be treated as an example, not as a portable guarantee for unrelated automation platforms. [14](https://docs.github.com/en/actions/concepts/security/github_token)

**Safe repair criterion:** terminating a loop must leave a known business state. “We stopped after ten hops” is protection against resource consumption; it is not proof the correct system won.

This answers **“Two automations trigger each other. How do you stop the infinite loop?”**: first remove semantic self-triggering, then carry provenance, and use a hop budget as a circuit breaker—not as the primary correctness rule.

### Replay and race conditions

Suppose a historical ingestion replay sends revisions `1`, `2` and `3` again.

A safe replay should normally be permitted to reconstruct analytical data without replaying historical customer-facing effects. A useful explicit attribute is:

```text
processing_mode = LIVE | REPLAY | REPAIR
```

and downstream business-action rules can require `LIVE` or an individually approved `REPAIR`.

BigQuery documents offsets as one way to obtain exactly-once stream writes, but also explicitly recommends asking whether offsets are necessary when duplicates can be identified after ingestion. [15](https://cloud.google.com/bigquery/docs/write-api-best-practices)

A race between two workers processing revision 2 and revision 3 requires the business-state write to be guarded atomically, for example conceptually:

```sql
UPDATE order_state
SET revision = 3, state = 'CANCELED'
WHERE order_id = 'O-1042'
  AND revision < 3;
```

The exact SQL depends on storage technology. The principle is that an older writer must not be able to overwrite a newer accepted version.

**Safe repair criterion:** replaying any accepted historical event sequence leaves the same latest business state and creates no unintended new outbound operation.

### Dependency unavailable and dead-letter handling

A dependency outage should move work through bounded retries and then into an explicitly owned recovery state rather than disappearing. Google's Pub/Sub dead-letter documentation is particularly useful here because it warns that the configured delivery-attempt limit is approximate, and describes the dead-letter subscription as a place for analysis/debugging. [12](https://cloud.google.com/pubsub/docs/dead-letter-topics)

Proposed terminal categories:

```text
VERIFIED_COMPLETE
SUPERSEDED
COMPENSATED
HUMAN_WAIVED
QUARANTINED_REQUIRES_OWNER
```

`DEAD_LETTERED` should **not** be a business terminal category; it describes where the technical message currently resides.

## Automation variants, lifecycle economics and staged rollout

### Same process, three operating models

| Dimension | Manual/native | Deterministic automation | Bounded AI triage |
|---|---|---|---|
| Source of correctness | Human checks source/provider state | Explicit state machine + ledgers + reconciliation | Same deterministic foundation |
| Side effects | Human performs them | Deterministic executor performs allowed operations | AI **cannot** send messages, authorize payment or write production state |
| Duplicate protection | Human recognition/native provider controls | Message dedup + business-operation idempotency | Unchanged |
| Out-of-order handling | Human inspects current source state | Source revision/current-state reconciliation | AI may summarize why item is ambiguous |
| Recovery | Human from start | Automated for known-safe classes; human for ambiguity | AI proposes classification/repair plan; human decides |
| Main upkeep | Operator time/training | Code, schemas, provider contracts, runbooks, monitoring | All deterministic upkeep + model/evaluation/prompt/version upkeep |
| False-positive consequence | Human investigates unnecessarily | Depends on automated rule | Additional human review; no direct business write |
| Best fit | Low scale/high variability | Repeated, well-specified process | High enough exception-review burden to justify extra layer |

The recommendation is **not** to start with an agent and make recovery deterministic later. Idempotency, durable work, source versioning and operation identity are correctness mechanisms that an AI classifier does not eliminate.

### Lifecycle maintenance deserves first-class treatment

BigQuery's documentation gives one concrete illustration of provider/API drift: schema updates can take time to propagate to Storage Write API connections, and configurations that allow unknown fields to be dropped can lead to unintended data loss. This is a provider-specific example of why “the workflow still runs” is not sufficient maintenance evidence. [15](https://cloud.google.com/bigquery/docs/write-api-best-practices)

The minimum recurring maintenance budget should therefore cover:

```text
source event contract changes
schema evolution
idempotency-window changes
API-version/provider behavior changes
retry/quota changes
credential/auth changes
reconciliation drift
runbook testing
manual-override testing
dependency retirement/replacement
AI evaluation/version drift, if AI is present
```

The provider documents also change over time: for example, the Pub/Sub exactly-once and flow-control pages inspected here were last updated on 9 October 2026, one day before this research snapshot. [4](https://cloud.google.com/pubsub/docs/exactly-once-delivery)

### Synthetic economics

The following numbers are **not observed customer economics**. They are a transparent decision model showing how to evaluate whether additional automation is worthwhile.

Assumptions:

```text
N = business orders per month
exception rate before automation = 3% of orders
manual review time = 4 minutes / exception
loaded human cost = €60 / hour

deterministic automation:
  initial build = 60 engineering hours
  evaluation horizon = 12 months
  amortized build = 5 hours/month
  recurring maintenance/recovery = 8 hours/month
  platform cost = €150/month
  deterministic rules eliminate 85% of manual exception reviews
  residual review rate = 15% × 3% = 0.45% of orders

AI triage incremental:
  model/service cost = €100/month
  evaluation/calibration/maintenance = 4 hours/month
  time saved on each residual human review = 2 minutes
```

Manual cost per exception:

\[
4\text{ min} / 60 \times €60/\text{h} = €4
\]

Manual monthly recovery cost:

\[
C_{manual} = 0.03N \times €4 = €0.12N
\]

Deterministic fixed monthly cost:

\[
(5 + 8)\text{ h} \times €60/\text{h} + €150
= €930
\]

Residual human cost:

\[
0.0045N \times €4 = €0.018N
\]

Therefore:

\[
C_{deterministic} = €930 + €0.018N
\]

Synthetic break-even against manual:

\[
€0.12N = €930 + €0.018N
\]

\[
N = \frac{€930}{€0.102/\text{order}}
\approx 9{,}118\text{ orders/month}
\]

This is **not a recommendation that 9,118 is a universal threshold**. Change any assumption—labor cost, build effort, exception frequency, consequence of one duplicate payment—and the result changes.

AI adds:

\[
€100 + 4\text{ h}\times€60/\text{h} = €340/month
\]

Saving two minutes per residual review is:

\[
2/60\text{ h}\times€60/\text{h}=€2/\text{review}
\]

There are `0.0045N` residual reviews, so:

\[
AI\ savings = €0.009N
\]

AI triage breaks even against deterministic-only operation when:

\[
€0.009N = €340
\]

\[
N \approx 37{,}778\text{ orders/month}
\]

Equivalent formulation:

\[
€340 / €2 = 170\text{ residual reviews/month}
\]

Under these synthetic assumptions, adding AI at 10,000 orders/month would be economically worse even though AI might be technically impressive.

| Synthetic monthly order volume | Manual | Deterministic | Deterministic + AI triage |
|---:|---:|---:|---:|
| 3,000 | €360 | €984 | €1,297 |
| 10,000 | €1,200 | €1,110 | €1,360 |
| 30,000 | €3,600 | €1,470 | €1,540 |
| 50,000 | €6,000 | €1,830 | €1,720 |

At the illustrative 10,000 orders/month, recovery/control cost per verified order—assuming all 10,000 are ultimately reconciled—would be:

```text
manual          €1,200 / 10,000 = €0.120
deterministic   €1,110 / 10,000 = €0.111
AI-assisted     €1,360 / 10,000 = €0.136
```

The model deliberately excludes the monetary consequence of an unrecoverable duplicate payment, missed cancellation, customer complaint or compliance failure because no defensible generic value was found. Where a single error is expensive, an expected-loss term should be added:

\[
C_{total}
=
C_{build}
+
C_{maintenance}
+
C_{infra}
+
C_{human}
+
\sum_i P(failure_i)\times Impact_i
\]

Those probabilities and impacts must come from the team's own incidents, shadow testing or business estimates—not from this dossier.

### Proposed staged rollout

I would stage this process in four control-depth steps.

**Baseline instrumentation first.** Run the manual/native process long enough to count accepted orders, exceptions, stale events, duplicates, reconciliation mismatches, manual recovery time and business deadlines. Do not infer automation ROI before there is a denominator.

**Deterministic shadow mode second.** Ingest events durably, compute the intended state and operation ledger, but do not execute customer-facing side effects. Compare the calculated state against the source/native outcome.

**Controlled deterministic side effects third.** Enable only operations with explicit business identity, understood retry semantics and a tested recovery path. High-consequence or ambiguous operations remain human-approved.

**Bounded AI triage last.** Give the AI read-only access to quarantined event history, source snapshots and runbook documentation. It may propose “stale,” “probable duplicate,” “source lookup required” or a human repair sequence. It may not authorize a payment, send a customer message or directly write production state.

Proposed go/no-go criteria are design choices rather than industry thresholds: every seeded duplicate/out-of-order test should converge to the intended business state; replay must not regenerate live side effects; timeout ambiguity must enter `UNKNOWN_OUTCOME` rather than automatically become `FAILED`; every dead-letter item must have a named owner; and reconciliation must detect deliberately seeded source-versus-warehouse mismatches.

A sensible stop/retirement criterion is equally important. Retire or simplify the automation when the source offers a reliable native integration that makes custom recovery redundant, when business rules change faster than safe automation can be maintained, when monthly maintenance is persistently greater than verified operational benefit, or when authoritative state/version information becomes unavailable.

## Smallest workable implementation and portable test asset

### Minimum architecture

The smallest deterministic design I would consider production-worthy for the scoped process is:

```text
                     +--------------------+
Webhook ------------>| durable event inbox|----+
                     +--------------------+    |
                                                v
                                         +-------------+
                                         | state reducer|
                                         +-------------+
                                           |         |
                                      valid|         |ambiguous
                                           v         v
                                  +------------+   QUARANTINE
                                  | order state |       |
                                  +------------+        v
                                       |           HUMAN OWNER
                                       v
                                  +-----------+
                                  | warehouse |
                                  +-----------+
                                       |
                         meaningful live operation?
                                |              |
                               no             yes
                                |              v
                                |       +----------------+
                                |       |operation ledger|
                                |       +----------------+
                                |              |
                                |              v
                                |       external provider
                                |              |
                                +-------+------+
                                        v
                                  RECONCILIATION
                                        |
                                        v
                                VERIFIED_COMPLETE
```

Its minimum persistence model is three conceptual tables:

```text
event_inbox
  source
  event_id
  order_id
  revision
  event_type
  payload
  received_at
  processing_status
  UNIQUE(source, event_id)

order_state
  order_id PRIMARY KEY
  source_revision
  business_state
  source_updated_at
  locally_updated_at

operation_ledger
  order_id
  source_revision
  operation_intent
  idempotency_key
  status
  provider_reference
  last_error
  attempts
  updated_at
  UNIQUE(order_id, source_revision, operation_intent)
```

The first uniqueness constraint addresses transport duplicates. The last addresses repeated execution of the same business intent. They deliberately do not share an identity.

For external providers, the ledger should persist longer than the provider's relevant retry/idempotency horizon. Stripe API v1's documented ability to prune keys after at least 24 hours is a concrete reason not to treat the provider's cache as the application's permanent business ledger. [7](https://docs.stripe.com/api/idempotent_requests)

The corresponding recovery ownership should be explicit:

| State | Automatic action | Human responsibility |
|---|---|---|
| Duplicate transport event | Suppress | None unless abnormal rate |
| Stale lower revision | Retain, no-op | None normally |
| Same revision, contradictory state | Quarantine | Check authoritative source |
| External outcome known successful | Mark confirmed | None |
| External outcome known absent | Safe retry | None if retry policy allows |
| External outcome ambiguous | Stop blind retries | Investigate provider/source |
| DLQ/quarantine beyond deadline | Alert | Repair, compensate or waive |
| Feedback-loop budget exhausted | Disable chain/quarantine | Determine correct owner/state |

The observability dashboard should put business state before orchestration state. A useful proposed top row is:

```text
accepted business operations
verified complete
unreconciled
past business deadline
unknown external outcome
compensation pending
quarantined without owner
oldest unreconciled age
```

Then, underneath it, show technical diagnostics such as webhook success rate, workflow failure rate, retries, `429`s, dedup hits and queue depth.

### Portable reducer fixture

The following is the complete synthetic test asset used in this research. It tests transport duplicates, a separate event expressing the same business transition, late revisions and same-revision conflicting state.

**Execution status:** executed locally in this research session using Python. Assertions passed.  
**Not tested:** live Stripe, AWS, Google Cloud or other provider APIs; actual network failures; database transaction isolation; concurrency between real worker processes.

```python
from dataclasses import dataclass
from typing import Dict, Set, Tuple


@dataclass(frozen=True)
class Event:
    event_id: str
    order_id: str
    revision: int
    state: str
    intent: str


class Reducer:
    def __init__(self) -> None:
        self.current: Dict[str, Tuple[int, str]] = {}
        self.seen_events: Set[str] = set()
        self.seen_transitions: Set[Tuple[str, int, str, str]] = set()

    def apply(self, e: Event) -> str:
        # Transport identity: same delivery/event ID.
        if e.event_id in self.seen_events:
            return "transport_duplicate"
        self.seen_events.add(e.event_id)

        # Business identity: same entity revision + semantic operation.
        business_key = (e.order_id, e.revision, e.state, e.intent)
        if business_key in self.seen_transitions:
            return "business_duplicate"

        current = self.current.get(e.order_id)
        if current is not None:
            current_rev, current_state = current

            if e.revision < current_rev:
                return "stale"

            if e.revision == current_rev:
                if e.state == current_state:
                    self.seen_transitions.add(business_key)
                    return "business_duplicate"

                return "conflict_quarantine"

        # Synthetic assumption:
        # source revision is authoritative and monotonically increasing.
        self.current[e.order_id] = (e.revision, e.state)
        self.seen_transitions.add(business_key)
        return "applied"


fixtures = [
    Event(
        "evt-cancel-r3",
        "O-1042",
        3,
        "CANCELED",
        "set_order_state",
    ),
    Event(
        "evt-create-r1",
        "O-1042",
        1,
        "CREATED",
        "set_order_state",
    ),
    Event(
        "evt-cancel-r3",
        "O-1042",
        3,
        "CANCELED",
        "set_order_state",
    ),
    Event(
        "evt-cancel-r3-copy",
        "O-1042",
        3,
        "CANCELED",
        "set_order_state",
    ),
    Event(
        "evt-update-r2",
        "O-1042",
        2,
        "UPDATED",
        "set_order_state",
    ),
    Event(
        "evt-conflict-r3",
        "O-1042",
        3,
        "UPDATED",
        "set_order_state",
    ),
]

expected = [
    "applied",
    "stale",
    "transport_duplicate",
    "business_duplicate",
    "stale",
    "conflict_quarantine",
]

reducer = Reducer()
actual = [reducer.apply(e) for e in fixtures]

assert actual == expected, (actual, expected)
assert reducer.current["O-1042"] == (3, "CANCELED")

print(list(zip([e.event_id for e in fixtures], actual)))
print(reducer.current)
```

Executed result:

```text
[
  ('evt-cancel-r3', 'applied'),
  ('evt-create-r1', 'stale'),
  ('evt-cancel-r3', 'transport_duplicate'),
  ('evt-cancel-r3-copy', 'business_duplicate'),
  ('evt-update-r2', 'stale'),
  ('evt-conflict-r3', 'conflict_quarantine')
]

{'O-1042': (3, 'CANCELED')}
```

The fixture proves only the behavior of this small in-memory reducer. It does not prove database atomicity, provider idempotency or end-to-end correctness. In particular, the assumption that source revision is authoritative is deliberately visible in the code; removing that assumption requires source lookup/reconciliation rather than blindly applying the reducer.

## Claim-to-source dossier and evidence quality

The table distinguishes standards, provider documentation, provider-reported design practice, original research/design literature and editorial inference. “Supported” means the source directly supports the bounded claim, not that it establishes a universal business effect.

| Claim or draft target | Primary source and exact scope | Source date / inspected | Evidence type | Assessment | Editorial action |
|---|---|---|---|---|---|
| **A successful intermediate run/ack does not prove the application did the requested business work.** | Saltzer, Reed & Clark, *End-to-End Arguments in System Design*, “Delivery guarantees” and conclusions. DOI `10.1145/357401.357402`; `https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf` [1](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf) | Nov 1984 / 2026-10-10 | Original systems-design paper | **Supported** | **Keep.** This is the core conceptual claim. |
| **HTTP acceptance can be weaker than completion.** | RFC 9110 §15.3.3, `202 Accepted`; `https://www.rfc-editor.org/rfc/rfc9110.html` [6](https://www.rfc-editor.org/rfc/rfc9110.html) | Jun 2022 / 2026-10-10 | Internet standard | **Supported** | **Keep**, but do not imply every workflow uses HTTP 202. |
| **429 tells you rate limiting occurred; retry timing can be signaled with Retry-After, but business priority is separate.** | RFC 6585 §4, `429 Too Many Requests`; `https://www.rfc-editor.org/rfc/rfc6585.html`; RFC 9110 §10.2.3 `Retry-After`. [9](https://www.rfc-editor.org/rfc/rfc6585.html) | Apr 2012 / Jun 2022; inspected 2026-10-10 | Internet standards | **Supported / conditional** | **Keep** transport claim; **qualify** prioritization as application design. |
| **Webhook delivery may duplicate and arrive out of order.** | Stripe Webhooks, “Event ordering,” “Handle duplicate events,” “Automatic retries”; `https://docs.stripe.com/webhooks` [2](https://docs.stripe.com/webhooks) | Living documentation, publication date not shown / 2026-10-10 | Provider documentation | **Supported for Stripe** | **Keep** as concrete example; do not universalize Stripe's retry schedule. |
| **Transport event ID and underlying business identity can differ.** | Stripe “Handle duplicate events” says separate Event objects can represent duplicate underlying activity; CloudEvents `id`, `source`, `subject`; `https://docs.stripe.com/webhooks`; `https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md` [2](https://docs.stripe.com/webhooks) | Stripe living; CloudEvents main showed 1.0.3-wip / 2026-10-10 | Provider docs + open specification | **Supported** | **Keep.** Strong basis for separate message/business identity. |
| **Hashing identical parameters is not a universally safe way to infer same intent.** | AWS Builders' Library, “Reducing client complexity with idempotent API design”; `https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/` [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) | Undated living page / 2026-10-10 | Attributable provider engineering practice | **Contradicts simplistic dedup** | **Rewrite** advice around caller/business intent. |
| **Recording an idempotency token and the associated mutation requires an atomic boundary.** | Same AWS article, lines describing ACID/all-or-nothing idempotency session + mutation. `https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/` [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) | Undated living page / 2026-10-10 | Provider design practice | **Supported** | **Keep**, noting external side effects require another pattern. |
| **Idempotency keys have retention semantics; do not assume forever.** | Stripe API Reference, “Idempotent requests,” API v1 behavior; `https://docs.stripe.com/api/idempotent_requests`; AWS “Late arriving requests.” [7](https://docs.stripe.com/api/idempotent_requests) | Living pages / 2026-10-10 | Provider documentation/practice | **Supported, provider-specific** | **Keep and qualify.** Local ledger retention must be an explicit design choice. |
| **Standard SQS can redeliver messages; consumers should be idempotent.** | AWS SQS Developer Guide, “Amazon SQS standard queues and at-least-once delivery”; `https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html` [8](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html) | Living documentation / 2026-10-10 | Provider documentation | **Supported for SQS standard** | **Keep** as concrete at-least-once example. |
| **SQS FIFO “exactly-once processing” is not an end-to-end business exactly-once guarantee.** | AWS FIFO deduplication docs describe five-minute send deduplication interval; `https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html` [16](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html) | Living documentation / 2026-10-10 | Provider documentation | **Supported as queue-level scope; broader claim contradicted** | **Qualify every “exactly once” statement by boundary.** |
| **Pub/Sub exactly-once has explicit boundaries and can still see publish-side duplicate copies.** | Google Cloud Pub/Sub “Exactly-once delivery,” regional considerations, push-subscription limitation and “Things to know”; `https://cloud.google.com/pubsub/docs/exactly-once-delivery` [4](https://cloud.google.com/pubsub/docs/exactly-once-delivery) | Last updated 2026-10-09 / 2026-10-10 | Provider documentation | **Supported** | **Keep.** Strong counterexample to end-to-end interpretation. |
| **Exactly-once storage writes may be unnecessary when duplicates can be detected elsewhere.** | BigQuery Storage Write API best practices, “Manage stream offsets to achieve exactly-once semantics”; `https://cloud.google.com/bigquery/docs/write-api-best-practices` [15](https://cloud.google.com/bigquery/docs/write-api-best-practices) | Last updated 2026-09-30 / 2026-10-10 | Provider documentation | **Supported** | **Keep.** Important counterweight to maximal control. |
| **Backpressure should bound outstanding work when publishing outruns capacity.** | Pub/Sub “Flow control,” “About flow control”; `https://cloud.google.com/pubsub/docs/flow-control-messages` [10](https://cloud.google.com/pubsub/docs/flow-control-messages) | Last updated 2026-10-09 / 2026-10-10 | Provider documentation | **Supported for Pub/Sub client behavior** | **Keep**, but business deadlines remain editorial. |
| **A configured dead-letter attempt count should not be treated as mathematically exact.** | Pub/Sub “Dead-letter topics,” “How maximum delivery attempts are calculated”; `https://cloud.google.com/pubsub/docs/dead-letter-topics` [12](https://cloud.google.com/pubsub/docs/dead-letter-topics) | Living documentation; inspected 2026-10-10 | Provider documentation | **Contradicts exact-count assumption** | **Rewrite** DLQ logic around ownership/recovery, not exact attempt count. |
| **Retries can increase pressure on a failing dependency; jitter/backoff reduce synchronization.** | AWS Builders' Library, *Timeouts, retries, and backoff with jitter*; `https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/` [11](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/) | Undated living article / 2026-10-10 | Provider-reported engineering practice | **Supported as AWS practice** | **Keep, qualify** against universal effect-size claims. |
| **Compensation can fail and can require manual recovery.** | Microsoft Azure Architecture Center, “Compensating Transaction pattern”; `https://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction` [13](https://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction?utm_source=chatgpt.com) | 2026-04-20 / 2026-10-10 | Provider architecture guidance | **Supported** | **Keep.** Do not present compensation as perfect rollback. |
| **“Dvě automatizace se spouštějí navzájem. Jak zastavit nekonečné kolečko?” / Two automations trigger each other.** | GitHub Actions `GITHUB_TOKEN` documentation documents provider-specific recursion prevention; `https://docs.github.com/en/actions/concepts/security/github_token` [14](https://docs.github.com/en/actions/concepts/security/github_token) | Living documentation / 2026-10-10 | Provider documentation | **Conditional** | **Rewrite:** provenance + semantic no-op + bounded loop; GitHub only as concrete example. |
| **“Systém omezuje počet požadavků. Co může počkat a co se nesmí ztratit?” / What can wait under a quota?** | RFC 6585 `429`, RFC 9110 `Retry-After`, Pub/Sub flow control. [9](https://www.rfc-editor.org/rfc/rfc6585.html) | 2012–2026 / 2026-10-10 | Standards + provider docs | **Supported premise; business answer conditional** | **Keep** distinction between retry timing, durable acceptance and business deadline. |
| **“Nejdřív přišlo zrušení, potom potvrzení. Který stav má automatizace použít?” / Cancellation arrived first; which state wins?** | Stripe “Event ordering” demonstrates delivery order is not guaranteed. `https://docs.stripe.com/webhooks` [2](https://docs.stripe.com/webhooks) | Living documentation / 2026-10-10 | Provider documentation | **Supported premise; source-version solution conditional** | **Keep after rewrite:** authoritative revision/current source state wins, not arrival time. |
| **Exactly-once is not a sufficiently precise universal description of all stream correctness properties.** | Trofimov et al., *Delivery, consistency, and determinism: rethinking guarantees in distributed stream processing*, arXiv `1907.06250`; `https://arxiv.org/abs/1907.06250` [17](https://arxiv.org/abs/1907.06250?utm_source=chatgpt.com) | 2019-07-14 / 2026-10-10 | Original research paper | **Supported within distributed stream-processing scope** | **Qualify**; do not extrapolate its performance results to business workflows. |
| **A green workflow-run count is an adequate end-to-end business KPI.** | Contradicted by end-to-end acknowledgement argument and bounded provider guarantees above. [1](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf) | Multiple | Synthesis / interpretation | **Contradicted** | **Remove as primary success criterion.** Retain only as technical diagnostic. |

### Evidence-type caution

The AWS Builders' Library sources are valuable because they are attributable descriptions by AWS engineers of how Amazon/AWS has approached retries and idempotent APIs. They are **reported engineering practice**, not randomized empirical evidence that the same designs produce a specified improvement in a small Czech or US team. [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/)

The provider documentation establishes what those provider features promise as documented on the inspection date. It does not establish customer ROI.

The 1984 Saltzer/Reed/Clark paper supplies a design argument rather than a modern cloud benchmark. Its relevance is conceptual: it explicitly analyzes duplicate suppression, message sequencing, delivery acknowledgments and why application-level verification may remain necessary even with lower-level reliability. The authors also caution that the end-to-end argument is a guideline, not an absolute rule, and that lower layers may usefully add reliability for performance. [1](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf)

## Contradictions, gaps, article angles and freshness triggers

### Where the recommendation reverses

**Do not build the full recovery architecture just because distributed-systems literature says failures happen.** At small enough volume, a native source report plus a human exception queue may be cheaper, clearer and safer. AWS's own idempotency article says rigorous contracts carry cost and complexity and are not right for every solution. [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/)

**Do not enable every “exactly-once” option by default.** BigQuery explicitly recommends considering its simpler default stream when upstream is already at-least-once or duplicates are easy to detect. Pub/Sub documents additional requirements and significantly higher publish-to-subscribe latency for exactly-once subscriptions. [15](https://cloud.google.com/bigquery/docs/write-api-best-practices)

**Do not interpret the end-to-end argument as “lower-layer reliability is useless.”** Saltzer, Reed and Clark explicitly allow lower-level reliability as a performance enhancement and say the correct placement depends on the application. [1](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf)

**Do not make an AI agent the cure for ambiguous state.** An AI classifier cannot manufacture an authoritative source revision, provider transaction record or idempotency contract. In this scope its leverage is triage: compressing context for the human recovery owner after deterministic controls have identified an exception. The economics above show a plausible synthetic case where AI increases, rather than decreases, lifecycle cost.

**Do not automatically retry an irreversible side effect after every timeout.** A timeout can mean “the action happened and the response was lost.” AWS's motivating example for idempotency is exactly this ambiguity. [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/)

**Do not use arrival time as business ordering where providers disclaim delivery order.** That advice reverses only if the source explicitly defines arrival order as authoritative—which the Stripe webhook example does not. [2](https://docs.stripe.com/webhooks)

**Do not assume compensation restores the prior world.** Some effects are irreversible, and Microsoft's guidance explicitly says compensations are business-specific, can fail and sometimes require manual intervention. [13](https://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction?utm_source=chatgpt.com)

### Unresolved gaps and inaccessible material

The original 1987 Garcia-Molina and Salem *Sagas* paper was identified at DOI `10.1145/38713.38742`. The ACM full-text endpoint returned an access error in this research environment, and a Cornell-hosted PDF mirror also failed to fetch. I therefore did **not** rely on the full paper for substantive claims about compensation. Compensation claims above instead use accessible current Microsoft primary documentation. The DOI is reported for provenance, not as evidence silently substituted from secondary summaries.

No live Stripe, AWS, Google Cloud, email-provider or payment-provider experiments were performed. The only executed test was the local deterministic Python fixture shown in full above.

No primary evidence located in this pass establishes a universal rate for duplicate webhook delivery, out-of-order events, recovery labor, AI triage accuracy, financial loss from duplicate effects or ROI from idempotency controls. Any article should avoid turning documented *possibility* into invented *frequency*.

The outbound messaging example is intentionally provider-neutral and synthetic. Actual production advice should be re-evaluated against the chosen provider's current idempotency, message-status and retention semantics.

Likewise, a `revision` field is a synthetic assumption in the worked order model. Some real sources offer sequence numbers, object versions, ETags or only current-object retrieval; some may provide no total ordering. The reducer should not be transplanted without verifying those source semantics.

There is no evidence in the technical primary sources inspected that the core retry/deduplication architecture should differ categorically between Czech and US teams. Regulatory, contract and retention requirements may change controls around the architecture, but such differences were intentionally not forced into this dossier.

### Candidate distinct article angles

| Angle | Core argument | Strongest evidence hook | Main qualification |
|---|---|---|---|
| **Your automation is green. The customer was still charged twice.** | Technical run success is weaker than business success. | End-to-end acknowledgment; timeout ambiguity; provider idempotency boundaries. [1](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf) | Use synthetic payment example, never imply an observed customer incident. |
| **Exactly once — exactly where?** | Provider exactly-once guarantees end at product boundaries. | Pub/Sub regional/pull scope, BigQuery write-offset scope, SQS FIFO window. [4](https://cloud.google.com/pubsub/docs/exactly-once-delivery) | Avoid saying exactly-once features are useless. |
| **The cancellation arrived before the order creation. Now what?** | Arrival order is not business chronology. | Stripe explicitly disclaims event ordering. [2](https://docs.stripe.com/webhooks) | Revision-based solution only works with trustworthy source versioning. |
| **Two automations keep triggering each other. Deduplication won't save you.** | A feedback loop can consist entirely of unique messages. | GitHub's explicit recursion protection as provider example. [14](https://docs.github.com/en/actions/concepts/security/github_token) | Provenance/hop-budget architecture is editorial, not a universal platform feature. |
| **429 is not just a retry error. It is a prioritization decision.** | Backoff controls traffic; the business must decide what may wait. | RFC 429/Retry-After + Pub/Sub flow control. [9](https://www.rfc-editor.org/rfc/rfc6585.html) | Deadlines and priorities are process-specific. |
| **An idempotency key is not a magic duplicate remover.** | Message identity, operation intent and key lifetime differ. | AWS client-token semantics; Stripe key lifetime. [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) | Provider contracts differ materially. |
| **The cheapest recovery system may be a human with a good reconciliation report.** | Control depth should follow consequence, volume and maintenance cost. | AWS complexity caveat; BigQuery simpler-default recommendation. [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) | Economics must use the team's own volumes and incident costs. |
| **AI belongs in the exception desk before it belongs in the payment path.** | AI can reduce interpretation work without becoming the authority for irreversible effects. | This dossier's synthetic lifecycle model; deterministic evidence base above. | No empirical AI ROI or accuracy result is claimed here. |

### Optional reusable assets

The most useful companion asset would be an **outcome-first recovery worksheet** with five columns:

```text
Business operation
Authoritative identity/version
What can duplicate?
How is external success verified?
What is the safe repair if success is unknown?
```

A second useful asset is an **event timeline debugger** showing, per root correlation ID:

```text
received_at
source_event_time
event_id
business_entity_id
source_revision
operation_intent
origin
causation_id
hop_count
processing_state
provider_reference
reconciliation_result
```

A third is a **recovery readiness matrix** that fault-injects the exact synthetic cases in this dossier: duplicate delivery, second event for the same business transition, cancellation-before-creation, acknowledgment-before-crash, timeout-after-side-effect, `429`/quota backlog, dependency outage, historical replay, concurrent revision race and two-automation feedback loop.

No separate downloadable file is referenced; the complete executed reducer fixture is included inline above.

### Freshness triggers

This dossier should be rechecked rather than treated as timeless when any of the following changes:

A source provider changes webhook ordering, retry duration, event identity or API-version semantics; Stripe changes idempotency-key retention behavior; Pub/Sub changes its exactly-once regional/push limitations or latency/quota characteristics; SQS changes FIFO deduplication scope; BigQuery changes Storage Write API offset or schema-evolution behavior; a downstream payment/message provider changes idempotency or lookup support; the order source introduces/removes a trustworthy revision field; the system moves to multi-region subscribers; retry volume begins hitting new quotas; a previously reversible business action becomes irreversible; or an AI model/tool version changes enough to invalidate its triage evaluation. Current Pub/Sub documentation in particular was updated as recently as 9 October 2026, illustrating why provider guarantees deserve explicit inspection dates. [4](https://cloud.google.com/pubsub/docs/exactly-once-delivery)

**Bottom-line decision:** measure recovery success as **“the intended business state was reached, external effects are known, and discrepancies are reconciled”**. Treat green workflow runs, successful message deliveries and provider-level exactly-once guarantees as valuable subordinate evidence—not as substitutes for that outcome. The smallest useful implementation is not an agentic recovery system; it is a durable inbox, explicit business identity/version, operation ledger, bounded retries, reconciliation, a human-owned quarantine path and outcome-level observability. Only after that deterministic foundation is demonstrably carrying enough exception-review cost should bounded AI triage be added.

## Portable citation index
1. [web.mit.edu](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf)
2. [docs.stripe.com](https://docs.stripe.com/webhooks)
7. [docs.stripe.com](https://docs.stripe.com/api/idempotent_requests)
3. [references-details-empty](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/)
11. [Timeouts, retries, and backoff with jitter | AWS Builder Center](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/)
4. [Exactly-once delivery  |  Pub/Sub  |  Google Cloud Documentation](https://cloud.google.com/pubsub/docs/exactly-once-delivery)
10. [Flow control  |  Pub/Sub  |  Google Cloud Documentation](https://cloud.google.com/pubsub/docs/flow-control-messages)
12. [Dead-letter topics  |  Pub/Sub  |  Google Cloud Documentation](https://cloud.google.com/pubsub/docs/dead-letter-topics)
15. [BigQuery Storage Write API (gRPC) best practices  |  Google Cloud Documentation](https://cloud.google.com/bigquery/docs/write-api-best-practices)
5. [spec/cloudevents/spec.md at main · cloudevents/spec · GitHub](https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md)
6. [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110.html)
9. [www.rfc-editor.org](https://www.rfc-editor.org/rfc/rfc6585.html)
8. [Amazon SQS at-least-once delivery - Amazon Simple Queue Service](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html)
16. [Exactly-once processing in Amazon SQS - Amazon Simple Queue Service](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html)
13. [Compensating Transaction Pattern - Azure Architecture ...](https://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction?utm_source=chatgpt.com)
14. [GITHUB_TOKEN - GitHub Docs](https://docs.github.com/en/actions/concepts/security/github_token)
17. [Delivery, consistency, and determinism: rethinking guarantees in distributed stream processing](https://arxiv.org/abs/1907.06250?utm_source=chatgpt.com)

## Import verification and asset availability
Genuine Deep Research completed in 9 minutes and was captured on 2026-10-10 after the first due time. Original export, exact independent prompt, canonical chat and citation URLs are retained. Complete inline examples/code were preserved; no missing sandbox download target is claimed as a delivered asset. Report code, test outcomes and execution assertions were not independently run or verified during this import. Delivery is not independent claim verification; Evidence status remains Captured.