Opakování požadavků, duplicity a idempotence

Retries, Duplicates, and Idempotency: Evidence Dossier

Research snapshot: 2026-10-10
Scope: Small-team, event/webhook-driven order workflow feeding a cloud analytical warehouse and downstream business automation. Public sources only; examples and fixtures below are synthetic.
Decision supported: Explain recovery design through concrete business outcomes rather than green workflow-run status.

Executive answer and uncertainties

The evidence strongly supports the decision.

A successful HTTP delivery, queue acknowledgment, workflow execution, or even a provider-specific “exactly once” guarantee is not equivalent to the business action having happened correctly. This distinction is old and fundamental: Saltzer, Reed, and Clark’s end-to-end argument notes that an acknowledgment that a message reached a host does not establish that the application actually performed the requested action. 1 Modern provider documentation makes the same boundary visible. Stripe labels a webhook delivery successful when the endpoint returns a successful HTTP status; separately, Stripe recommends asynchronous processing and warns that events can be duplicated and arrive out of order. 2

For the bounded order process in this dossier, “done” should mean something like this:

The authoritative order revision has been durably accepted; the canonical business state and warehouse agree with that revision; every required downstream business intent has a known outcome or an explicitly owned unresolved state; prohibited older actions did not occur; and the whole chain can be reconciled from an audit trail.

A green run is useful diagnostic evidence. It is not that outcome.

The key distinctions are:

Concept	What it actually answers	What it does not establish
Retry safety	“Can I make this attempt again without an unacceptable additional effect?” HTTP defines idempotency in terms of the intended server effect of repeated identical requests. 3	That another system has not already independently performed the same business intent.
Idempotency key	“Are these attempts the same logical request?” Stripe, for example, allows callers to associate POST retries with a client-generated key. 4	Permanent deduplication. Provider key retention is finite and provider-specific.
Transport deduplication	“Have I seen this delivery/message identity before?” SQS FIFO, Cloud Tasks, Shopify and Pub/Sub expose different mechanisms and windows. 5	That two different messages do not represent the same business action.
Business idempotency	“Has this order/revision/intent already produced the required business effect?”	It cannot normally be inferred from a transport ID alone. Saltzer et al. specifically observed that application-generated duplicate requests can look like distinct network messages. 1
Exactly-once claim	A guarantee over a specifically documented boundary. Pub/Sub, Kafka, SQS and BigQuery attach materially different scopes and conditions to such guarantees. 6	End-to-end “customer was charged/notified exactly once” correctness across independent systems.
Compensation	A subsequent business operation that offsets an already-completed effect when atomic rollback is impossible.	Erasure of every consequence: an email already read, shipment already dispatched, or financial transaction already settled may not be fully reversible.

Most important practical conclusion: use provider-level duplicate suppression and exactly-once features when they simplify the design, but keep the business invariant above them. Pub/Sub explicitly limits exactly-once delivery to pull subscriptions and a regional scope, while acknowledging that publish-side duplicates can still result in multiple messages; Kafka states that exactly-once processing for non-Kafka destinations generally needs cooperation from the destination system. 6 BigQuery similarly distinguishes a default at-least-once write stream from client-offset-based exactly-once streams. 7

There are important uncertainties. Provider documentation establishes specified behavior, not measured real-world reduction in incidents, maintenance effort, or duplicate financial effects. I found no strong primary comparative study quantifying the business ROI of manual versus deterministic versus AI-assisted recovery for a small team. The recommendation that AI stay outside irreversible authorization in this workflow is therefore editorial risk containment, not an empirically demonstrated effect size. Likewise, the dossier does not perform a Czech-versus-US compliance comparison. The technical failure semantics reviewed are primarily provider/service/configuration dependent; regional configuration can matter—for example Pub/Sub’s exactly-once scope is regional—but I found no reason in the reviewed technical evidence to make Czech versus US company location the main architecture distinction. 6

Bounded process and viable alternatives

The process must be defined before the automation.

Synthetic bounded process: order-state propagation and recovery.

An e-commerce or order-management source emits creation, update and cancellation events. The workflow must preserve the authoritative state in an analytical warehouse and, for selected meaningful transitions, create downstream business actions such as stopping fulfillment, issuing a refund, or sending a customer notification. The synthetic system deliberately permits retries, replay, duplicates and out-of-order delivery because those are documented characteristics of common webhook and messaging systems. Stripe states that webhook endpoints can receive duplicate events and that event order is not guaranteed; Shopify similarly documents duplicate deliveries and recommends idempotent processing or explicit webhook-ID tracking. 2

Trigger: receipt of an authenticated webhook or event.

Minimum input model:

delivery_id        identity of this transport delivery
event_id           identity of the source event/action, if supplied
order_id           identity of the business entity
source_revision    authoritative monotonic revision / sequence, if available
event_type         created | updated | confirmed | cancelled | ...
operation_intent   semantic action derived locally, e.g. refund:cancellation:r12
source_timestamp   informational timestamp
received_at        local receipt timestamp
origin             source / automation-A / automation-B / operator
payload            source fields required for state/reconciliation

These identities should not be collapsed. Shopify provides a particularly concrete example: it documents a webhook delivery identifier separately from an event identifier, with the latter usable to correlate deliveries arising from the same merchant action. 8 Stripe goes one step further for some duplicate cases: separate Stripe Event objects can represent the same object/action, so its documentation advises using the object ID together with the event type when identifying those duplicates. 2

Business owner: Order Operations owns the meaning of CONFIRMED, CANCELLED, deadlines and acceptable compensation.
Technical owner: the integration/data team owns durable acceptance, queueing, state reduction, provider contracts and observability.
Recovery owner: a named operator owns reconciliation failures and uncertain external side effects. One person can wear several hats in a small team; the responsibilities should still be explicit.

A useful synthetic business-state model is:

CREATED ──> CONFIRMED ──> CANCELLED
   │             │
   └─────────────┴──────> CANCELLED

Newer authoritative revision wins.
An older event may be retained in the event ledger but must not regress canonical state.
Same revision + contradictory state => quarantine, not arbitrary winner.

The processing state is different:

RECEIVED
   ↓
DURABLY_ACCEPTED
   ↓
STATE_REDUCED
   ├──> WAREHOUSE_PENDING ──> WAREHOUSE_OBSERVED
   └──> BUSINESS_INTENT_PENDING ──> EFFECT_OBSERVED
                                  └──> OUTCOME_UNCERTAIN
                                             ↓
                                        RECONCILED
                                             ↓
                                          COMPLETE

The separation matters. A CANCELLED order can have a processing state of OUTCOME_UNCERTAIN; pretending those are the same dimension is how green dashboards hide business failures.

Observable completion for synthetic order O-42, revision 12 = CANCELLED, could therefore be defined as:

source_current_revision(O-42) == 12
canonical_state(O-42) == CANCELLED @ revision 12
warehouse_state(O-42) == CANCELLED @ revision 12
required_refund_intent(O-42, rev12) has known final provider status
required_cancellation_notice(O-42, rev12) has known accepted/sent status
no older CONFIRMED event subsequently changed canonical state
no second refund exists for the same cancellation intent
reconciliation_status == matched

Notice the wording around a message: unless the delivery channel exposes stronger evidence, “provider accepted/sent it” is observable while “the human actually read it” may not be.

Manual/native fallback is deliberately boring: an operator opens the authoritative order, verifies its current revision/state, checks the payment or messaging provider, compares the warehouse record, and performs or approves the missing deterministic repair. The audit record captures what the operator saw and why the repair was safe.

The same process can then be implemented at three autonomy levels:

Variant	Same business outcome	Normal path	Recovery	Upkeep profile	Best fit
Manual/native baseline	Source, warehouse and required downstream action reconcile.	Native source UI/export/connectors; operator checks exceptions and executes actions.	Human compares authoritative source with downstream systems.	Lowest custom-code surface; recurring human work and slower response.	Low volume, rare exceptions, high consequence, or immature state semantics.
Deterministic automation — recommended default	Same.	Durable inbox → revision-aware reducer → queue/outbox → warehouse/effect → reconciler.	Replay safe operations; query uncertain effects; compensate or escalate where needed.	More engineering: schemas, API changes, queue configuration, reconciliation and runbooks.	Stable process with enough repetition to justify maintenance.
Bounded AI triage assistant	Still the same.	Deterministic pipeline remains authoritative. AI summarizes DLQ items, groups likely causes, proposes a repair and surfaces relevant audit records.	Human approves a deterministic recovery command.	Adds model/prompt/evaluation drift and review burden.	Exception diagnosis is cognitively expensive but authorization risk should remain bounded.

The AI variant does not authorize a refund/payment, send a customer message, or make a production write in this design. That restriction is a design recommendation rather than a provider-documented necessity or measured empirical result.

The simplest baseline should win when it already meets the outcome. More autonomy is not inherently superior.

Evidence synthesis around recovery outcomes

Acknowledge durable work, not wishful work. Stripe recommends returning a successful response quickly and handling complex webhook processing asynchronously. Shopify likewise recommends queueing webhook work, especially for bursts, and separately recommends reconciliation for missed data. 2 The critical design inference is to combine “fast ACK” with durable acceptance before ACK: verify enough of the request to accept it, persist an immutable receipt or durable work item, then return success and perform expensive work asynchronously. Otherwise a crash after 200 OK but before persistence can produce an acknowledged-but-lost business event. Stripe’s delivery UI explicitly treats a successful HTTP response as successful endpoint delivery, not proof that later accounting or business work happened. 2 This is precisely the distinction made by the end-to-end argument: application correctness requires evidence that the requested application action occurred, not just lower-level delivery. 1

Retries solve transient failures but introduce ambiguity and load. HTTP itself permits clients to retry idempotent methods after communication failure because repeating their intended effect is defined as equivalent; it warns clients not to automatically retry non-idempotent methods unless the client knows the operation is actually safe or knows the original request was not applied. 3 AWS’s retry guidance adds an operational warning: retries can amplify overload, and the caller can time out even when the remote side effect actually happened. It recommends bounded retries, exponential backoff and jitter, and avoiding retries independently at every stack layer. Its illustrative calculation shows that five layers each making up to three attempts can drive 3⁵ = 243 attempts against the deepest dependency—not an observed universal effect size, but a straightforward worst-case fan-out model. 9

The planning formula is visible:

\text{maximum downstream attempts} = r^d

where r is attempts made independently per layer and d is the number of retrying layers. For r = 3, d = 5, the result is 243. The implication is not “never retry”; it is to decide where retry ownership lives and make retry budgets visible.

Idempotency-key lifetime is not business-memory lifetime. Stripe API v1 documentation says idempotency keys can be removed after they are at least 24 hours old; reusing a pruned key can therefore result in a fresh request. 4 SQS FIFO deduplication recognizes a send retry within a five-minute deduplication interval. 5 Cloud Tasks remembers a task name for up to 24 hours after task deletion. 10 Those are three materially different windows. Therefore a payment, cancellation or notification that must remain unique across week-old replay needs an application/business identity whose lifetime is derived from the process—not merely the provider’s short-term dedup token.

A robust operation identity might be:

operation_id =
    SHA256(order_id + ":" + semantic_transition + ":" + source_revision)

Example:
O-42 : refund_due_to_cancellation : 12

The transport delivery can change; the business intent remains the same.

“Exactly once” is useful only after asking “exactly once where?” Google Pub/Sub’s exactly-once feature prevents redelivery after successful acknowledgment under documented conditions, but it is limited to pull subscriptions and regional operation. Its documentation also warns that multiple unique publish retries can still yield multiple messages even with exactly-once delivery enabled. 6 Apache Kafka 4.3 documents exactly-once read/process/write semantics for Kafka topics using transactions and read-committed consumers, but explicitly says exactly-once delivery into other destination systems generally requires cooperation from those systems. 11 BigQuery’s Storage Write API similarly provides exactly-once stream appends when the client uses committed streams with offsets, while the default stream remains at-least-once. 7 SQS FIFO’s “exactly-once processing” documentation specifically means that repeated sends within its deduplication window do not introduce duplicate queue messages. 5

These claims are not contradictory once their boundaries are preserved. They become misleading only when translated into “therefore the customer cannot be charged twice.”

Out-of-order arrival should not define business truth. Pub/Sub ordered delivery is conditional on the same ordering key and publishing region and does not establish a global order across keys; redelivery behavior can also cause later messages on an ordering key to be redelivered. 12 BigQuery’s CDC documentation directly recognizes the problem: ingest time is normally used to choose precedence between records with the same primary key, but where update order is not guaranteed Google provides _CHANGE_SEQUENCE_NUMBER so the producer can supply the intended logical order. 13 BigQuery also maintains a two-day retention window specifically to support CDC DELETE operations arriving out of order. 13

For the article-target question “Cancellation arrived before confirmation. Which state should automation use?”, the answer is:

Use the state attached to the newest authoritative source revision, not the event that happened to arrive last.

If revision 12 = CANCELLED arrives before revision 11 = CONFIRMED, revision 12 remains current when revision 11 eventually appears. If the source does not expose a trustworthy monotonic revision, the safer recovery is to read current authoritative source state before an irreversible effect. If neither a version nor an authoritative current-state query exists, contradictory events should be quarantined; arrival timestamps alone do not manufacture certainty.

Rate limiting is a business-priority problem as well as a queue problem. Cloud Tasks lets an operator bound dispatch rate and concurrency, applies rate limits to first attempts and retries, and supports configurable retry duration/backoff. 14 BigQuery recommends buffering important records, retrying transient insertion failures, handling 429 quota exhaustion, and persisting records elsewhere if every row is crucial and an outage could outlast the available buffer. Conversely, its documentation explicitly allows abandoning data when missing it is acceptable. 7

That gives a cleaner answer to “What can wait and what must not be lost?”:

Can wait      = work whose business deadline has not expired.
Can be lost   = work the business explicitly declares reconstructable or disposable.
Must persist  = work whose disappearance would violate the business invariant.
Must lead     = work with the shortest consequential business deadline.

For example, a warehouse enrichment calculated again from the source may safely wait or be reconstructed. A cancellation intended to stop imminent fulfillment may have a much harder deadline. The queue does not know that distinction until the application encodes it.

A useful queue design is therefore based on (loss tolerance, deadline), not simply “failed/succeeded”:

durable accepted work
        │
        ├── deadline-critical queue
        │        └── bounded retry → reconcile/escalate before deadline
        │
        ├── normal durable queue
        │        └── backoff/jitter → DLQ → human repair
        │
        └── reconstructable/best-effort
                 └── delay, batch, or discard under explicit policy

Feedback loops should be stopped at the meaningful-transition boundary, not merely by counting invocations. AWS Lambda documents a native recursion detector that stops the next invocation after approximately 16 invocations in a chain for supported service combinations, but it also says detection covers only particular AWS services/SDK paths; loops involving unsupported services can escape it. 15 AWS therefore provides a useful emergency fuse, not a universal semantic solution.

For “Two automations trigger each other. How do we stop the infinite loop?”, the deterministic design is:

1. Trigger only on a meaningful business transition, not every write.
2. Suppress no-op writes: new_value == current_value => do nothing.
3. Carry origin/causation metadata.
4. Suppress an echo generated by the same operation_id.
5. Keep a hop/time budget as an emergency circuit breaker.
6. Alert and quarantine when the circuit breaker fires.

If automation A writes sync_status=CANCELLED and automation B writes that same status back, neither write should qualify as a new business transition. A recursion counter is defense in depth. It should not be the primary definition of correctness.

Worked failure dossier and portable test asset

The following timeline is synthetic, not a reported customer case. It combines failure modes documented separately by webhook, queue and warehouse providers. Stripe documents retries, duplicates and unordered webhook delivery; Cloud Tasks documents configurable retry/rate behavior; BigQuery documents offset-based deduplication and source-order controls. 2

Time	Input / failure	Unsafe implementation	Safe interpretation and repair criterion
10:00:00	O-42 r10 CONFIRMED, delivery d1, event evt10	Apply and immediately trigger every downstream automation.	Durably accept, apply revision 10, create only semantic intents implied by transition.
10:00:01	Exact redelivery of d1	Execute again.	Same delivery_id: record duplicate and no-op.
10:00:03	O-42 r12 CANCELLED, d2/evt12 arrives before r11	Arrival-order reducer sets cancelled—for now.	Apply because 12 > 10; create cancellation/refund intents identified by (O-42, intent, r12).
10:00:04	Refund API accepts request, client times out before response	“Workflow failed” → issue a new refund request with a new key.	Outcome is unknown, not failed. Query provider or retry the same valid idempotency identity. A communication timeout does not prove the first side effect failed. HTTP and AWS guidance both distinguish this ambiguity. 3
10:00:07	Downstream says 429	Tight loop retry.	Leave intent durable; retry according to bounded backoff/deadline policy. Retries themselves must obey rate limits. 14
10:00:09	O-42 r11 CONFIRMED, d3/evt11 arrives late	“Latest received wins” → resurrect CONFIRMED.	Store for audit, but do not apply because 11 < 12. BigQuery exposes a source-supplied sequence mechanism for this class of problem. 13
10:02:00	Replay sends evt12 again under new delivery d4	Delivery ID differs, therefore treat as new cancellation.	Event identity reveals replay; even if event ID also changed, (order, cancellation/refund intent, rev12) should already exist.
10:03:00	Automation A writes sync_status=CANCELLED; automation B mirrors it back	Each write triggers the other indefinitely.	Same semantic value + same causal operation => no transition, no new output. Emergency hop budget catches defects. AWS’s provider-native detector demonstrates the value and limits of a separate recursion fuse. 15
10:15:00	Warehouse becomes available after backlog	Re-run everything from scratch including side effects.	Replay immutable events through state reduction; warehouse writes can be retried independently of already-observed payment/message effects.
10:30:00	Reconciler runs	Sees all workflow runs green and closes incident.	Compare source r12, canonical r12, warehouse r12 and provider-side refund/message records. Only close on invariant match. Saltzer et al.’s end-to-end argument is the foundational rationale. 1

The recovery rule is consequently state-dependent, not “retry on red”:

Observed state	Safe action
No side effect attempted	Retry when dependency policy permits.
Side effect returned a definitive non-application result	Retry same logical operation.
Side effect may have succeeded but reply was lost	Reconcile/query first, or retry with the same provider idempotency identity while its documented guarantee still applies.
Idempotency key window expired and provider outcome is unknown	Do not blindly create a new operation; query by business/reference identity or escalate.
Duplicate delivery/event	Store audit evidence; no-op.
Different events represent same business intent	Deduplicate by business operation identity.
Older source revision	Store but do not apply to canonical state.
Same source revision contains contradictory states	Quarantine for authoritative reconciliation.
Wrong reversible effect definitely happened	Run an explicitly defined compensation.
Wrong irreversible effect happened	Do not pretend rollback exists; contain consequences, record them, and require human recovery.
Work has exhausted retry policy	DLQ/quarantine must preserve enough immutable input and context for deterministic replay.

Portable executed fixture

The following Python fixture is complete and self-contained. It exercises duplicate delivery, replay under a different delivery ID, and an out-of-order older revision. It was executed during this research session. It is a synthetic model, not a provider integration test.

from dataclasses import dataclass
from typing import Dict, List, Tuple


@dataclass(frozen=True)
class Event:
    delivery_id: str
    event_id: str
    order_id: str
    revision: int
    state: str


@dataclass
class Canonical:
    revision: int
    state: str


class Reducer:
    def __init__(self) -> None:
        self.seen_deliveries = set()
        self.seen_events = set()
        self.orders: Dict[str, Canonical] = {}
        self.intents = set()
        self.audit: List[Tuple] = []

    def process(self, event: Event) -> None:
        # Transport identity.
        if event.delivery_id in self.seen_deliveries:
            self.audit.append(
                (event.delivery_id, "duplicate_delivery_skipped")
            )
            return
        self.seen_deliveries.add(event.delivery_id)

        # Source event identity is different from delivery identity.
        duplicate_event = event.event_id in self.seen_events
        self.seen_events.add(event.event_id)

        current = self.orders.get(event.order_id)

        # Business/source ordering beats arrival ordering.
        if current and event.revision < current.revision:
            self.audit.append(
                (
                    event.delivery_id,
                    "stale_revision_stored_not_applied",
                    event.revision,
                    current.revision,
                )
            )
            return

        # Identical source revision should not repeatedly create effects.
        if current and event.revision == current.revision:
            if event.state == current.state:
                self.audit.append(
                    (
                        event.delivery_id,
                        "same_revision_same_state_noop",
                        duplicate_event,
                    )
                )
                return

            # Contradictory truth at the same revision is not safely resolvable
            # from arrival order.
            self.audit.append(
                (
                    event.delivery_id,
                    "conflict_quarantine",
                    event.revision,
                    current.state,
                    event.state,
                )
            )
            return

        self.orders[event.order_id] = Canonical(
            event.revision, event.state
        )
        self.audit.append(
            (
                event.delivery_id,
                "state_applied",
                event.revision,
                event.state,
            )
        )

        # Semantic/business identity: independent of delivery ID.
        if event.state == "CANCELLED":
            operation_id = (
                event.order_id,
                "notify_cancellation",
                event.revision,
            )

            if operation_id not in self.intents:
                self.intents.add(operation_id)
                self.audit.append(
                    (
                        event.delivery_id,
                        "intent_created",
                        operation_id,
                    )
                )


events = [
    Event("d1", "evt10", "O-42", 10, "CONFIRMED"),
    Event("d1", "evt10", "O-42", 10, "CONFIRMED"),   # delivery duplicate
    Event("d2", "evt12", "O-42", 12, "CANCELLED"),  # newer event arrives
    Event("d3", "evt11", "O-42", 11, "CONFIRMED"),  # late older event
    Event("d4", "evt12", "O-42", 12, "CANCELLED"),  # replay, new delivery ID
]

reducer = Reducer()

for event in events:
    reducer.process(event)

print(reducer.orders)
print(reducer.intents)

for row in reducer.audit:
    print(row)

Executed result:

{'O-42': Canonical(revision=12, state='CANCELLED')}

{('O-42', 'notify_cancellation', 12)}

('d1', 'state_applied', 10, 'CONFIRMED')
('d1', 'duplicate_delivery_skipped')
('d2', 'state_applied', 12, 'CANCELLED')
('d2', 'intent_created', ('O-42', 'notify_cancellation', 12))
('d3', 'stale_revision_stored_not_applied', 11, 12)
('d4', 'same_revision_same_state_noop', True)

What this fixture does not test: an actual webhook signature, real queue durability, real Stripe idempotency behavior, BigQuery CDC, provider outages, concurrent workers, database transaction isolation, real rate limits, real message delivery, or an AI assistant. Those tests remain unperformed.

A smallest workable production evaluation should therefore inject at least these cases before launch:

duplicate same delivery ID
duplicate same source event with new delivery ID
two different events representing same business action
newer cancellation followed by older confirmation
same revision with contradictory payload
crash immediately before durable acceptance
crash immediately after durable acceptance but before ACK
crash after side effect but before recording response
429 / 503 dependency responses
queue backlog beyond normal latency
replay older than provider idempotency-key lifetime
two concurrent workers for same order
automation-A → automation-B → automation-A loop
manual override while automated work is queued
DLQ replay after source has advanced again

The acceptance assertion for every case should be a business invariant, not merely “test runner exited zero.”

Lifecycle, observability, and smallest workable implementation

The minimum deterministic architecture is smaller than a full orchestration platform:

Source webhook
     │
     ▼
Authenticate / minimally validate
     │
     ▼
┌──────────────────────────────┐
│ Durable inbox / event ledger │
│ unique delivery/event fields │
└──────────────────────────────┘
     │   durable acceptance
     ├───────────────> HTTP 2xx
     │
     ▼
Revision-aware state reducer
     │
     ├──> canonical order state
     │
     ├──> warehouse work
     │
     └──> business-operation outbox
                    │
                    ▼
              rate-limited worker
                    │
          ┌─────────┴──────────┐
          ▼                    ▼
  external side effect    outcome uncertain
          │                    │
          └─────────┬──────────┘
                    ▼
                reconciler
                    │
             matched / repair

The important transaction boundary is not “everything across every SaaS API is atomic”—usually it is not. It is that the system should not acknowledge accepted work while having no durable record from which to recover. The external side effect then lives behind its own persistent operation identity and reconciliation state. This design is consistent with provider recommendations to ACK webhooks quickly and process them asynchronously, while adding the end-to-end correctness check that transport documentation cannot supply. 2

For warehouse ingestion, BigQuery documents both sides of the trade-off. When each record matters, Google recommends buffering before insertion, persisting data when necessary through prolonged unavailability, putting poison records aside for investigation, and using explicit offsets when exactly-once append semantics are required. If exactly-once behavior is unnecessary, the default stream has lower complexity and higher throughput characteristics. 7 That is a useful general lesson: stronger guarantees have cost and complexity and should be bought where the outcome needs them.

The dashboard should therefore lead with:

accepted but not reconciled orders
oldest unreconciled business deadline
orders whose source revision != canonical revision
orders whose canonical revision != warehouse revision
business intents pending / uncertain / compensated
duplicate delivery count
duplicate business-intent suppression count
stale-event count
same-revision conflict count
rate-limit / dependency-error count
DLQ age and count
feedback-loop fuse activations
manual recoveries and reason codes

Provider/workflow run success belongs below those metrics. Pub/Sub itself documents higher latency for exactly-once subscriptions and throughput implications when exactly-once is combined with ordered delivery, illustrating why infrastructure metrics and business correctness metrics need separate treatment. 6

An audit record should contain enough information to answer an incident without reconstructing it from scattered logs:

{
  "delivery_id": "d4",
  "event_id": "evt12",
  "order_id": "O-42",
  "source_revision": 12,
  "source_state": "CANCELLED",
  "received_at": "2026-10-10T10:02:00Z",
  "operation_id": "O-42:refund_due_to_cancellation:12",
  "provider_idempotency_key": "O-42:refund_due_to_cancellation:12",
  "idempotency_key_created_at": "...",
  "idempotency_key_known_valid_until": "... or provider-specific/unknown",
  "attempt": 2,
  "provider_object_id": "...",
  "provider_status": "succeeded | declined | unknown",
  "canonical_decision": "stale | apply | no-op | quarantine",
  "reconciliation_status": "matched | mismatch | pending",
  "human_actor": null,
  "human_reason": null
}

This also exposes an often-hidden maintenance burden: provider guarantees themselves are configuration and version dependencies. Stripe’s idempotency semantics differ between API v1 and v2 on its current documentation page; Pub/Sub’s guarantee depends on subscription type and regional conditions; Cloud Tasks has configurable retry limits; Kafka exactly-once behavior depends on transactions and consumer isolation; BigQuery distinguishes stream modes. 4

A lifecycle-cost model should therefore include more than build cost:

C_{\text{annual}}
=
\frac{C_{\text{build}}}{A}
+
C_{\text{platform}}
+
C_{\text{monitoring}}
+
C_{\text{schema/API change}}
+
C_{\text{reconciliation}}
+
C_{\text{incidents}}
+
C_{\text{human exceptions}}
+
C_{\text{retirement}}

where A is the chosen amortization period in years. This is a planning model, not an empirical cost estimate; no numerical market benchmark was found in the reviewed primary evidence.

The manual baseline remains preferable when:

• volume and exception cost are below the lifecycle burden of automation;
• the source/native integration already supplies adequate reconciliation;
• the business cannot define a stable identity or authoritative ordering rule;
• the external side effect is high-consequence but offers neither idempotency nor a reliable status lookup;
• changes to the underlying process are frequent enough that the automation would continuously chase them.

Conversely, deterministic automation becomes attractive when the transition rules, identities and source authority are stable and there is enough repeated work to amortize the reconciler and maintenance.

There are also cases where the recommendation to maximize reliability reverses. Saltzer et al. explicitly use real-time voice as an example in which retransmission delay can be worse than a small loss, showing that reliability controls depend on the application’s actual requirements. 1 Low-consequence telemetry may likewise rationally use at-most-once or best-effort behavior. Fully Kafka-contained processing can rely more heavily on Kafka transactions because the guarantee boundary encompasses both consumed offsets and produced Kafka records; that argument weakens as soon as the outcome moves into an independent destination. 11 Intentionally recursive Lambda designs are also possible—AWS permits recursion protection to be disabled—but AWS itself recommends caution and guardrails. 15

The smallest workable implementation I would choose for this order workflow is therefore:

one durable inbox, one canonical revision-aware order table, one durable table of semantic business intents, bounded queue workers, and one periodic reconciler. Add provider-specific exactly-once features where cheap. Add a DLQ only together with a named repair path. Add AI only after deterministic repair primitives exist.

That is materially more valuable than adding another orchestration layer whose principal achievement is turning the run green.

Claim-to-primary-source matrix

The matrix separates provider behavior, foundational argument, attributable engineering practice, and editorial interpretation. “Supported” means the cited primary source directly supports the bounded claim; it does not imply an independently measured business effect.

Claim target / exact scope	Primary source and exact section	Source date	Inspected	Evidence type	Assessment	Editorial action
Repeating an HTTP request is automatically safe only when its semantics are idempotent, or the client otherwise knows repetition is safe.	RFC 9110 §9.2.2, Idempotent Methods. 3 URL: https://www.rfc-editor.org/rfc/rfc9110.html#section-9.2.2	2022-06	2026-10-10	Internet standard	Supported	Keep. Use as definition of retry safety, not business dedup.
A lower-level successful delivery does not prove the requested application action completed; duplicate suppression and ordering can require application knowledge.	Saltzer, Reed & Clark, End-to-End Arguments in System Design, “Delivery guarantees,” “Duplicate message suppression,” “Guaranteeing FIFO message delivery.” 1 DOI: https://doi.org/10.1145/357401.357402; manuscript: https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf	1984-11	2026-10-10	Peer-reviewed foundational systems paper	Supported	Keep. This is the strongest conceptual basis for “business outcome, not green run.”
Stripe POST retries can be tied to an idempotency key; API v1 retains results once endpoint execution begins, and keys can be pruned after at least 24h.	Stripe API, Idempotent requests. 4 URL: https://docs.stripe.com/api/idempotent_requests	Undated live documentation	2026-10-10	Provider documentation	Supported within Stripe/API-version scope	Qualify. Never imply infinite lifetime or universal provider semantics.
A Stripe webhook may be delivered repeatedly and out of order; successful HTTP delivery is distinct from downstream business processing.	Stripe, Webhooks: event delivery behavior, automatic retries, event ordering, duplicate events, asynchronous handling. 2 URL: https://docs.stripe.com/webhooks	Undated live documentation	2026-10-10	Provider documentation	Supported	Keep. Excellent concrete example.
Delivery identity and source-event identity are not necessarily the same thing.	Shopify, Verify webhook deliveries: duplicate deliveries and Webhook-Id versus Event-Id. 8 URL: https://shopify.dev/docs/apps/build/webhooks/verify-deliveries	Undated live documentation	2026-10-10	Provider documentation	Supported within Shopify scope	Keep, then generalize only as an architecture distinction.
SQS FIFO prevents a retry from introducing a duplicate message only within its defined deduplication mechanism/window; this is not an end-to-end business guarantee.	AWS SQS Developer Guide, Exactly-once processing. 5 URL: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html	Undated live documentation	2026-10-10	Provider documentation	Supported/conditional	Rewrite any naked “exactly once” claim to include the queue boundary and five-minute dedup interval.
Pub/Sub exactly-once is conditional: pull subscriptions, regional scope; publish-side duplicates can still produce multiple messages.	Google Cloud Pub/Sub, Exactly-once delivery, including “Exactly-once delivery and push subscriptions” and “Things to know.” 6 URL: https://cloud.google.com/pubsub/docs/exactly-once-delivery	Last updated 2026-10-09 UTC	2026-10-10	Provider documentation	Supported	Keep. Strong counterexample to treating the phrase as universal.
Provider ordering does not remove the need for business ordering; Pub/Sub ordering is per ordering key and has redelivery/latency implications.	Google Cloud Pub/Sub, Order messages. 12 URL: https://cloud.google.com/pubsub/docs/ordering	Last updated 2026-10-09 UTC	2026-10-10	Provider documentation	Supported/conditional	Keep, qualified to Pub/Sub.
Rate limits and retries share capacity; retries should be explicitly bounded/backed off rather than hidden inside an unbounded workflow.	Google Cloud Tasks, Configure queue routing, limits, and retries, rate limits and retry parameters. 14 URL: https://cloud.google.com/tasks/docs/configuring-queues	Last updated 2026-10-07 UTC	2026-10-10	Provider documentation	Supported	Keep. Business prioritization on top is editorial design.
Cloud Tasks transport deduplication has a finite identity lifetime: task names are remembered for up to 24h after deletion.	Google Cloud Tasks, Understand Cloud Tasks → Task deduplication. 10 URL: https://cloud.google.com/tasks/docs/dual-overview	Live documentation; page date not captured	2026-10-10	Provider documentation	Supported	Keep as TTL counterexample.
BigQuery default-stream writes are at-least-once; committed streams plus caller-supplied offsets can provide exactly-once append behavior. Crucial data should be buffered/persisted through outages.	BigQuery, Introduction to the Storage Write API, stream types and “Handle unavailability” / reliability guidance. 7 URL: https://cloud.google.com/bigquery/docs/write-api	Last updated 2026-10-07 UTC	2026-10-10	Provider documentation	Supported	Keep. Do not translate stream-level dedup into business-level exact-once.
When record arrival order is unreliable, BigQuery CDC can use a caller-supplied _CHANGE_SEQUENCE_NUMBER; ingest-time order is insufficient in such cases.	BigQuery, Change data capture → Manage custom ordering. 13 URL: https://cloud.google.com/bigquery/docs/change-data-capture	Last updated 2026-10-07 UTC	2026-10-10	Provider documentation	Supported	Keep. Strong concrete support for “source version beats arrival order.”
Out-of-order deletes are a real enough ingestion case that BigQuery CDC maintains a two-day delete-retention window.	BigQuery, Change data capture → Limitations/DELETE behavior. 13 Same URL as above.	Last updated 2026-10-07 UTC	2026-10-10	Provider documentation	Supported within BigQuery CDC	Keep as concrete example, not universal timing advice.
Native recursion protection can limit damage but is provider/path-specific and therefore cannot replace semantic loop prevention.	AWS Lambda, Use Lambda recursive loop detection to prevent infinite loops, “Understanding,” “Supported AWS services,” and “Responding.” 15 URL: https://docs.aws.amazon.com/lambda/latest/dg/invocation-recursion.html	Undated live documentation	2026-10-10	Provider documentation	Supported/conditional	Keep with limitation. “Approximately 16” is an emergency fuse, not a design target.
Retries can amplify overload; backoff, jitter and centralized retry ownership reduce retry storms.	Marc Brooker/AWS, Timeouts, retries, and backoff with jitter. 9 URL: https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter	Published 2026-06-12; modified 2026-06-15	2026-10-10	Attributable primary company engineering guidance, not controlled empirical study	Supported as reported practice/design guidance	Keep but label the 243× example as illustrative, not measured general effect.
Kafka exactly-once is strong inside its transactional boundary; writes to external systems need destination cooperation or coordinated state.	Apache Kafka 4.3, Design → Message Delivery Semantics / Using Transactions. 11 URL: https://kafka.apache.org/43/design/design/	Docs last modified 2026-05-22	2026-10-10	Official open-source project documentation	Supported	Keep. Excellent “exactly once where?” evidence.
“AI triage should not authorize payment/message/production writes” improves safety in this process.	No primary comparative evidence located for this exact bounded workflow.	—	2026-10-10	Editorial risk-control design	Unresolved empirically	Keep only as explicit recommendation/hypothesis, not evidence-backed outcome claim.
A custom deterministic system will have lower total lifecycle cost than manual/native operation at a particular volume.	No appropriate primary quantitative evidence located.	—	2026-10-10	Evidence gap	Unresolved	Do not claim. Use a local cost model and measured exception data.

An original source on sagas/compensating transactions was identified as Garcia-Molina and Salem, Sagas, DOI https://doi.org/10.1145/38713.38742, 1987. The Princeton-hosted PDF fetch was inaccessible during this research session; I therefore did not rely on its detailed contents to establish any claim. The DOI/ACM metadata was accessible. 16

Contradictions, unresolved gaps, article angles, and freshness triggers

There is no fundamental contradiction between provider claims of exactly-once behavior and the recommendation to retain application-level business controls. There is a terminology trap.

SQS FIFO uses exactly-once language around duplicate messages within its deduplication mechanism. 5 Pub/Sub defines exactly-once delivery for a particular subscription mode and region while separately warning about publish-side duplicates. 6 Kafka can atomically coordinate offsets and output records inside Kafka, then explicitly requires external-system cooperation outside that boundary. 11 BigQuery can reject duplicate appends by offset within a committed write stream, while saying nothing about whether a payment API called by the same application executed twice. 7

So the defensible article language is not:

“Exactly once is impossible.”

Nor:

“Exactly once solves duplicates.”

It is:

Exactly-once claims have a boundary. Name the boundary before using the claim to reason about a business outcome.

A second apparent tension is fast acknowledgment versus durable acceptance. Stripe and Shopify want webhook receivers to respond quickly rather than perform complex synchronous processing. 2 That should not be rewritten into “return 200 before preserving the event.” The stronger editorial design is to make the synchronous path tiny: authenticate, durably record accepted work, return success, process asynchronously. This is an inference from the combination of provider delivery semantics and the end-to-end argument rather than an exact vendor-prescribed transaction architecture. 1

A third tension is ordering versus throughput/availability. Ordering is useful, but Pub/Sub documents restrictions and latency implications, while exactly-once plus ordered acknowledgment adds further throughput constraints. 6 Ordering should therefore not become an expensive substitute for a source revision when the business source already knows which state is newer.

The largest unresolved evidence gaps are empirical rather than semantic. Public primary documentation is strong on provider behavior; it is weak on the questions leaders ultimately care about: how many duplicate charges a specific architecture prevents, what reconciliation costs a five-person company, when a DLQ pays for itself, or whether an AI triage assistant reduces mean recovery time without introducing dangerous advice. Those values should be measured locally rather than invented.

The recommendation also weakens or reverses under several conditions. A single transactional local database update does not warrant a distributed recovery framework. Best-effort analytics may rationally tolerate loss. A mature native integration may be safer and cheaper than custom glue. Very latency-sensitive applications can prefer loss over delayed retries, as Saltzer et al.’s real-time voice example illustrates. 1 And inside a single transactional Kafka boundary, built-in exactly-once semantics can replace a significant amount of custom deduplication logic. 11

Candidate distinct article angles:

Angle	Core reader payoff
The workflow is green. The customer was charged twice.	Makes the central distinction between execution telemetry and business correctness concrete.
Exactly once where?	Explains SQS, Pub/Sub, Kafka and BigQuery as different scoped guarantees rather than arguing over the phrase abstractly.
The four identities an automation needs	Delivery ID, source event ID, business entity ID and semantic operation ID; add source revision as the ordering dimension.
Cancellation arrived before confirmation	A practical explanation of source revision versus arrival order, stale events and reconciliation.
Two automations are triggering each other	Meaningful transitions, no-op suppression, causation metadata and an emergency hop fuse.
The API is rate-limiting us: what waits and what must survive?	Frames backpressure through business deadline and loss tolerance rather than retry settings.
Retrying a failed payment may be the thing that duplicates it	Timeout-after-side-effect ambiguity, status lookup, idempotency lifetime and compensation.
Where AI helps recovery—and where I would not let it click	Deterministic correctness path plus bounded AI exception triage.

A useful compact editorial asset is the following “recovery sentence” template:

When [failure] happens,
we preserve [durable evidence],
decide current truth using [business identity + source revision],
retry only [safe operation],
verify success by observing [business outcome],
and if that cannot be established by [deadline],
[owner] performs [reconcile / compensate / quarantine / manual repair].

That template forces an explanation such as:

When a refund request times out, we keep the refund intent and its idempotency identity, do not infer failure from the timeout, query or safely repeat the same operation, and close it only after the payment provider and order ledger reconcile. If the result remains ambiguous after the safe automated path, Order Operations owns manual reconciliation.

That is substantially more informative than “the workflow retries three times.”

Freshness triggers for this dossier should be provider-contract changes rather than an arbitrary calendar interval. Recheck the evidence when Stripe changes idempotency-key semantics or webhook retry behavior; Google changes Pub/Sub exactly-once/ordering, Cloud Tasks retention/retry behavior, or BigQuery Storage Write/CDC semantics; AWS changes SQS deduplication or Lambda recursion handling; Kafka changes transaction/exactly-once behavior; or the bounded business process gains a new irreversible side effect. The reviewed Google Pub/Sub pages were updated as recently as 2026-10-09 and the BigQuery/Cloud Tasks pages on 2026-10-07, one to three days before this snapshot, underscoring that provider semantics deserve version-aware maintenance. 6

Bottom-line decision: explain recovery in the vocabulary of “Which real-world state is true? What irreversible action occurred? What remains uncertain? Who repairs it, and by when?” Treat retries, idempotency keys, queues, exactly-once modes and green workflow runs as mechanisms and evidence underneath that contract—not as the contract itself. This conclusion is supported both by the foundational end-to-end systems argument and by the sharply bounded guarantees documented by current webhook, queue, streaming and warehouse providers. 1

Portable citation index

1. [web.mit.edu](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf) 2. [docs.stripe.com](https://docs.stripe.com/webhooks) 4. [docs.stripe.com](https://docs.stripe.com/api/idempotent_requests) 3. [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110.html) 5. [Exactly-once processing in Amazon SQS - Amazon Simple Queue Service](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html) 15. [Use Lambda recursive loop detection to prevent infinite loops - AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/invocation-recursion.html) 6. [Exactly-once delivery  |  Pub/Sub  |  Google Cloud Documentation](https://cloud.google.com/pubsub/docs/exactly-once-delivery) 7. [Introduction to the Storage Write API (gRPC)  |  BigQuery  |  Google Cloud Documentation](https://cloud.google.com/bigquery/docs/write-api) 10. [Understand Cloud Tasks  |  Google Cloud Documentation](https://cloud.google.com/tasks/docs/dual-overview) 12. [Order messages  |  Pub/Sub  |  Google Cloud Documentation](https://cloud.google.com/pubsub/docs/ordering) 13. [Stream table updates with change data capture ingestion  |  BigQuery  |  Google Cloud Documentation](https://cloud.google.com/bigquery/docs/change-data-capture) 8. [Verify webhook deliveries](https://shopify.dev/docs/apps/build/webhooks/subscribe/https) 9. [Timeouts, retries, and backoff with jitter | AWS Builder Center](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/) 11. [Design | Apache Kafka](https://kafka.apache.org/43/design/design/) 14. [Configure queue routing, limits, and retries  |  Cloud Tasks  |  Google Cloud Documentation](https://docs.cloud.google.com/tasks/docs/configuring-queues) 16. [Sagas | Proceedings of the 1987 ACM SIGMOD ...](https://dl.acm.org/doi/10.1145/38713.38742?utm_source=chatgpt.com)

Import verification and asset availability

Genuine Deep Research completed in 8 minutes and was captured on 2026-10-10 after the first due time. Original export, exact independent prompt, canonical chat and citation URLs are retained. Complete inline examples/code were preserved; no missing sandbox download target is claimed as a delivered asset. Report code, test outcomes and execution assertions were not independently run or verified during this import. Delivery is not independent claim verification; Evidence status remains Captured.

Retries, Duplicates and Idempotency: Evidence Dossier for Recovery Design

Research snapshot: 2026-10-10
Perspective: Academic, with primary provider documentation where implementation semantics matter
Decision supported: Explain recovery design through observable business outcomes, not workflow “green run” status
Active-work scope: A small team operating an event/webhook-driven order workflow into a cloud analytical warehouse and downstream business automation
Evidence boundary: Public sources only. No private systems, customer data, vendor tests, firsthand execution, or claimed customer outcomes. Worked timelines, fixtures, cost examples, and fault injections below are synthetic and reasoned, not executed.

Concise answer and uncertainty

The strongest defensible recommendation is:

A recovery design is correct when the intended business state can be demonstrated after retries, duplicates, reordered events, partial failures and repair—not when every automation run is green.

That distinction follows from several independent bodies of evidence.

Saltzer, Reed and Clark’s end-to-end argument is directly relevant: functions such as duplicate suppression, recovery and acknowledgement can exist in lower layers, but lower-layer correctness can be insufficient for the application’s end-to-end requirement. 1 HTTP makes a similarly narrow distinction: an idempotent method is one whose intended server effect is the same when repeated; it does not promise that every internal side effect or response is identical. 2 Modern messaging and warehouse products reinforce the boundary problem. Pub/Sub exactly-once is scoped to supported pull subscriptions and a region; BigQuery exactly-once is obtained through stream offsets; Kafka’s adopted KIP scopes idempotence and transactions to particular producer/log and consume-transform-produce boundaries. 3 None of those guarantees, by itself, proves that a customer was charged once, a shipment exists once, the latest order state won, or a cancellation was honored.

A useful design therefore needs at least four different identities:

Identity	Example	What it answers
Transport/message identity	source + event_id	“Have I seen this delivered event before?”
Business entity identity	order_id = O-7421	“Which real-world thing is this about?”
Source version	revision = 43	“Which known state supersedes which?”
Operation intent identity	fulfillment:create:O-7421:L1	“Is this retry the same business action or a new action?”

CloudEvents explicitly defines an event identity through the combination of source and id, and also notes that one occurrence can result in more than one event. That is a useful formal separation between an event record and the underlying business occurrence. 4 Stripe makes the same distinction operationally: duplicate delivery of one event can be detected by event ID, but two separate Event objects may describe the same relevant change, in which case Stripe recommends looking at the underlying object ID plus event type. 5 AWS likewise reports that byte-for-byte-equivalent requests are not necessarily the same intent; its preferred API pattern is a caller-supplied request identifier. 6

The main uncertainties are not about whether duplicates and disorder occur—they are explicitly documented by major providers—but about application semantics. A generic infrastructure layer cannot decide whether a late confirmation should override a cancellation, whether two identical-looking resource creates are intentional, whether a sent message can be safely repeated, or what compensation means. Those rules have to come from the process owner and source-of-truth contract. Lamport’s foundational work explains why receiving messages in a particular order is not itself proof of business causality, and modern systems such as Stripe explicitly decline to guarantee event delivery order. 7

There is also no evidence in the researched sources that adding an AI agent improves this process’s recovery correctness. The defensible AI role here is therefore bounded exception triage and explanation, behind deterministic controls and human approval—not payment authorization, messaging, or production writes.

For Czech versus US use, the core distributed-systems conclusions do not materially differ. The relevant technical distinctions found here are provider, region and configuration boundaries—for example Pub/Sub’s same-region exactly-once condition—not Czech-versus-US engineering principles. 3 Legal record-retention, accounting, payment-dispute and consumer-communication differences were outside this technical dossier and should not be inferred from it.

Evidence synthesis around the decision

Start with the business process, not the automation

A bounded process makes “recovery” concrete.

Synthetic process: maintain the current order state in an analytical warehouse and, when an order first becomes confirmed, create one downstream fulfillment case. If that order is subsequently cancelled, no active fulfillment case may remain. The source commerce/order system is authoritative.

Process element	Bounded specification
Trigger	Source emits an order-created, order-updated or order-cancelled webhook/event.
Inputs	Transport event ID and source; order_id; source revision/version where available; order lifecycle; status; payload/schema version; occurrence time; received time.
Business owner	Order/operations lead owns the meaning of “confirmed,” “cancelled,” reopen rules and acceptable recovery delay.
Technical owner	Data/automation engineer owns durable ingestion, reducers, queues, observability and repair tooling.
Exception owner	Named operations person or rota owns unresolved discrepancies and external side effects whose result is unknown.
Business state	For this fixture: UNKNOWN → CREATED → CONFIRMED → CANCELED. CANCELED is terminal for lifecycle L1; intentional reopening creates lifecycle L2. This is a synthetic policy, not a universal order rule.
Observable completion	Authoritative source and warehouse agree on latest relevant source revision; required downstream action exists no more than once; obsolete downstream action is compensated; every durably accepted event has a terminal disposition or named unresolved owner.
Manual fallback	Operator reads the authoritative order, downstream object and operation ledger; chooses apply/ignore/compensate; records reason and repair ID.

A green workflow run is deliberately absent from the completion definition. It is useful telemetry, but it is not the business invariant.

This is a direct application of the end-to-end principle rather than a claim that the 1984 paper studied SaaS automation: Saltzer et al. explicitly discuss duplicate message suppression, crash recovery and acknowledgement among their examples and argue that application requirements determine where correctness must ultimately be established. 1 The translation from that systems principle to “verify the order, shipment or payment outcome” is an editorial design inference, not an empirical effect measured by the paper.

Three ways to operate the same process

The right comparison is not “automation versus nothing.” It is the simplest viable baseline versus deterministic automation versus bounded AI assistance.

Dimension	Manual/native baseline	Deterministic automation	Bounded AI triage
Intake	Native order dashboard/webhook log; periodic review/export	Webhook → durable inbox	Same deterministic intake
State choice	Operator checks source	Reducer compares business identity + source revision	Same reducer
Duplicate handling	Operator notices/reconciles	Unique transport ID plus semantic operation ledger	Same controls
External effect	Operator creates/cancels fulfillment case	Deterministic outbox/worker	AI forbidden to execute it
Ambiguous order	Operator inspects source	Rule sends to exception queue	AI summarizes evidence/proposes repair
Rate limit	Operator waits	Durable queue + deadlines + backoff	AI may classify backlog, not bypass quota
Audit	Native histories + manual notes	Event, decision, operation and repair ledgers	AI recommendation stored as advisory evidence
Upkeep	Human process training	Schema, API, retries, runbooks, monitors	All deterministic upkeep plus model/prompt/evaluation upkeep
Best fit	Low volume, high ambiguity, inexpensive manual work	Repetitive semantics and meaningful consequence of duplicate/loss	High exception-reading burden where human remains accountable

The manual/native approach is a serious alternative. For five consequential exceptions a month, a boring dashboard and checklist can be better engineering than an event platform. Automation adds software states, provider contracts, queues, credentials, schemas and on-call responsibilities. The Dataflow authors make a related, though much larger-scale, observation: correctness, latency and cost are competing dimensions, not a free bundle. 8

The AI variant is not a fourth reliability layer. It inherits the deterministic workflow’s safety envelope. A reasonable bounded task is:

“Here are the source order, accepted events, warehouse row and downstream operation ledger. Explain the discrepancy, cite the conflicting revisions, and recommend ignore-stale, re-fetch-source, retry-safe-operation, compensate, or human-review.”

The assistant does not authorize a payment, send a customer message, cancel a shipment, or write production state. No primary evidence reviewed here establishes that an AI assistant improves recovery rates or lowers total lifecycle cost for this process, so any such benefit remains an evaluation hypothesis.

Retry safety, idempotency and exactly-once are not synonyms

The terms are often collapsed in automation discussions, and that is dangerous.

Retry safety asks whether repeating an attempted operation after uncertainty is acceptable. HTTP explicitly permits automatic retry reasoning for idempotent methods after communication failure. 2

Idempotency says repeated expression of the same intent has the same intended effect. It does not mean the request occurred only once. Stripe’s API, for example, lets clients attach an idempotency key so a failed connection can be retried without creating the same object or update twice; for API v1, keys can be pruned once they are at least 24 hours old, after which reuse can be treated as a new request. Stripe also rejects same-key retries with mismatching parameters. 9 AWS’s Builders Library similarly emphasizes caller-provided intent identifiers and warns that idempotency records cannot reasonably be retained forever. 6

Deduplication asks whether a receiver recognizes a delivery as one already seen. Cloud Tasks, for example, rejects a duplicate named task and remembers deleted task names for up to 24 hours; SQS FIFO suppresses duplicate sends within its documented five-minute deduplication interval. 10 Those windows are infrastructure features, not a universal lifetime for business identity.

Exactly-once only has meaning together with its boundary and assumptions. Pub/Sub’s feature applies to pull/StreamingPull and its guarantee is regional; it relies on Pub/Sub message identity and successful acknowledgements. 3 BigQuery’s default Storage Write API stream is at-least-once, while application-created streams can obtain exactly-once write semantics through offsets; Google’s documentation explicitly describes the added control and complexity. 11 Kafka’s adopted KIP explains how producer IDs and sequence numbers prevent duplicate persistence to the Kafka log, and how transactions can make Kafka consume-transform-produce operations atomic; it also explicitly limits some consumer-side implications and describes those guarantees from particular Kafka viewpoints. 12

This creates an important counterexample to simplistic advice: provider exactly-once features are useful. The answer is not “never trust infrastructure.” The answer is “do not silently expand its guarantee.” A BigQuery offset can establish that a row was not appended twice to one write stream; it does not establish that the source sent only one business occurrence or that a customer was charged once. That conclusion is an inference from the documented guarantee boundary. 11

Compensation is different again. Some business actions cannot be transactionally rolled back with your database. The original Saga work is foundational here; the original SIGMOD DOI is 10.1145/38713.38742, and the Princeton technical-report version is publicly surfaced as TR-070-87. The ACM page was inaccessible to this research session and the Princeton PDF could not be rendered reliably, so I do not rely on uninspected passages from it. 13 Current Microsoft architecture documentation describes the resulting pattern explicitly: record how to undo completed steps, recognize that compensation itself can fail, make compensating steps retry-safe where possible, and allow manual intervention where automation cannot safely recover. 14 This is provider guidance rather than empirical proof.

Failure must be modeled before recovery can be trusted

Distributed execution creates uncertainty even under surprisingly strong assumptions. FLP’s theorem is often abused, so its scope matters: Fischer, Lynch and Paterson prove a nontermination possibility for deterministic consensus in a completely asynchronous model with one crash, and the paper expressly assumes reliable exactly-once message delivery for the theorem’s model. 15 Their conclusion explicitly says the result does not mean distributed problems cannot be solved in practice; rather, practical systems introduce stronger assumptions or weaker requirements. 15 It therefore does not prove that “exactly once is impossible.” It does demonstrate why words such as timeout, failure and committed outcome depend on system assumptions.

Lamport provides the complementary ordering result: in a distributed system, causality gives a partial order, and messages can be received in a different order from the order a naive central observer expects. 7 Stripe turns this from theory into a current webhook contract: it does not guarantee delivery in generation order and instructs consumers not to depend on that order. 16 Pub/Sub’s default contract is likewise at-least-once with no ordering guarantee, unless the relevant ordering feature is explicitly configured. 17

For the bounded process, that yields this failure model:

Failure	Unsafe naive behavior	Safe criterion
Same event delivered twice	Run action twice	Same transport ID has one terminal processing disposition
Two different events represent same business effect	Event-ID dedup misses it	Operation intent/business identity prevents duplicate effect
Cancellation arrives before older confirmation	“Last message processed wins” resurrects order	Higher authoritative source revision wins
No source revision exists	Guess from receive time	Fetch authoritative current state or escalate ambiguity
Receiver replies success before durable persistence	Event can vanish after accepted response	Persist accepted work before acknowledging provider
Receiver persists then response is lost	Provider retries	Duplicate intake returns success without repeating effect
External side effect succeeds but client times out	Blind retry can duplicate payment/message/shipment	Reuse stable idempotency key or inquire by business operation ID before retry
Two workers race	Both observe “not done”	Atomic unique operation claim / transactional update
Dependency is down	Drop or hot-loop retries	Durable backlog, bounded retry, deadline, owner
Provider returns 429	Immediate retries worsen pressure	Honor Retry-After where provided; otherwise bounded backoff plus jitter
Full ingestion history is replayed	Old commands rerun	Reducer/version semantics plus durable operation history make replay safe
Human corrects production state	Automation immediately undoes correction	Override is a versioned/auditable state considered by automation
Compensation fails	Mark original workflow successful anyway	Compensation remains open with an owner until verified
Automation A triggers B triggers A	Infinite writes/cost	Trigger on meaningful transitions, provenance guard, bounded hops/circuit breaker

The rate-limit behavior has direct standards and provider support. HTTP 429 means too many requests and may, not must, include Retry-After. 18 Cloud Tasks rate limits apply both to initial attempts and retries, use queue dispatch controls, and retry failed tasks with exponential backoff; Google also warns that bursts can exceed a nominal steady dispatch rate and produce 429 or 503 responses. 19 AWS reports from its own operational practice that correlated retries can recreate overload and recommends adding jitter to spread retry attempts. 20 This is provider-reported operational experience, not a controlled cross-vendor trial.

Feedback loops are also not hypothetical infrastructure behavior. AWS EventBridge documentation explicitly warns that a rule can respond to a change, make another change and thereby fire itself indefinitely. 21 The business-safe solution should therefore not merely impose max_runs=5. It should stop defining every record mutation as a meaningful business transition.

Worked recovery design and portable synthetic assets

Worked example: one order, several identities

This section is synthetic and unperformed.

Order O-7421, lifecycle L1, has authoritative source revisions:

revision 41: CREATED
revision 42: CONFIRMED
revision 43: CANCELED
revision 44: CANCELED + metadata-only update

Transport events:

{"source":"shop.example/orders","event_id":"evt-41","order_id":"O-7421","lifecycle":"L1","revision":41,"type":"order.created","status":"CREATED"}
{"source":"shop.example/orders","event_id":"evt-42","order_id":"O-7421","lifecycle":"L1","revision":42,"type":"order.updated","status":"CONFIRMED"}
{"source":"shop.example/orders","event_id":"evt-43","order_id":"O-7421","lifecycle":"L1","revision":43,"type":"order.canceled","status":"CANCELED"}
{"source":"shop.example/orders","event_id":"evt-43","order_id":"O-7421","lifecycle":"L1","revision":43,"type":"order.canceled","status":"CANCELED"}
{"source":"shop.example/orders","event_id":"evt-42","order_id":"O-7421","lifecycle":"L1","revision":42,"type":"order.updated","status":"CONFIRMED"}
{"source":"shop.example/orders","event_id":"evt-44","order_id":"O-7421","lifecycle":"L1","revision":44,"type":"order.updated","status":"CANCELED","note":"address metadata corrected"}

The transport key is:

(shop.example/orders, evt-43)

The business entity key is:

order = O-7421, lifecycle = L1

The source-state version is:

revision = 43

The operation identities are intentionally different:

fulfillment:create:O-7421:L1
fulfillment:cancel:O-7421:L1

That is crucial. evt-42 and fulfillment:create:O-7421:L1 answer different questions. The event could be redelivered with the same event ID, represented by another provider event ID, or followed by metadata-only events. The business action should still occur once.

CloudEvents’ source + id rule is useful for the first identity, but its specification does not claim that event identity is business-operation identity. 22 AWS’s request-identifier experience supports making intent explicit rather than deriving intent merely from identical parameters. 6

Synthetic duplicate / late / cancel timeline

Assume the events are received in this order:

10:00:00  evt-41 / rev 41 / CREATED
10:00:03  evt-43 / rev 43 / CANCELED
10:00:04  evt-43 / rev 43 / CANCELED    [transport duplicate]
10:00:09  evt-42 / rev 42 / CONFIRMED   [late]
10:00:15  evt-44 / rev 44 / CANCELED    [metadata only]

A dangerous reducer says:

current_status = status_of_last_received_event

At 10:00:09, that produces CONFIRMED: a green run and a wrong order.

The safe synthetic reducer is closer to:

if event.revision < stored.revision:
    disposition = STALE_IGNORED

elif event.revision == stored.revision
     and payload_semantics_already_applied:
    disposition = DUPLICATE_OR_EQUIVALENT

else:
    atomically:
        update business state to event.revision
        record meaningful transition
        enqueue any transition-driven side effect

After the timeline, the required observable result is:

warehouse.order_revision        = 44
warehouse.order_status          = CANCELED
active_fulfillment_cases        = 0
duplicate_fulfillment_creates   = 0
unresolved_accepted_events      = 0

evt-44 advances source version but does not cause a second “cancel fulfillment” action because the relevant business state did not transition from active to cancelled.

This is also why event timestamps are a poor universal resolver. Stripe explicitly warns its consumers not to infer event ordering from its snapshot-event creation timestamp and tells them not to depend on webhook delivery order. 16 Dataflow was designed for large-scale input where old data may arrive or be retracted and emphasizes explicit correctness/latency/cost trade-offs rather than assuming an eventually neat arrival sequence. 8

Durable acceptance and the transaction boundary

The smallest deterministic architecture I would recommend is:

Source webhook
     |
     v
[verify/authenticate]
     |
     v
[durable inbox INSERT] ----> only then acknowledge receipt
     |
     v
[state reducer]
     |
     +---- atomic DB transaction ----+
     |                               |
     | update current order state    |
     | insert semantic outbox row    |
     +-------------------------------+
                     |
                     v
              [rate-limited queue]
                     |
                     v
             [side-effect worker]
                     |
          stable operation identity
                     |
          +----------+-----------+
          |                      |
          v                      v
   [known outcome]       [unknown outcome]
          |                      |
     record result      inquire/reconcile
                                 |
                          retry or human

Stripe requires webhook endpoints to return a successful status quickly, before lengthy business processing. 16 The specific recommendation to put a minimal durable write before that success response is an editorial recovery design: otherwise “accepted” can mean “the remote sender will stop trying” while the local process has no recoverable record. Complex processing remains asynchronous, consistent with Stripe’s recommendation to queue webhook work. 16

The local transaction boundary should include, where supported by the datastore:

BEGIN;

INSERT INTO accepted_event (...)
ON CONFLICT (event_source, event_id) DO NOTHING;

-- If genuinely new and newer than known source state:
UPDATE order_state
SET source_revision = :revision,
    status          = :status
WHERE order_id = :order_id
  AND source_revision < :revision;

-- Only on a meaningful transition:
INSERT INTO operation_outbox (
    operation_key,
    operation_type,
    order_id,
    status
)
VALUES (
    'fulfillment:create:O-7421:L1',
    'FULFILLMENT_CREATE',
    'O-7421',
    'PENDING'
)
ON CONFLICT (operation_key) DO NOTHING;

COMMIT;

This SQL is a portable conceptual asset, not tested against a named SQL engine. Syntax and concurrency semantics must be adapted and verified for the actual database.

The external fulfillment API is deliberately outside that transaction. A database transaction cannot magically roll back an already-created shipment at an independent provider. The operation ledger therefore needs a state such as:

PENDING
IN_FLIGHT
SUCCEEDED
UNKNOWN_EXTERNAL_OUTCOME
COMPENSATION_REQUIRED
COMPENSATED
HUMAN_RESOLVED

A simple binary SUCCESS/FAILURE is inadequate.

Timeout after an external side effect

Synthetic execution:

10:01:00.000 worker calls create_fulfillment(op_key=fulfillment:create:O-7421:L1)
10:01:00.300 provider creates case FC-9981
10:01:00.320 response packet is lost
10:01:05.000 client times out

The worker does not know whether nothing happened or FC-9981 exists.

Blind retry is safe only when the destination gives the required idempotent contract for the same business intent and the key is still valid. Stripe documents this pattern for API requests; AWS describes the same uncertainty and the value of stable caller request IDs. 9

Without such a contract, safe repair is:

1. Query destination by operation/business identity if possible.
2. If exactly one matching effect exists -> record SUCCEEDED.
3. If none exists -> retry according to policy.
4. If multiple effects exist -> stop; compensate/escalate.
5. If destination cannot establish outcome -> HUMAN_REVIEW,
   unless business policy explicitly accepts duplicate risk.

This is particularly important for payments and messages. A payment call that timed out after a charge was accepted is not made safe merely because the surrounding workflow failed red. A second call with a different idempotency key is a new operation from an API perspective. Stripe’s documented key-retention semantics illustrate why a provider key cannot be treated as permanent business memory. 9 Likewise, deduplicating an inbound webhook does not itself deduplicate an unrelated outbound email, SMS, payment or shipment.

Rate-limited queue: what may wait and what must not disappear

The queue should separate durability from dispatch urgency.

For this process, a synthetic business classification could be:

Work	May wait?	May disappear?	Deadline behavior
Raw accepted event	Yes briefly	No	Persist before normal async work
Cancellation reconciliation	Limited	No	Escalate when deadline approaches
Payment/outbound-effect uncertainty	Limited	No	Stop blind retry; reconcile
Warehouse freshness update	Usually	No, unless periodic rebuild is authoritative	Backlog acceptable within freshness SLO
Nonessential enrichment	Yes	Potentially, by explicit business policy	Expire/cancel instead of infinite retry
AI exception summary	Yes	Yes; human can inspect raw evidence	Never block core recovery

HTTP 429 alone does not say how valuable a request is; that prioritization is application policy. The standard says a 429 response may include Retry-After. 18 Cloud Tasks provides dispatch-rate and concurrency limits and applies rate controls to retries as well as first attempts. 19

A bounded retry calculation should be visible. One illustrative policy is:

base delay b = 2 seconds
attempt index n = 0..7
cap C = 300 seconds

uncapped(n) = b * 2^n
capped(n)   = min(C, uncapped(n))
actual wait = jitter(0, capped(n))

So the caps before jitter are:

2 s, 4 s, 8 s, 16 s, 32 s, 64 s, 128 s, 256 s

Total maximum nominal pre-jitter waiting if the full cap were used at every listed attempt:

2 + 4 + 8 + 16 + 32 + 64 + 128 + 256 = 510 seconds

Those numbers are synthetic policy inputs, not provider recommendations. The evidence-backed principles are bounded retries, backoff for repeated failure, and jitter to avoid synchronized retry storms. 20

Do not let “max retries exhausted” become “finished.” The terminal state is instead something such as:

DEAD_LETTER:
  business_deadline: 2026-10-10T10:15:00Z
  reason: downstream_429_exhausted
  source_revision: 43
  owner: order-ops
  repair_action: reconcile_current_source_before_replay

The dead-letter queue is a recovery queue, not a success queue. That is an editorial design rule, not a provider guarantee.

Feedback-loop timeline

Synthetic loop:

T0 Automation A:
   sees "order row changed"
   -> writes CRM field crm_synced=true

T1 Automation B:
   sees "CRM order changed"
   -> writes warehouse field crm_last_seen=T1

T2 Automation A:
   sees "order row changed"
   -> writes CRM again

T3 Automation B:
   ...

AWS documents the same failure shape in EventBridge: an event rule can cause a change that recreates the event that triggered the rule, resulting in an infinite loop. 21

A safer design is:

trigger_on =
    meaningful_business_transition
    AND origin != self
    AND transition_id not already applied

with metadata such as:

{
  "correlation_id": "order-O-7421-L1",
  "causation_id": "evt-42",
  "origin": "order-sync-A",
  "semantic_transition": "CREATED->CONFIRMED",
  "hop_count": 1
}

Safe repair criteria are not merely “loop stopped.” They are:

authoritative order state still correct
downstream state reconciled
unwanted loop-generated mutations identified
no legitimate business change discarded
trigger rule corrected
replay performed from known source state, not blindly from loop history

Idempotency-key lifetime must match the failure horizon

Provider windows differ sharply. Stripe API v1 keys may be pruned after at least 24 hours; Cloud Tasks remembers deleted task names for up to 24 hours; SQS FIFO deduplication covers a documented five-minute interval. 9 AWS’s Builders Library explicitly discusses late requests and says retention needs vary by service and resource. 6

Therefore:

provider idempotency window ≠ business deduplication horizon

The local retention decision should be driven by:

H_business =
max(
    automatic_retry_horizon,
    manual_resend_horizon,
    expected_late_delivery_horizon,
    supported_replay/backfill_horizon,
    external_outcome_reconciliation_horizon
)

subject to legitimate storage/privacy constraints.

Stripe itself illustrates why this can be much longer than an automatic retry window: its webhooks retry automatically for up to three days, while dashboard and CLI manual resend mechanisms can extend substantially beyond that. 16 A 24-hour local dedup table would therefore be an unsafe universal assumption for such a webhook process.

Reproducible fault-injection evaluation

This is a proposed test protocol; it was not run.

Known truth:

order_id: O-7421
lifecycle: L1
authoritative_final_revision: 44
authoritative_final_status: CANCELED
expected_active_fulfillment_cases: 0
expected_ai_production_writes: 0

Run the same fixture under these injections:

Injection	Mechanism	Required result
Duplicate	Deliver every event twice	Same final state; same semantic effect count
Out of order	Deliver rev43 before rev42	Rev42 cannot restore confirmed state
Replay	Replay rev41–44 from scratch after completion	No duplicate downstream operation
Crash before durable acceptance	Kill receiver before inbox commit and before ack	Sender retry can recover event
Negative control: ack before persistence	Return success, then crash before write	Test should expose unrecoverable gap
Ack lost after persistence	Persist, then make response disappear	Redelivery safely deduplicated
Race	Process rev42 and rev43 concurrently	Final reducer state rev43/44 cancelled
External side effect + timeout	Destination commits, response lost	No second effect without outcome check/idempotency
Dependency outage	Fail destination for 15 minutes	Accepted work survives and ages visibly
Quota	Return HTTP 429 repeatedly	Backoff; deadline visible; no silent loss
Loop	Make A’s write trigger B and vice versa	Guard/circuit stops loop; business state reconciled
Human override	Operator marks operation held	Automation does not silently overwrite override
Compensation failure	Cancel effect fails	Case stays unresolved with owner; not green-complete

The key assertions should query business truth, for example:

-- Pseudocode: exact syntax engine-dependent.

-- Latest source revision materialized:
assert warehouse_order.revision = 44;
assert warehouse_order.status = 'CANCELED';

-- Business effect:
assert count(active_fulfillment
             where order_id='O-7421'
               and lifecycle='L1') = 0;

-- Recovery accountability:
assert count(accepted_events
             where terminal_disposition is null
               and age > recovery_deadline) = 0;

A test that merely asserts:

workflow_runs.failed = 0

does not test the decision this dossier is intended to support.

Practical implications, ownership and lifecycle cost

The smallest workable implementation

For a small team, I would not begin with Kafka transactions, a workflow engine, a general saga platform, an agent framework and five observability products. Provider features can be excellent, but complexity itself has an upkeep cost. BigQuery explicitly characterizes application-created streams as offering additional control, including exactly-once behavior, at additional complexity. 23

The smallest implementation worth evaluating is:

1. One durable inbox table keyed by transport identity.
2. One current-state table keyed by business entity/lifecycle with source revision.
3. One operation/outbox table keyed by stable business-operation intent.
4. One rate-limited worker queue with bounded retries.
5. One reconciliation query/job comparing source truth, warehouse state and unresolved side effects.
6. One exception view/runbook with a named owner.

No AI is needed to establish the core recovery properties.

The system can later adopt Pub/Sub exactly-once, BigQuery offsets, Kafka transactions, FIFO queues or provider idempotency where their guarantees reduce code. Pub/Sub can eliminate duplicates within its documented exactly-once boundary; BigQuery offsets can eliminate repeated writes within a write stream; Kafka transactions can make Kafka-side consume-transform-produce atomic. 3 That should delete redundant custom machinery, not weaken the end-to-end invariant.

Observability should distinguish processing health from outcome health

A useful dashboard would have two panels.

Transport/processing signals:

webhook delivery attempts
accepted events
duplicate transport events
queue depth
oldest queue age
retry count
429/5xx rate
dead-letter count
worker failure rate

Business/recovery signals:

orders behind authoritative source revision
confirmed orders missing required fulfillment
canceled orders with active fulfillment
operation intents with UNKNOWN_EXTERNAL_OUTCOME
uncompensated operations
accepted events past recovery deadline
manual overrides awaiting reconciliation
source-to-warehouse divergence age

Only the second group answers whether recovery achieved the business outcome. This separation is an application of the end-to-end principle and documented provider guarantee boundaries, not a published empirical finding that this exact dashboard improves a measured KPI. 1

Ownership cannot be delegated to the queue

A recovery design should state, before launch:

Business invariant owner:
    decides what correct business state means

Automation owner:
    owns deterministic processing, retries and observability

Exception owner:
    resolves UNKNOWN_EXTERNAL_OUTCOME / semantic ambiguity

Provider owner:
    tracks API/schema/guarantee changes

Incident decision:
    who may pause processing, replay, compensate or retire automation

This matters because infrastructure can identify “retry exhausted” but cannot generically decide whether a cancelled hotel booking should trigger a refund, another booking, or human contact. Modern compensating-transaction guidance explicitly notes that compensation is application-specific and sometimes needs human intervention. 14

Lifecycle cost should be measured in upkeep, not just build hours

A reusable staff-hour model is:

H_{year1}
=
H_{build}
+
12H_{monthly\ ops}
+
N_{changes}H_{change}
+
N_{incidents}H_{recovery}

and for a mature year:

H_{recurring}
=
12H_{monthly\ ops}
+
N_{changes}H_{change}
+
N_{incidents}H_{recovery}

A more complete financial model is:

C =
H \times C_{loaded\ hour}
+
C_{provider}
+
C_{incident\ impact}

where all three terms must use a declared currency and period if actual monetary comparison is performed.

A deliberately synthetic sensitivity example:

Manual/native:
  recurring reconciliation = 8 staff-hours/month
  build                    = 0 h
  first-year staff-hours   = 8 × 12 = 96 h

Deterministic:
  initial build            = 48 h
  routine upkeep           = 2 h/month = 24 h/year
  provider/schema changes  = 4/year × 3 h = 12 h
  recovery incidents       = 2/year × 2 h = 4 h
  year-one total           = 48 + 24 + 12 + 4 = 88 h

Bounded AI layer added:
  integration/evaluation   = 24 h
  additional upkeep        = 1 h/month = 12 h
  two evaluation changes   = 2 × 4 h = 8 h
  incremental year-one     = 44 h

Under these invented assumptions only, deterministic automation saves eight staff-hours in year one versus the manual process, while adding the AI layer is economically worse unless it saves more than:

44/12 \approx 3.67\ \text{staff-hours/month}

elsewhere.

Those numbers are not estimates of a real implementation. They show the calculation that a real pilot should fill with observed maintenance and recovery time.

Upkeep is not theoretical. Stripe webhook payload structure depends on API versioning, and BigQuery documents behavior that applications need to handle when table schemas change during streaming. 16 Kafka semantics are configuration- and version-sensitive; the adopted KIP itself was last updated in March 2026. 12 Provider contracts therefore belong in the maintenance inventory.

A sensible retirement condition is also part of the design:

Retire/simplify when:
- native integration now provides the required invariant;
- volume/value no longer justifies upkeep;
- source semantics become too ambiguous for deterministic operation;
- recovery effort exceeds labor saved;
- provider/API changes invalidate the safety contract.

Before shutdown:
- stop new intake deliberately;
- drain or explicitly dispose of accepted work;
- reconcile source and downstream truth;
- resolve outstanding compensation;
- retain operation evidence for the applicable recovery/record horizon;
- document which system becomes authoritative afterward.

These are editorial operational recommendations rather than findings from a controlled study.

Claim-to-primary-source dossier

The table distinguishes what the evidence actually establishes from what an article should say. Inspected date for every row: 2026-10-10.

Candidate claim	Primary source and exact scope	Source date	Evidence type	Assessment	Editorial action
“A green lower-level run does not establish the application outcome.”	Saltzer, Reed & Clark, End-to-End Arguments in System Design, abstract/introduction/conclusion. Low-level duplicate suppression/recovery may be redundant or insufficient relative to application requirements. https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf DOI: 10.1145/357401.357402 1	Nov 1984	Foundational design argument, not experiment	Supported as interpretation	Keep, qualify: translate principle to outcome verification; do not claim the paper studied workflow dashboards.
“Arrival order is not necessarily causal/business order.”	Lamport, Time, Clocks, and the Ordering of Events in a Distributed System, “The Partial Ordering” / logical clocks. https://lamport.azurewebsites.net/pubs/time-clocks.pdf DOI: 10.1145/359545.359563 7	Jul 1978	Formal/distributed-systems model	Supported	Keep. Do not stretch it into a claim that every SaaS webhook is unordered.
“Even reliable exactly-once messaging does not eliminate distributed coordination difficulty.”	Fischer, Lynch & Paterson, §§1 and 5. The model actually assumes correct exactly-once message delivery yet proves a nontermination possibility for asynchronous deterministic consensus with one crash. https://groups.csail.mit.edu/tds/papers/Lynch/jacm85.pdf DOI: 10.1145/3149.214121 15	Apr 1985	Formal impossibility theorem	Supported within strict assumptions	Qualify heavily. Never rewrite as “exactly once is impossible.”
“Long-running distributed actions may require compensation rather than rollback.”	Garcia-Molina & Salem, Sagas. Princeton TR-070-87 surfaced at https://www.cs.princeton.edu/techreports/1987/070.pdf; proceedings DOI 10.1145/38713.38742. Full primary-text inspection was impaired in this session. 24	1987	Foundational transaction design	Supported bibliographically; primary passage not fully inspected	Keep as foundational reference, qualify access gap. Current implementation detail should rely on an inspected source.
“Out-of-order/late data is a first-class processing concern and entails correctness/latency/cost choices.”	Akidau et al., The Dataflow Model, Abstract and §1. https://research.google.com/pubs/archive/43864.pdf DOI: 10.14778/2824032.2824076 8	2015	System model plus reported real-world design experience	Supported	Keep. Do not infer small-team ROI from Google-scale experience.
“Stateful stream processing can provide strong framework-level fault tolerance.”	Akidau et al., MillWheel, Google Research publication/abstract. https://research.google.com/pubs/pub41378.html DOI: 10.14778/2536222.2536229 25	2013	System paper / deployment report	Supported within framework boundary	Qualify: framework record processing is not an external payment guarantee.
“Transactional serverless workflows are technically feasible rather than merely theoretical.”	Zhang et al., Beldi, OSDI 2020, description/evaluation; tested applications over AWS Lambda. https://www.usenix.org/conference/osdi20/presentation/zhang-haoran 26	Nov 2020	Academic system implementation and controlled evaluation	Supported	Keep as feasibility evidence; not evidence that small teams should adopt equivalent machinery.
“HTTP idempotency is about intended effect, not identical execution history.”	RFC 9110 §9.2.2, “Idempotent Methods.” https://www.rfc-editor.org/rfc/rfc9110.html#section-9.2.2 2	Jun 2022	Internet standard	Supported	Keep.
“Message identity should not be conflated with business identity.”	CloudEvents 1.0.3-wip main specification, Context Attributes id/source; source + id uniquely identifies distinct events; one occurrence may generate multiple events. https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md 22	Current main snapshot	Open standard/specification	Supported	Keep, qualify: CloudEvents does not define your order ID or business revision.
“Webhook events can be duplicated.”	Stripe, “Handle duplicate events”; same event may arrive more than once; separate Events can also represent the relevant same object/type. https://docs.stripe.com/webhooks 5	Current docs	Provider documentation	Supported for Stripe	Keep provider-specific; do not claim every provider behaves identically.
“Webhook event generation/delivery order cannot safely be assumed.”	Stripe, “Event ordering”; delivery order is not guaranteed. https://docs.stripe.com/webhooks 16	Current docs	Provider documentation	Supported for Stripe	Keep. Generalization should be phrased conditionally.
“Idempotency keys have a lifetime and same key should represent the same intent.”	Stripe API Reference, “Idempotent requests”; API v1 keys may be pruned ≥24h and parameter mismatch is rejected. https://docs.stripe.com/api/idempotent_requests 9	Current docs	Provider documentation	Supported for Stripe API v1	Keep with API-version qualifier.
“Content equality alone cannot establish duplicate business intent.”	AWS Builders Library, “Making retries safe with idempotent APIs,” client request identifiers, late requests and same ID/different intent sections. https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/ 6	Current article	Attributable provider engineering practice	Supported as AWS-reported practice	Keep but label reported practice, not theorem.
“Exactly-once provider guarantees have explicit scope.”	Google Pub/Sub, “Exactly-once delivery”: pull/StreamingPull only, regional guarantee, acknowledgement requirements. https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery 3	Current docs	Provider documentation	Supported	Keep; central evidence.
“A warehouse can offer exactly-once writes without proving end-to-end business correctness.”	BigQuery Storage Write API: default stream at-least-once; application-created streams + offsets exactly-once. https://docs.cloud.google.com/bigquery/docs/write-api-grpc 11	Current docs	Provider documentation	Documented guarantee; end-to-end limitation is inference	Keep, explicitly separate the two.
“Kafka exactly-once has a defined transaction boundary.”	Apache Kafka KIP-98, “Motivation” and “Summary of Guarantees”; producer sequencing, cross-partition transactions and consume-transform-produce atomicity, with stated consumer caveats. https://cwiki.apache.org/confluence/spaces/KAFKA/pages/66854913/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging 12	Last updated Mar 4 2026	Adopted open-source design specification	Supported	Keep; do not extrapolate to arbitrary external effects.
“Default at-least-once delivery requires duplicate tolerance.”	AWS SQS standard queue documentation; copies can be redelivered and applications should be idempotent. https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html 27	Current docs	Provider documentation	Supported for SQS Standard	Keep provider-specific.
“SQS ‘exactly-once processing’ terminology has a finite deduplication scope.”	SQS FIFO documentation: retried SendMessage within five-minute deduplication interval does not introduce duplicates into queue. https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html 28	Current docs	Provider documentation	Supported with scope	Rewrite headline claims: “exactly-once within documented queue dedup semantics,” not business effect once.
“Queue throttling applies to retries too.”	Google Cloud Tasks, “Define rate limits” / “Set retry parameters.” https://docs.cloud.google.com/tasks/docs/configuring-queues 19	Current docs; page surfaced with 2026 update	Provider documentation	Supported	Keep.
“429 does not automatically mean Retry-After is present.”	RFC 6585 §4. https://www.rfc-editor.org/rfc/rfc6585.html#section-4 18	Apr 2012	Internet standard	Supported	Keep; say “honor if supplied,” not “always obey the header.”
“Jitter reduces synchronized retry pressure.”	AWS, Timeouts, retries, and backoff with jitter, “Jitter.” https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter 20	Jun 12 2026; modified Jun 15	Provider-reported engineering practice	Supported as reported practice	Keep, qualify evidence type.
“Two automations can trigger each other indefinitely.”	AWS EventBridge, “Avoid writing infinite loops.” https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-patterns-best-practices.html 21	Current docs	Provider documentation	Supported	Keep. The provenance/state-transition remedy is editorial design.
“Compensation may fail and sometimes requires a person.”	Microsoft Azure Architecture Center, “Compensating Transaction pattern.” https://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction 14	Apr 20 2026	Provider architecture guidance	Supported as guidance	Keep, label guidance rather than measured effect.

Direct assessment of the three draft claim targets

“Dvě automatizace se spouštějí navzájem. Jak zastavit nekonečné kolečko?”
Status: supported, but the remedy needs qualification. Event-driven infinite loops are explicitly documented by AWS. 21 The evidence supports the existence of the failure mode. “Track origin, trigger only on a meaningful state transition, suppress an already-applied transition, and add a circuit breaker” is an editorial architecture proposal, not a provider-tested universal recipe. Advice: keep the claim; qualify the solution.

“Systém omezuje počet požadavků. Co může počkat a co se nesmí ztratit?”
Status: supported plus editorial business classification. Standards and queue documentation support rate limiting, bounded dispatch and retry behavior. 18 They do not decide whether a cancellation is more important than a warehouse enrichment job. Advice: keep, explicitly say that infrastructure controls rate while the process owner defines deadlines and loss tolerance.

“Nejdřív přišlo zrušení, potom potvrzení. Který stav má automatizace použít?”
Status: strongly supported as a real class of problem; the answer is conditional. Distributed messages do not inherently give a reliable global business order, and Stripe specifically does not guarantee webhook delivery order. 7 Advice: rewrite any answer of “use the latest event” to “use the authoritative source version/business transition; if no trustworthy version exists, re-fetch authoritative state or escalate rather than guessing from arrival time.” The precise cancellation-vs-confirmation rule remains domain-specific.

Contradictions, limits and conditions where the recommendation reverses

The evidence does not support a maximalist “always build an idempotency platform” position.

Cases where simpler recovery is better

One transactional system. If the state update and all meaningful effects occur inside one database transaction, an outbox/saga architecture can be unnecessary overhead. The additional machinery exists mainly when atomicity stops at a system boundary.

Naturally idempotent overwrite. A well-defined PUT that sets a resource to a desired final representation is explicitly within HTTP’s idempotent-method model. 2 A complicated semantic operation ledger may add little if retries are already harmless and the result is directly verifiable.

Provider guarantee covers the entire required boundary. For a purely Kafka-internal transform, Kafka’s transaction boundary can be the relevant business boundary. 12 For a warehouse-only ingestion where duplicate rows are the sole concern, BigQuery stream offsets may resolve the exact problem. 11 In such cases, custom dedup code can be worse than using the provider correctly.

Low-value, low-volume process. If an operator spends one hour a month reconciling ten records and an error has trivial cost, manual/native operation can dominate an engineered recovery system on lifecycle cost. This is an economic inference, not an empirical threshold established by the literature reviewed here.

Best-effort analytics with authoritative rebuild. If an analytics table can safely be dropped and regenerated from an authoritative snapshot, elaborate event-level compensation may have little value. The recovery outcome is then “rebuild matches authoritative snapshot,” not “every event processed once.”

Cases where automatic retry itself should stop

The recommendation becomes more conservative when an external effect is high-impact, irreversible, or unqueryable.

Suppose a payment/transfer/message/shipment endpoint:

- may execute before the client observes timeout;
- has no usable idempotency contract;
- cannot be queried by your operation identity;
- has material duplicate cost.

Automatic retry can be the wrong recovery action. The safest state may be:

UNKNOWN_EXTERNAL_OUTCOME -> reconcile/human

rather than:

UNKNOWN_EXTERNAL_OUTCOME -> retry

AWS’s discussion of lost responses and stable request tokens shows why outcome uncertainty exists; provider idempotency contracts are one mechanism for resolving it, not a universal property of APIs. 6

“Exactly once is impossible” should be removed

That phrase is contradicted by the source record if used without a boundary. Pub/Sub, BigQuery, Kafka and SQS all document forms of exactly-once or duplicate suppression under explicit conditions. 3 FLP does not establish the claim either: its theorem concerns deterministic consensus termination under a particular asynchronous fault model, and its message subsystem is assumed reliable and exactly-once. 15

A better statement is:

Exactly once can be a real and valuable property inside a specified system boundary. The mistake is treating that boundary guarantee as proof that an externally visible business outcome occurred exactly once.

“Deduplicate by event ID” is insufficient as a universal rule

CloudEvents makes source + id a legitimate event-level duplicate identity, and Stripe recommends event IDs for duplicate receipts. 22 But Stripe also documents cases where different Event objects correspond to the relevant same object and event type. 5 AWS likewise warns that apparently identical requests can represent different intended resource creates. 6

Thus:

event deduplication
≠ business occurrence deduplication
≠ operation idempotency

An article that merges those should be rewritten.

“Last event wins” is not a general ordering strategy

Last authoritative source revision may win if the source contract gives a monotonic revision and the business process is a last-write state model. Last arrival at your webhook handler has no such general justification. Lamport establishes the distributed ordering problem formally; Stripe supplies a concrete current provider contract with unordered deliveries. 7

If the source exposes neither revision nor a reliably queryable current object, automatic conflict repair remains unresolved. A timestamp tie-breaker can produce deterministic behavior, but deterministic is not synonymous with correct.

Evidence gaps

No source reviewed here establishes a cross-industry effect size such as “idempotency reduces incidents by X%.” None should be invented.

No controlled study reviewed here compares manual workflows, deterministic automation and AI-assisted recovery for a small Czech or US business.

No AI reliability study was included that would justify autonomous exception repair in this process.

The Saga primary source was identified through ACM/Princeton, but ACM returned access restrictions and the Princeton PDF could not be reliably rendered in the research tooling. Its bibliographic role is retained, while detailed implementation claims are grounded in inspected contemporary documentation instead. 13

The Dataflow PDF was successfully inspected for its abstract/introduction but a subsequent rendered page request failed after a redirect; claims here are therefore limited to the text that was actually accessible. 8

Provider documentation documents provider behavior, not independent business outcomes. AWS Builders Library material is attributable engineering practice and experience, not an independent controlled experiment. Stripe/Google/AWS docs establish API contracts, not causal evidence that a proposed architecture reduces a customer’s financial loss.

The fault-injection plan, SQL, order fixture, cost calculations and repair timelines in this dossier are unperformed synthetic assets. They should not be described later as a test result.

Reusable article angles, assets and freshness triggers

Several distinct articles can legitimately be supported from this dossier without turning them into the same “idempotency explainer.”

Green workflow, wrong business

Thesis: “Finished successfully” describes execution, not necessarily recovery.

The narrative can start with the late-confirmation-after-cancellation fixture: every step returns 200, the workflow is green, yet a stale confirmation resurrects a cancelled order. The core intellectual backing is the end-to-end argument plus documented unordered webhook delivery. 1

The useful visual asset is:

green transport
+ green workflow
+ green queue
≠ verified business invariant

followed by:

source order = CANCELED rev43
warehouse    = CANCELED rev43+
fulfillment  = inactive
exceptions   = 0 past deadline
= observable recovery

Exactly once — exactly where?

Thesis: Every “exactly once” claim needs a box drawn around it.

One comparison graphic could show:

SQS FIFO       -> duplicate-send suppression window
Pub/Sub        -> supported message delivery/ack boundary
Kafka          -> producer/log + transactional Kafka boundary
BigQuery       -> stream/offset write boundary
Stripe key     -> one API operation intent during retained key lifetime
Business       -> source state + external side effects + compensation

The provider differences are documented and make this a strong anti-hand-waving article. 28

The cancellation arrived first

Thesis: Receive order is not business truth.

The article can distinguish:

event ID
order ID
source revision
receive timestamp
event timestamp
operation intent

Lamport supplies the foundational ordering perspective, while Stripe provides a concrete webhook case. 29

Retrying is load, not magic

Thesis: A retry consumes scarce capacity and can make an overloaded service worse.

Start from HTTP 429, move to a durable backlog and business deadlines, then use backoff/jitter rather than a hardcoded “retry three times.” RFC 6585, Cloud Tasks and AWS retry guidance provide primary support. 18

The editorial business question is the stronger hook:

Which work may become late, and which work may never silently disappear?

Two automations are arguing with each other

Thesis: Triggering on “record changed” is often too weak; trigger on meaningful state transitions.

AWS’s EventBridge warning gives a recognizable documented failure mode. 21 The synthetic A↔B timeline then makes provenance, causation IDs and semantic transition guards concrete.

Idempotency keys expire; business history may not

Thesis: Infrastructure deduplication windows are not the same as the period in which a duplicate business effect remains harmful.

Stripe’s ≥24-hour key behavior, Cloud Tasks’ up-to-24-hour task-name memory and SQS FIFO’s five-minute deduplication interval make the point without claiming any one provider is deficient. 9 They solve different problems with different scopes.

Useful reusable assets from this dossier

The following assets are complete enough to reuse without an external download:

Identity card

message identity   = source + transport event ID
business identity  = entity ID + lifecycle
source version     = authoritative revision/sequence
operation intent   = stable ID for one semantic side effect

Business terminal dispositions

APPLIED
TRANSPORT_DUPLICATE
SEMANTIC_DUPLICATE
STALE_IGNORED
SUPERSEDED
SIDE_EFFECT_SUCCEEDED
COMPENSATED
HUMAN_RESOLVED
UNRESOLVED   <-- never count this as completed

Recovery outcome checklist

[ ] Latest authoritative business state known
[ ] Warehouse agrees with authoritative source
[ ] Semantic side effect exists exactly as intended
[ ] Duplicate/obsolete effects absent or compensated
[ ] Every accepted item has a terminal disposition
[ ] Unknown external outcomes have a named owner
[ ] Human overrides are visible to automation
[ ] Replay has been tested against the same invariants

Fault fixture

The six-line JSONL O-7421 fixture above is the portable test fixture. It is synthetic and unperformed.

Fault matrix

The twelve injection cases above constitute a portable evaluation specification. They are unperformed.

Freshness triggers

Re-check this dossier rather than reusing provider claims unchanged when any of these occurs:

Trigger	What to re-verify
Stripe API/webhook version change	Idempotency-key retention, POST/DELETE semantics, webhook retry/resend horizon, event ordering and duplicate guidance. 9
Pub/Sub feature/config change	Subscription types supported by exactly-once, region restriction, ordering interaction, acknowledgement semantics. 3
BigQuery ingestion architecture change	Default-stream semantics, application-created offsets/transactions, schema behavior. 23
Kafka major-version/config change	Current implementation defaults versus the adopted KIP’s original configuration text and guarantee boundary. The KIP was updated March 4, 2026. 12
Cloud Tasks queue change	Task-name deduplication retention, dispatch/burst/retry behavior. 10
SQS FIFO change	Deduplication window and exact provider meaning of “exactly-once processing.” 28
Source adds/removes revision numbers	Conflict resolver and replay strategy; this can materially change whether automatic ordering is safe.
New external irreversible effect	Re-evaluate inquiry, idempotency and compensation before enabling retries.
AI role expands beyond advice	Require a separate evaluation of authorization boundaries, false repair recommendations, auditability and rollback before allowing any production write.

The durable editorial conclusion survives those freshness checks:

Retries are a transport/recovery mechanism. Idempotency is an operation contract. Deduplication is recognition of repeated identity. Exactly-once is a scoped processing guarantee. Compensation is a business repair action. None of them, alone, is evidence that the business outcome happened.

For leaders deciding what to automate, that yields a practical test: before discussing retry counts, queue products or agents, write down the authoritative business state, the externally observable effect, the acceptable exception state, and who repairs uncertainty. Then design recovery so those facts can still be demonstrated after the ugly timeline—not merely after the happy green run. 30

Portable citation index

1. [web.mit.edu](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf) 30. [web.mit.edu](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf) 2. [RFC 9110 - HTTP Semantics](https://httpwg.org/specs/rfc9110.html) 3. [Exactly-once delivery  |  Pub/Sub  |  Google Cloud Documentation](https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery) 10. [Understand Cloud Tasks  |  Google Cloud Documentation](https://docs.cloud.google.com/tasks/docs/dual-overview) 11. [Introduction to the Storage Write API  |  BigQuery  |  Google Cloud Documentation](https://docs.cloud.google.com/bigquery/docs/write-api-intro) 17. [Subscription overview  |  Pub/Sub  |  Google Cloud Documentation](https://docs.cloud.google.com/pubsub/docs/subscription-overview) 19. [Configure queue routing, limits, and retries  |  Cloud Tasks  |  Google Cloud Documentation](https://docs.cloud.google.com/tasks/docs/configuring-queues) 23. [Introduction to the Storage Write API (gRPC)  |  BigQuery  |  Google Cloud Documentation](https://docs.cloud.google.com/bigquery/docs/write-api-grpc) 4. [spec/cloudevents/spec.md at main · cloudevents/spec · GitHub](https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md) 22. [spec/cloudevents/spec.md at main](https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md?utm_source=chatgpt.com) 5. [docs.stripe.com](https://docs.stripe.com/events/manage-webhook-endpoints) 9. [docs.stripe.com](https://docs.stripe.com/api/idempotent_requests) 16. [docs.stripe.com](https://docs.stripe.com/webhooks) 6. [references-details-empty](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) 7. [lamport.azurewebsites.net](https://lamport.azurewebsites.net/pubs/time-clocks.pdf) 8. [research.google.com](https://research.google.com/pubs/archive/43864.pdf) 25. [MillWheel: Fault-Tolerant Stream Processing at Internet Scale](https://research.google.com/pubs/pub41378.html?ref=zuar.com) 12. [KIP-98 - Exactly Once Delivery and Transactional Messaging - Apache Kafka - Apache Software Foundation](https://cwiki.apache.org/confluence/display/KAFKA/KIP-98%2B-%2BExactly%2BOnce%2BDelivery%2Band%2BTransactional%2BMessaging) 13. [Sagas | ACM SIGMOD Record](https://dl.acm.org/doi/10.1145/38714.38742?utm_source=chatgpt.com) 14. [Compensating Transaction Pattern - Azure Architecture ...](https://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction?utm_source=chatgpt.com) 15. [groups.csail.mit.edu](https://groups.csail.mit.edu/tds/papers/Lynch/jacm85.pdf) 18. [RFC 6585: Additional HTTP Status Codes | RFC Editor](https://www.rfc-editor.org/info/rfc6585/) 20. [Timeouts, retries, and backoff with jitter | AWS Builder Center](https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter) 21. [Best practices for Amazon EventBridge event patterns - Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-patterns-best-practices.html) 27. [Amazon SQS at-least-once delivery - Amazon Simple Queue Service](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html) 28. [Exactly-once processing in Amazon SQS - Amazon Simple Queue Service](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html) 24. [Sagas - Princeton CS](https://www.cs.princeton.edu/techreports/1987/070.pdf?utm_source=chatgpt.com) 26. [Fault-tolerant and transactional stateful serverless workflows | USENIX](https://www.usenix.org/conference/osdi20/presentation/zhang-haoran) 29. [Time, Clocks and the Ordering of Events in a Distributed System - Microsoft Research](https://www.microsoft.com/en-us/research/publication/time-clocks-ordering-events-distributed-system/)

Import verification and asset availability

Genuine Deep Research completed in 8 minutes and was captured on 2026-10-10 after the first due time. Original export, exact independent prompt, canonical chat and citation URLs are retained. Complete inline examples/code were preserved; no missing sandbox download target is claimed as a delivered asset. Report code, test outcomes and execution assertions were not independently run or verified during this import. Delivery is not independent claim verification; Evidence status remains Captured.

Retries, Duplicates and Idempotency: Recovery Design as a Business Outcome

Practitioner evidence dossier · Research snapshot: October 10, 2026 · Scope: event/webhook-driven order workflow → analytical warehouse → downstream automation

Executive answer and uncertainties

Decision

Yes: recovery should be explained and designed around concrete business outcomes, not a green workflow run.

A green execution can mean little more than “this invocation returned successfully under the platform’s semantics.” Google’s Firebase documentation gives an unusually direct example: under one retry configuration, a background function can report successful execution and show 200 OK in logs even though the function encountered an error. Stripe, meanwhile, explicitly tells webhook handlers to return 2xx before doing complex downstream work such as updating an accounting system. Google Pub/Sub’s exactly-once feature is scoped to particular subscription and regional conditions and can still expose multiple copies arising from publish-side duplicates. Taken together, these documented behaviors make “run succeeded” an inadequate proxy for “the business process is correct.” 1

For the bounded order process in this dossier, the stronger completion test is:

The authoritative order state is represented at the newest accepted source revision, every required side effect has a known terminal disposition, prohibited or duplicate side effects did not occur, and any unresolved accepted work is visible to an owner who can repair it.

That definition is an editorial operational design, not a provider guarantee. It deliberately connects transport mechanics to observable business evidence.

The important distinction is therefore not “at-least-once versus exactly-once” in isolation. It is between five separate identities and promises:

Layer	Question it answers	Example
Transport delivery	“Have I seen this delivery attempt?”	delivery_id=del-17
Producer event	“Have I seen this event occurrence?”	source=shop, event_id=evt-91
Business entity	“Which thing changed?”	order_id=ORD-42
Source revision	“Which version of that thing is authoritative?”	revision=3
Operation intent	“Which external business action should happen at most once?”	notify_cancel:ORD-42:r3

This separation is supported by CloudEvents’ distinction between an event’s source + id and its subject, and by Debezium’s outbox model, which separately carries a unique event ID and an aggregate/business identity. CloudEvents says consumers may regard identical source + id pairs as duplicate events; it does not say that event identity is the same thing as business-object identity. 2

What the evidence supports strongly

Retries are appropriate mainly for transient failures, and retries without an idempotent contract can reproduce side effects. AWS explicitly describes retries for network, server and rate-limit faults while warning that the retry approach assumes the operation can be repeated without unintended side effects; its concrete example is accidentally creating two EBS volumes. AWS Prescriptive Guidance likewise recommends backoff for temporary unavailability and 429 throttling and says retrying non-idempotent operations risks corrupting state. 3

Duplicate protection at the message layer is not sufficient for duplicate protection at the business layer. Stripe says webhook deliveries may be duplicated, recommends tracking event IDs, and additionally notes that distinct Event objects can sometimes describe the same underlying object/event type. Its webhook documentation also explicitly says event delivery order is not guaranteed and warns against using the event created timestamp to establish ordering. 4

An “exactly-once” transport feature is a bounded provider contract, not proof of an exactly-once business side effect. Google Pub/Sub limits exactly-once delivery to pull subscriptions and a single-region subscriber condition, notes significantly higher latency, and says publish-side duplication can still produce multiple message copies. AWS SQS, by contrast, describes FIFO queues as providing “exactly-once message processing”; the differing scopes themselves are a reason not to translate a queue-level label into a claim that a payment, message, shipment, or warehouse update occurred exactly once. 5

Important uncertainty

The evidence base here is mainly primary provider documentation, standards work, and attributable provider engineering practice, because the question concerns contractual delivery and API semantics. This research did not identify or rely on a credible controlled empirical study establishing an effect size such as “idempotency lowers business incidents by X%.” No such effect size should be inferred from this dossier.

No real commerce platform, warehouse, payment processor, message provider, or customer system was accessed. The implementation below is a synthetic local SQLite execution, not a production benchmark or customer result.

The IETF Idempotency-Key work is useful conceptually but should not be treated as a finished standard in this snapshot. Draft -07 was published October 15, 2025 and states that it expired April 18, 2026; the draft itself says Internet-Drafts are works in progress. Its useful model is that a client-generated key lets a resource recognize retries, that keys must not be reused for a different payload, and that expiry semantics belong to the resource’s documented contract. 6

No material Czech-versus-US difference appears in the technical retry mechanisms examined. The more relevant distinction in the inspected evidence is provider/product configuration—for example, Google’s exactly-once regional boundary—rather than whether the operating company is Czech or American. Any legal, retention, or sector-specific treatment of stored event payloads is outside this dossier’s evidence scope. 5

Bounded process and automation choices

The recognizable process

The process is intentionally narrow:

A small commerce team receives order events from a source application, reflects the latest valid order state in a cloud analytical warehouse, and triggers a small number of downstream business automations.

The synthetic source produces CREATED, CONFIRMED, and CANCELLED events. Events can be retried, replayed, duplicated under a different delivery attempt, or delivered out of order. Stripe documents all of the relevant provider-side phenomena in a real webhook context: automatic retry, manual resend, duplicate events, and lack of guaranteed event ordering. 4

The proposed process contract is:

Element	Bounded definition
Trigger	Signed webhook/event from the order source
Core inputs	source, delivery_id, event_id, order_id, source_revision, intent, payload, occurrence time
Recovery metadata	operation_key, origin, causation_id, optional hop_count, processing disposition
Business owner	Commerce/operations owner responsible for the correctness of the order outcome
Technical owner	Data/platform owner responsible for durable ingestion, retries, reconciliation and audit evidence
Downstream state	Warehouse order projection plus durable side-effect intentions
Possible side effects	Notification, CRM/ERP update, fulfilment instruction, payment operation where applicable
Observable completion	Latest authoritative source revision represented; required side effects verified; no unresolved accepted work beyond the operating deadline
Manual fallback	Place order automation on hold; retrieve current source state; compare revision/audit trail; repair warehouse; reconcile external actions; compensate where necessary; record resolution

The ownership division above is recommended operating design, not documented vendor practice.

State rather than arrival order

A useful synthetic transition sequence is:

source rev 1: CREATED
source rev 2: CONFIRMED
source rev 3: CANCELLED

Now deliberately deliver it as:

arrives first:  rev 1 CREATED
arrives second: rev 3 CANCELLED
arrives third:  rev 2 CONFIRMED

The correct final state is CANCELLED @ revision 3. The late revision-2 confirmation is stale and must not resurrect the order.

That policy assumes the source provides a trustworthy monotonically comparable revision. Where it does not, a consumer should not manufacture certainty from arrival time. Stripe expressly says not to use its event created timestamp to determine order and recommends retrieving missing/current objects instead. 4

“Cancellation wins forever” is not a universal rule. If the domain permits a legitimate REOPENED @ revision 4, revision 4 should supersede revision 3. The invariant is newest authoritative business revision, not a hard-coded preference for one status.

Three viable operating models

Model	How it works	Recovery characteristics	Where I would use it
Manual/native baseline	Source UI remains authoritative; operator reviews current order and performs or approves downstream updates; simple native export/connector refreshes analytics	Humans naturally reconcile context; few distributed failure modes, but slower and labor-intensive; audit quality depends on process discipline	Low volume, rare/high-consequence actions, unstable process
Deterministic automation	Durable inbox → revision guard → warehouse projection + transactional outbox → idempotent/reconciled worker → audit/reconciliation	More engineering, but failure states become explicit and repeatable	Default choice once routine volume or latency justifies it
Bounded AI triage assistant	Reads dead-letter/reconciliation cases and audit evidence, groups likely cause, drafts repair steps; deterministic system or human executes	Can reduce cognitive work on messy exceptions but adds model/evaluation/configuration upkeep	Only after deterministic state and recovery evidence exist

The AI variant in this design cannot authorize payments, send customer messages, or write production state. It does not decide the authoritative revision. It may summarize, classify and propose a repair; the deterministic rules and human operator remain the execution boundary.

That is not merely conservatism. Idempotency correctness rests on stable identity, atomicity and business semantics. AWS’s account of idempotent APIs emphasizes that the idempotency token and related mutations need a sound atomic relationship, while transactional-outbox guidance exists specifically because a database mutation and message publication can otherwise split into inconsistent outcomes. These are poor places to substitute probabilistic inference for deterministic constraints. 3

Lifecycle burden

There is no defensible generic market benchmark for the maintenance hours of this design in the sources examined. The following is therefore an explicit synthetic planning model, useful for deciding whether automation is worth owning.

Let:

• (V) = routine events per month
• (m) = manual minutes per routine event
• (B_d) = deterministic automation build hours
• (U_d) = deterministic monthly upkeep hours
• (B_a) = incremental AI-assistant build hours
• (U_a) = incremental AI monthly upkeep hours
• (R) = residual annual exception/recovery hours

Then:

H_{manual} = \frac{12Vm}{60}

H_{det,Y1} = B_d + 12U_d + R

H_{AI,Y1} = B_d + B_a + 12(U_d + U_a) + R_{AI}

Illustrative assumptions only, not benchmarks: (V=500), (m=2) minutes, (B_d=40) h, (U_d=3) h/month, (B_a=12) h, (U_a=1.5) h/month.

That gives:

• manual routine floor = (500×12×2/60 = 200) h/year;
• deterministic floor before residual exception work = (40 + 12×3 = 76) h in year one;
• deterministic + AI floor before residual exception work = (40+12+12×(3+1.5)=106) h in year one.

On these assumptions, adding AI consumes another 30 h in year one, so it only makes economic sense if its reduction in triage effort or other benefit exceeds that incremental burden. Those figures are a sensitivity example, not evidence about a real implementation.

Maintenance should explicitly budget for source schema/revision changes, API-version changes, retry or idempotency-contract changes, new downstream actions, reconciliation drift, operating documentation and eventual retirement. Stripe’s current API documentation alone demonstrates why provider contract drift matters: its documented v1 and v2 idempotency semantics and retention windows differ. 7

Failure and recovery evidence

Retry safety is not the same thing as idempotency

These terms are often collapsed into one bucket, which is exactly how duplicate side effects escape into production.

Mechanism	What it actually buys	What it does not buy
Retry	Another chance after a transient failure	Safety from duplicate effects
Idempotent operation	Repetition produces the same intended state/effect under its contract	Infinite dedupe lifetime
Idempotency key	Links repeated calls to one intended operation within provider rules	Business-state ordering; protection after key expiry
Transport dedupe	Suppresses a repeated message/event identity	Two distinct events representing the same business intent
Business-operation dedupe	Suppresses repeated semantic action such as cancel-notice:order42:r3	Correct source ordering by itself
Exactly-once broker feature	Stronger delivery/processing semantics inside a defined product boundary	End-to-end exactly-once payments/messages/shipments
Compensation	Repairs a business effect that already happened	Erasure of the historical fact that it happened

AWS’s retry guidance explicitly says transient errors such as throttling, temporary network faults and temporary service unavailability are appropriate retry candidates and warns that operations used with retry should be idempotent. Its Builders Library makes the same point using side-effect duplication as the failure case. 8

The current Stripe API illustrates why an idempotency key also has a lifetime and version scope. Its /v2 documentation says same-key requests qualify as idempotent replays within 30 days under the documented account/API conditions, while it describes API v1 replay as a 24-hour window. It also documents differing replay behavior after failures between v1 and v2. 7

That matters operationally because Stripe’s webhook documentation separately permits CLI resends for events as old as 30 days. A system that can replay 30-day-old input but relies exclusively on an outbound API’s shorter deduplication window can reopen duplicate-side-effect risk. This is an inference across two Stripe contracts, not a claim that every Stripe integration necessarily has that bug. 4

The practical rule is therefore:

Set local business-operation retention from the maximum replay/reconciliation horizon, not merely from the provider’s shortest convenient cache window.

For long-lived financially or operationally meaningful actions, retaining the durable business operation identity with the order can be safer than treating it as a disposable transport deduplication record. This is editorial design supported by documented finite provider idempotency windows and AWS’s discussion of retaining idempotency knowledge according to resource/request lifetime. 3

Durable acceptance closes a dangerous gap

Stripe tells webhook consumers to verify the request and return 2xx quickly, before complex downstream processing. It retries live webhook delivery for up to three days with exponential backoff when delivery fails. 4

Those instructions should not be interpreted as “acknowledge while the only copy is still in process memory.” The smallest robust interpretation for this process is:

receive
→ authenticate/validate enough to accept
→ durably insert minimal inbox envelope
→ return 2xx
→ process asynchronously

That ordering is an editorial inference, not a Stripe-prescribed architecture. It follows the same principle Google documents for exactly-once processing—maintain progress information until acknowledgement—and the same atomicity motivation behind the transactional-outbox pattern. 5

The resulting business distinction is useful:

HTTP received          ≠ durable accepted
durable accepted       ≠ business state applied
state applied          ≠ external action completed
external API returned  ≠ business outcome necessarily verified
workflow green         ≠ process reconciled

The local transaction boundary should be explicit

Suppose a handler updates orders and then publishes “order cancelled” as an unrelated second write.

UPDATE orders ... COMMIT
publish notification intent ... CRASH

The order is cancelled locally but the downstream automation never learns about it.

Reverse the order:

publish notification intent
UPDATE orders ... CRASH

and the downstream action can happen for a business-state change that never committed.

AWS describes this as the dual-write problem and presents the transactional outbox specifically to bind the database mutation and notification intent together. It also warns that the outbox publisher can itself emit duplicates, so downstream consumers still need idempotency. 9

Debezium’s documented outbox structure reinforces the identity split: its default model includes a unique id that can be used for duplicate removal and a separate aggregateid for the business aggregate. 10

For a small team, the smallest workable transaction is therefore:

BEGIN
  claim source event
  compare source revision
  update local business projection
  insert semantic side-effect intent into outbox
COMMIT

A separate worker may deliver the outbox repeatedly, because repetition is made safe at the business operation boundary.

Timeout after an external side effect

This is the classic dangerous sequence:

T0  worker sends "capture payment for order 42"
T1  provider commits payment
T2  response packet is lost
T3  local worker sees timeout

At T3, “failed” is not a fact. The result is unknown.

Blindly issuing a new payment operation can duplicate the charge. Reusing the provider-supported idempotency identity is safer within its documented contract; Stripe explicitly documents idempotency as protection against unintended repeated side effects. 7

Where the external system supports lookup by a stable business/request reference, the recovery order should be:

1. query/reconcile the previous operation
2. if completion is proven → record success locally
3. if non-completion is proven → retry
4. if outcome remains ambiguous → escalate rather than guessing

AWS’s idempotent-API discussion explains why semantically equivalent responses are valuable after a response is lost and why the request identifier must be stored atomically with the operation. 3

If the duplicate payment has already happened, idempotency no longer repairs history. The remedy becomes a compensating business action, such as refunding/reversing the extra transaction where the domain permits it. Saga guidance explicitly distinguishes forward recovery by retry from backward recovery through compensating transactions. 11

The same distinction applies to messages. A second email cannot be “rolled back” after somebody has received it. The system can prevent another send, record the incident, or issue a corrective communication; compensation semantics are business-specific.

Out-of-order events need source meaning, not queue optimism

Stripe explicitly states that it does not guarantee webhook event delivery order and says not to infer ordering from its event timestamp. 4

For our synthetic order:

10:00:00  order r1 CREATED
10:00:02  order r3 CANCELLED     ← delivered now
10:00:05  order r2 CONFIRMED     ← late

Safe state transition:

current r1
receive r3 → apply CANCELLED, current=r3
receive r2 → classify stale; preserve r3

An ordinary UPSERT can provide the uniqueness/update primitive, but a correct implementation must add the revision predicate. SQLite documents that UPSERT can turn a uniqueness conflict into an update or no-op and can condition whether the update occurs; that is the mechanism used in the synthetic harness below. 12

If no trustworthy source revision/version exists, the safe repair criterion changes: fetch the authoritative current object from the source and reconcile local state rather than pretending local arrival order is authoritative. Stripe itself recommends retrieving missing objects when event order cannot be relied upon. 4

Rate limits are a queueing and deadline problem

A 429 should usually mean wait, not “lose the business action” and not “hammer harder.” AWS’s retry guidance treats throttling as a transient condition suited to backoff and cautions that frequent retry can worsen service degradation. 8

Google Pub/Sub’s flow-control documentation makes the backpressure problem concrete: when publish requests outpace client capacity, requests can accumulate in memory until they fail with DEADLINE_EXCEEDED; it recommends bounding outstanding messages/bytes and blocking or otherwise controlling publication pressure. 13

For business automation, each queued intention therefore needs two separate properties:

Durability: should the work survive waiting?

Deadline: how long is completing it still useful?

For example, in this synthetic policy:

warehouse analytics refresh     durable, can wait hours
customer cancellation notice    durable, high priority, escalate after 15 min
fraud-review request            durable, explicit business deadline
obsolete "order confirmed"      supersede if newer cancellation arrives first

The exact deadlines are business policy, not evidence from providers.

Backpressure then means “accept no more active work than the downstream capacity can safely service while preserving already accepted work,” rather than using unbounded in-memory accumulation as an accidental queue. Google’s documented memory accumulation/failure mode is a concrete reason for that distinction. 13

Two automations triggering each other

Consider:

Automation A:
CRM status changed → update warehouse flag

Automation B:
warehouse flag changed → update CRM status

A write from A wakes B.
B writes back and wakes A.
A wakes B...

Retries are not the correct cure. Retrying can make a permanent loop more persistent; Firebase explicitly warns that permanent bugs with retry enabled can remain in retry loops and recommends a defined termination condition. 1

The smallest workable guard is a combination of:

one authoritative writer per business field where possible
+ origin/workflow identity
+ causation/correlation identity
+ "already handled this causation" ledger
+ optional hop/deadline ceiling
+ only emit when a meaningful value actually changed

CloudEvents formally provides event source and id, and permits extension attributes for additional correlation/identity metadata. causation_id and hop_count in this dossier are editorial extensions, not CloudEvents-standard attributes. 2

A hop ceiling is only a fuse. The actual fix is usually to remove the circular ownership relationship or make the second write non-triggering/semantically idempotent.

Worked recovery cases and observable outcomes

The timelines below are synthetic fixtures reasoned from the documented behaviors above. They are not reports of production incidents.

Duplicate webhook

09:00:00  delivery D1, event E7, order O42, revision 1 CREATED
09:00:01  apply O42@r1
09:00:03  delivery D2, event E7, order O42, revision 1 CREATED

Transport/event dedupe sees the same producer event and records the second arrival without applying it again. CloudEvents explicitly permits a retried duplicate to use the same event id; Stripe similarly recommends event-ID tracking for duplicated deliveries. 2

Safe repair criterion: one state application, one operation intention, both delivery attempts auditable.

Two distinct events with the same business intent

E7   order O42 r3 CANCELLED
E99  order O42 r3 CANCELLED

Event-ID dedupe alone sees two different events. The revision/state guard and semantic operation key:

notify_cancel:O42:r3

still protect the downstream business action.

This matters because Stripe explicitly documents cases where two separate Event objects can correspond to the same underlying object/event type; its suggested duplicate handling therefore goes beyond merely storing one event ID. 4

Safe repair criterion: both source events remain auditable, but only one meaningful cancellation transition and one cancellation-notification intent exist.

Cancellation overtakes confirmation

received       business version
--------------------------------
CREATED        r1
CANCELLED      r3
CONFIRMED      r2  ← late

Safe state:

O42 = CANCELLED @ r3
r2 disposition = stale

Stripe’s documented absence of delivery-order guarantees is the real-provider analogue. 4

Safe repair criterion: warehouse/source reconciliation agrees on revision 3; no downstream confirmation generated from stale revision 2.

Acknowledged but work was lost

Unsafe design:

receive webhook
return 200
enqueue only in process memory
process crashes

Synthetic outcome: source thinks delivery was acknowledged while no durable recovery record exists.

Proposed safe boundary:

verify
BEGIN
  INSERT durable inbox envelope
COMMIT
return 2xx

Then the heavier state transition occurs asynchronously. Stripe requires quick responses before complex logic, while Google’s exactly-once guidance explicitly tells consumers to preserve processing-progress information until acknowledgement; the durable-inbox sequencing is this dossier’s synthesis of those constraints. 4

Safe repair criterion: every acknowledged accepted event can be found in durable storage with a terminal or pending disposition.

Timeout after payment/message side effect

T0 send operation with key K
T1 provider performs operation K
T2 response lost
T3 local status = UNKNOWN, not FAILED
T4 reconcile K / repeat safely under provider idempotency contract

Provider idempotency is the strongest option where supported. Stripe documents same-key protection with explicit API-version-specific retention semantics. 7

Safe repair criterion: exactly one externally verified semantic operation is associated with K. If two exist, mark incident and use a defined compensation procedure rather than retrying again.

Quota exhaustion

T0 cancellation action accepted durably
T1 provider → 429
T2 action stays queued
T3 exponential backoff + jitter
T4 retry succeeds

The relevant success is not “the T1 workflow failed cleanly.” It is “the cancellation action remained durable during throttling and either completed before its business deadline or escalated visibly.” Backoff for rate-limit/transient failures and bounded flow control are directly supported by AWS and Google guidance. 8

Feedback loop

cause C7 enters A
A emits change tagged cause C7
B processes C7 and emits derived change tagged cause C7
A sees C7 again
A's seen-causation guard blocks another execution

A defined end condition is consistent with Firebase’s retry-loop guidance; carrying the causation metadata is an editorial implementation using CloudEvents-compatible extension metadata. 1

Safe repair criterion: loop terminates deterministically, legitimate unrelated changes still pass, and the audit trail shows which workflow stopped the repeated causation.

What the dashboard should report

A useful operational dashboard is outcome-oriented:

Signal	Why it is closer to business truth
source_latest_revision - warehouse_latest_revision	Detects stale analytical/business state
Accepted events with no terminal disposition	Finds acknowledged-but-unfinished work
Outbox age by business deadline	Shows work at risk even when workers are “healthy”
Unknown external side effects	Exposes timeout-after-commit ambiguity
Duplicate operation-key conflicts	Detects repeated business intent
Dead-letter count and oldest age	Shows recovery obligations
Reconciliation mismatches	Compares automation state to authoritative/external reality
Compensation outstanding	Shows business effects not yet repaired
Loop-block events	Shows circular automation rather than hiding it as successful runs

This dashboard is a recommended evidence model, not a provider feature.

An especially useful invariant is:

accepted event
    → applied
    | duplicate
    | stale
    | superseded
    | dead-lettered
    | manually resolved

There should be no invisible eighth state called “probably fine.”

Smallest workable implementation and recovery runbook

Minimal architecture

For a small team, I would not begin with a globally ordered event platform, distributed saga engine, or autonomous agent.

The smallest implementation I would trust for this process is:

Webhook
   │
   ▼
durable inbox / event claim
   │
   ├── duplicate event → audit + stop
   │
   ▼
source revision guard
   │
   ├── stale → audit + stop
   │
   ▼
atomic transaction
   ├── latest order projection
   └── durable outbox operation
             │
             ▼
        bounded worker
        retry/backoff
        reconcile unknown result
             │
     ┌───────┴─────────┐
     ▼                 ▼
 external verified   dead letter
     │                 │
     ▼                 ▼
   DONE            human recovery

The local state update and outbox write are one transaction because distributed dual writes can otherwise diverge; AWS explicitly documents this failure and also notes that duplicate outbox delivery remains possible, hence the idempotent consumer/operation boundary. 9

Recovery runbook

For a duplicate: establish whether it is the same transport delivery, the same producer event, or a distinct event expressing the same business revision/intent. Do not collapse those identities into one column. CloudEvents and Debezium both explicitly separate event IDs from broader source/aggregate identity. 2

For stale/out-of-order state: compare the source revision with current applied revision. If the source contract does not provide a safe revision, retrieve current source state before repair; do not sort arbitrary webhook timestamps and call that correctness. Stripe explicitly cautions against timestamp-based event ordering. 4

For timeout after side effect: mark result UNKNOWN; first reconcile using the external idempotency/business reference. Retry only when the operation’s contract makes the retry safe or non-completion is established. AWS’s idempotent API guidance describes this lost-response problem directly. 3

For rate limiting: retain the work, honor the provider’s documented throttling behavior where available, back off with jitter, enforce a business deadline, and escalate instead of silently dropping expired high-value work. AWS documents throttling as a retry/backoff use case; Google documents bounded flow control as protection from request accumulation. 8

For permanent failure: stop automated retries and dead-letter it with reason, payload reference, operation key, owner and repair instructions. Firebase explicitly distinguishes transient retry candidates from permanent bugs/errors and recommends end conditions. 1

For partial completion: decide between forward recovery and compensation based on business semantics. AWS saga guidance formally separates those two modes. 11

For feedback loops: pause the offending route if necessary, identify the shared causation chain, determine the authoritative writer, then eliminate or suppress the circular transition. A retry limit alone only limits damage; it does not fix ownership.

For replay: replay from the durable accepted record, never from an unaudited reconstruction. Preserve the original event identity and business revision while creating a distinct replay attempt identity. Confirm that the external idempotency horizon still covers the operation; Stripe demonstrates that such horizons can be finite and version-dependent. 7

Complete portable Python/SQLite harness

Evidence classification: firsthand synthetic execution.

The following code uses only the Python standard library and SQLite. It does not contact real systems.

It was executed during this research session. The executed fixtures verified:

initial create                    → applied
same transport delivery replay    → duplicate_delivery
same source event, new delivery   → duplicate_event
revision 3 cancellation           → applied
late revision 2 confirmation      → stale
timeout after remote side effect  → reconciled without second effect
429                               → retained and later completed
permanent synthetic failure       → dead letter
A → B → A feedback path           → second A blocked
final order                       → CANCELLED @ revision 3
external semantic effects         → exactly 2 fixture effects:
                                   notify_created:r1
                                   notify_cancel:r3
asserted invariants               → PASS

This is not evidence that any cloud provider behaves exactly like the simulated remote_effects table. It tests the proposed local recovery invariants.

#!/usr/bin/env python3
"""
Synthetic recovery harness: Python 3 standard library + SQLite only.

Models:
- delivery identity vs producer event identity
- source revision ordering
- durable event claim
- atomic local projection + outbox
- timeout after external side effect
- reconciliation before retry
- synthetic 429 with backoff
- dead-letter handling
- causation-based feedback-loop protection

No network access or real provider systems are used.
"""

import json
import random
import sqlite3

DB = ":memory:"
rng = random.Random(7)

db = sqlite3.connect(DB)
db.row_factory = sqlite3.Row

db.executescript("""
PRAGMA foreign_keys=ON;

CREATE TABLE deliveries(
  delivery_id TEXT PRIMARY KEY,
  source TEXT NOT NULL,
  event_id TEXT NOT NULL,
  order_id TEXT NOT NULL,
  revision INTEGER NOT NULL,
  intent TEXT NOT NULL,
  disposition TEXT NOT NULL,
  note TEXT
);

CREATE TABLE claimed_events(
  source TEXT NOT NULL,
  event_id TEXT NOT NULL,
  PRIMARY KEY(source,event_id)
);

CREATE TABLE orders(
  order_id TEXT PRIMARY KEY,
  revision INTEGER NOT NULL,
  state TEXT NOT NULL,
  source_event_id TEXT NOT NULL
);

CREATE TABLE outbox(
  op_key TEXT PRIMARY KEY,
  order_id TEXT NOT NULL,
  revision INTEGER NOT NULL,
  action TEXT NOT NULL,
  status TEXT NOT NULL DEFAULT 'pending',
  attempts INTEGER NOT NULL DEFAULT 0,
  next_at REAL NOT NULL DEFAULT 0,
  deadline_at REAL NOT NULL,
  external_ref TEXT,
  last_error TEXT
);

-- Simulates a queryable external provider keyed by our business operation key.
-- In production this state would be remote, outside the local transaction.
CREATE TABLE remote_effects(
  op_key TEXT PRIMARY KEY,
  external_ref TEXT NOT NULL
);

CREATE TABLE dead_letters(
  op_key TEXT PRIMARY KEY,
  reason TEXT NOT NULL
);

CREATE TABLE loop_seen(
  workflow TEXT NOT NULL,
  causation_id TEXT NOT NULL,
  PRIMARY KEY(workflow,causation_id)
);

CREATE TABLE audit(
  seq INTEGER PRIMARY KEY AUTOINCREMENT,
  kind TEXT NOT NULL,
  ref TEXT NOT NULL,
  detail TEXT NOT NULL
);
""")

STATE_FOR = {
    "CREATED": "CREATED",
    "CONFIRMED": "CONFIRMED",
    "CANCELLED": "CANCELLED",
}

ACTION_FOR = {
    "CREATED": "notify_created",
    "CONFIRMED": "analytics_sync",
    "CANCELLED": "notify_cancel",
}


def audit(kind, ref, **detail):
    db.execute(
        "INSERT INTO audit(kind,ref,detail) VALUES(?,?,?)",
        (kind, ref, json.dumps(detail, sort_keys=True)),
    )


def ingest(e):
    """
    Durably accept one delivery, claim its producer event, then apply only a
    newer authoritative source revision.
    """

    try:
        with db:
            db.execute(
                """
                INSERT INTO deliveries(
                    delivery_id,source,event_id,order_id,
                    revision,intent,disposition
                )
                VALUES(?,?,?,?,?,?,?)
                """,
                (
                    e["delivery_id"],
                    e["source"],
                    e["event_id"],
                    e["order_id"],
                    e["revision"],
                    e["intent"],
                    "accepted",
                ),
            )
    except sqlite3.IntegrityError:
        audit(
            "duplicate_delivery",
            e["delivery_id"],
            event_id=e["event_id"],
        )
        db.commit()
        return "duplicate_delivery"

    with db:
        cur = db.execute(
            """
            INSERT OR IGNORE
            INTO claimed_events(source,event_id)
            VALUES(?,?)
            """,
            (e["source"], e["event_id"]),
        )

        if cur.rowcount == 0:
            db.execute(
                """
                UPDATE deliveries
                SET disposition=?, note=?
                WHERE delivery_id=?
                """,
                (
                    "duplicate_event",
                    "same source+event_id",
                    e["delivery_id"],
                ),
            )
            audit(
                "duplicate_event",
                e["delivery_id"],
                event_id=e["event_id"],
            )
            return "duplicate_event"

        row = db.execute(
            """
            SELECT revision,state
            FROM orders
            WHERE order_id=?
            """,
            (e["order_id"],),
        ).fetchone()

        if row and e["revision"] <= row["revision"]:
            same_semantics = (
                e["revision"] == row["revision"]
                and STATE_FOR[e["intent"]] == row["state"]
            )

            disposition = (
                "semantic_duplicate"
                if same_semantics
                else "stale"
            )

            db.execute(
                """
                UPDATE deliveries
                SET disposition=?, note=?
                WHERE delivery_id=?
                """,
                (
                    disposition,
                    f'current_revision={row["revision"]}',
                    e["delivery_id"],
                ),
            )

            audit(
                disposition,
                e["delivery_id"],
                order_id=e["order_id"],
                incoming_revision=e["revision"],
                current_revision=row["revision"],
            )
            return disposition

        # Local business projection and side-effect intention are atomic.
        db.execute(
            """
            INSERT INTO orders(
                order_id,revision,state,source_event_id
            )
            VALUES(?,?,?,?)
            ON CONFLICT(order_id) DO UPDATE SET
                revision=excluded.revision,
                state=excluded.state,
                source_event_id=excluded.source_event_id
            """,
            (
                e["order_id"],
                e["revision"],
                STATE_FOR[e["intent"]],
                e["event_id"],
            ),
        )

        # Pending work from an older source revision may no longer be
        # meaningful after the newer business state arrives.
        db.execute(
            """
            UPDATE outbox
            SET status='superseded',
                last_error='newer source revision'
            WHERE order_id=?
              AND revision<?
              AND status IN ('pending','retry')
            """,
            (e["order_id"], e["revision"]),
        )

        action = ACTION_FOR[e["intent"]]

        # This is a BUSINESS operation identity, not a transport ID.
        op_key = (
            f'{action}:{e["order_id"]}:r{e["revision"]}'
        )

        # Synthetic business deadlines.
        deadline = (
            900.0
            if action == "notify_cancel"
            else 86400.0
        )

        db.execute(
            """
            INSERT OR IGNORE INTO outbox(
                op_key,order_id,revision,action,deadline_at
            )
            VALUES(?,?,?,?,?)
            """,
            (
                op_key,
                e["order_id"],
                e["revision"],
                action,
                deadline,
            ),
        )

        db.execute(
            """
            UPDATE deliveries
            SET disposition='applied'
            WHERE delivery_id=?
            """,
            (e["delivery_id"],),
        )

        audit(
            "applied",
            e["delivery_id"],
            order_id=e["order_id"],
            revision=e["revision"],
            state=STATE_FOR[e["intent"]],
            op_key=op_key,
        )

        return "applied"


class RateLimit(Exception):
    pass


class Permanent(Exception):
    pass


fault_plan = {}


def remote_call(op_key):
    """
    Synthetic external provider.

    timeout_after_commit:
      records the external effect,
      commits it,
      then loses the response.
    """

    plan = fault_plan.setdefault(
        op_key,
        ["success"],
    )

    outcome = (
        plan.pop(0)
        if plan
        else "success"
    )

    if outcome == "rate_limit":
        raise RateLimit(
            "429 synthetic quota"
        )

    if outcome == "permanent":
        raise Permanent(
            "400 synthetic permanent failure"
        )

    ref = (
        "ext_"
        + op_key.replace(":", "_")
    )

    db.execute(
        """
        INSERT OR IGNORE INTO remote_effects(
            op_key,external_ref
        )
        VALUES(?,?)
        """,
        (op_key, ref),
    )

    # Represents commit in a different system.
    db.commit()

    if outcome == "timeout_after_commit":
        raise TimeoutError(
            "response lost after remote side effect"
        )

    return ref


def run_worker(now):
    rows = db.execute(
        """
        SELECT *
        FROM outbox
        WHERE status IN ('pending','retry')
          AND next_at<=?
        ORDER BY rowid
        """,
        (now,),
    ).fetchall()

    for r in rows:
        op_key = r["op_key"]

        # Reconcile a possible previous external success BEFORE repeating.
        seen = db.execute(
            """
            SELECT external_ref
            FROM remote_effects
            WHERE op_key=?
            """,
            (op_key,),
        ).fetchone()

        if seen:
            with db:
                db.execute(
                    """
                    UPDATE outbox
                    SET status='done',
                        external_ref=?,
                        last_error=NULL
                    WHERE op_key=?
                    """,
                    (
                        seen["external_ref"],
                        op_key,
                    ),
                )

                audit(
                    "reconciled",
                    op_key,
                    external_ref=seen["external_ref"],
                )

            continue

        attempt = r["attempts"] + 1

        if now > r["deadline_at"]:
            with db:
                db.execute(
                    """
                    UPDATE outbox
                    SET status='dead'
                    WHERE op_key=?
                    """,
                    (op_key,),
                )

                db.execute(
                    """
                    INSERT OR REPLACE
                    INTO dead_letters
                    VALUES(?,?)
                    """,
                    (
                        op_key,
                        "deadline exceeded",
                    ),
                )

                audit(
                    "dead_letter",
                    op_key,
                    reason="deadline exceeded",
                )

            continue

        try:
            ref = remote_call(op_key)

            with db:
                db.execute(
                    """
                    UPDATE outbox
                    SET status='done',
                        attempts=?,
                        external_ref=?,
                        last_error=NULL
                    WHERE op_key=?
                    """,
                    (
                        attempt,
                        ref,
                        op_key,
                    ),
                )

                audit(
                    "effect_done",
                    op_key,
                    attempt=attempt,
                    external_ref=ref,
                )

        except RateLimit as ex:
            # Bounded exponential backoff plus deterministic jitter.
            wait = (
                min(
                    60.0,
                    2 ** (attempt - 1),
                )
                + rng.random()
            )

            with db:
                db.execute(
                    """
                    UPDATE outbox
                    SET status='retry',
                        attempts=?,
                        next_at=?,
                        last_error=?
                    WHERE op_key=?
                    """,
                    (
                        attempt,
                        now + wait,
                        str(ex),
                        op_key,
                    ),
                )

                audit(
                    "rate_limited",
                    op_key,
                    attempt=attempt,
                    retry_in=round(wait, 3),
                )

        except TimeoutError as ex:
            with db:
                db.execute(
                    """
                    UPDATE outbox
                    SET status='retry',
                        attempts=?,
                        next_at=?,
                        last_error=?
                    WHERE op_key=?
                    """,
                    (
                        attempt,
                        now + 1.0,
                        str(ex),
                        op_key,
                    ),
                )

                audit(
                    "unknown_result",
                    op_key,
                    attempt=attempt,
                )

        except Permanent as ex:
            with db:
                db.execute(
                    """
                    UPDATE outbox
                    SET status='dead',
                        attempts=?,
                        last_error=?
                    WHERE op_key=?
                    """,
                    (
                        attempt,
                        str(ex),
                        op_key,
                    ),
                )

                db.execute(
                    """
                    INSERT OR REPLACE
                    INTO dead_letters
                    VALUES(?,?)
                    """,
                    (
                        op_key,
                        str(ex),
                    ),
                )

                audit(
                    "dead_letter",
                    op_key,
                    reason=str(ex),
                )


def feedback_step(
    workflow,
    causation_id,
    origin,
    hop,
    max_hops=4,
):
    """
    Return metadata for the next synthetic event,
    or None when the feedback chain is blocked.
    """

    if hop >= max_hops:
        audit(
            "loop_blocked_hop_limit",
            causation_id,
            workflow=workflow,
            hop=hop,
        )
        db.commit()
        return None

    try:
        with db:
            db.execute(
                """
                INSERT INTO loop_seen(
                    workflow,causation_id
                )
                VALUES(?,?)
                """,
                (
                    workflow,
                    causation_id,
                ),
            )

            audit(
                "loop_processed",
                causation_id,
                workflow=workflow,
                origin=origin,
                hop=hop,
            )

    except sqlite3.IntegrityError:
        audit(
            "loop_blocked_seen",
            causation_id,
            workflow=workflow,
            hop=hop,
        )
        db.commit()
        return None

    return {
        "origin": workflow,
        "causation_id": causation_id,
        "hop": hop + 1,
    }


# -----------------------------
# Complete synthetic fixtures
# -----------------------------

create = dict(
    delivery_id="del-1",
    source="shop",
    event_id="evt-create",
    order_id="ord-42",
    revision=1,
    intent="CREATED",
)

# Same producer event, different transport attempt.
create_retry = dict(
    create,
    delivery_id="del-1b",
)

cancel = dict(
    delivery_id="del-3",
    source="shop",
    event_id="evt-cancel",
    order_id="ord-42",
    revision=3,
    intent="CANCELLED",
)

late_confirm = dict(
    delivery_id="del-2",
    source="shop",
    event_id="evt-confirm",
    order_id="ord-42",
    revision=2,
    intent="CONFIRMED",
)


print(
    "INGEST",
    ingest(create),
)

create_key = (
    "notify_created:ord-42:r1"
)

fault_plan[create_key] = [
    "timeout_after_commit",
    "success",
]

# Remote effect happens, response is lost.
run_worker(0)

# Worker reconciles existing remote effect.
# It does not produce a second effect.
run_worker(2)


# Exact same transport delivery.
print(
    "DUPLICATE DELIVERY",
    ingest(create),
)

# Same source event under another delivery.
print(
    "DUPLICATE EVENT",
    ingest(create_retry),
)


# Revision 3 deliberately arrives before revision 2.
print(
    "CANCEL",
    ingest(cancel),
)

print(
    "LATE CONFIRM",
    ingest(late_confirm),
)


cancel_key = (
    "notify_cancel:ord-42:r3"
)

fault_plan[cancel_key] = [
    "rate_limit",
    "success",
]

# 429: work remains durable.
run_worker(10)

# Later retry succeeds.
run_worker(20)


# Explicit poison/dead-letter example.
with db:
    db.execute(
        """
        INSERT INTO outbox(
            op_key,
            order_id,
            revision,
            action,
            deadline_at
        )
        VALUES(
            'manual_export:ord-99:r1',
            'ord-99',
            1,
            'manual_export',
            100
        )
        """
    )

fault_plan[
    "manual_export:ord-99:r1"
] = [
    "permanent",
]

run_worker(30)


# A -> B -> A.
# The second visit to A under the same causation is blocked.
m = feedback_step(
    "A",
    "cause-7",
    "source",
    0,
)

m = feedback_step(
    "B",
    m["causation_id"],
    m["origin"],
    m["hop"],
)

m = feedback_step(
    "A",
    m["causation_id"],
    m["origin"],
    m["hop"],
)


# -----------------------------
# Observable invariants
# -----------------------------

order = dict(
    db.execute(
        """
        SELECT *
        FROM orders
        WHERE order_id='ord-42'
        """
    ).fetchone()
)

effects = [
    dict(x)
    for x in db.execute(
        """
        SELECT *
        FROM remote_effects
        ORDER BY op_key
        """
    )
]

dead = [
    dict(x)
    for x in db.execute(
        """
        SELECT *
        FROM dead_letters
        ORDER BY op_key
        """
    )
]

outbox = [
    dict(x)
    for x in db.execute(
        """
        SELECT
            op_key,
            status,
            attempts,
            external_ref,
            last_error
        FROM outbox
        ORDER BY op_key
        """
    )
]

loop_blocks = db.execute(
    """
    SELECT count(*)
    FROM audit
    WHERE kind LIKE 'loop_blocked%'
    """
).fetchone()[0]


assert (
    order["revision"] == 3
    and order["state"] == "CANCELLED"
)

assert sum(
    x["op_key"] == create_key
    for x in effects
) == 1

assert sum(
    x["op_key"] == cancel_key
    for x in effects
) == 1

assert db.execute(
    """
    SELECT status
    FROM outbox
    WHERE op_key=?
    """,
    (create_key,),
).fetchone()[0] == "done"

assert db.execute(
    """
    SELECT status
    FROM outbox
    WHERE op_key=?
    """,
    (cancel_key,),
).fetchone()[0] == "done"

assert len(dead) == 1
assert loop_blocks == 1


print(
    "FINAL ORDER",
    order,
)

print(
    "REMOTE EFFECTS",
    json.dumps(
        effects,
        sort_keys=True,
    ),
)

print(
    "OUTBOX",
    json.dumps(
        outbox,
        sort_keys=True,
    ),
)

print(
    "DEAD LETTERS",
    json.dumps(
        dead,
        sort_keys=True,
    ),
)

print(
    "LOOP BLOCKS",
    loop_blocks,
)

print(
    "INVARIANTS PASS"
)

A limitation of this harness is deliberate: its simulated remote_effects table makes an external outcome queryable by operation key. Real providers differ. Where a real external system offers neither robust idempotency nor an authoritative way to query whether an ambiguous operation happened, full automatic recovery may be impossible without accepting duplicate risk. In that condition, the recommendation reverses toward serialization, human approval, or a provider/interface change.

Claim-to-primary-source dossier

All sources below are primary provider/standards/project material. “Documented design” means the source documents intended product behavior; it does not mean this research independently load-tested the behavior.

Claim / use	Primary source and exact scope	Source date	Inspected	Evidence type	Assessment / editorial advice
Stripe webhooks are retried, can be manually resent, and manual resend does not stop automatic retry	Stripe, Webhooks → Event delivery behaviors → Automatic retries / Manual retries. https://docs.stripe.com/webhooks 4	Live doc; no publication date shown	2026-10-10	Provider documentation	Supported. Keep, but preserve live-vs-sandbox and resend-window scope.
Stripe webhook delivery order is not guaranteed; event timestamp should not be used as ordering proof	Stripe, Webhooks → Event ordering. https://docs.stripe.com/webhooks 4	Live doc; no publication date shown	2026-10-10	Provider documentation	Supported. Keep. Strong support for source-version/reconciliation design.
Webhook handler should return 2xx before complex downstream logic	Stripe, Create a handler. https://docs.stripe.com/webhooks 4	Live doc; no publication date shown	2026-10-10	Provider documentation	Supported, qualify. Does not itself say to acknowledge before durable persistence.
Stripe API idempotency semantics and retention differ between v1 and v2	Stripe, API v2 overview → Idempotency / Idempotency differences between API v1 and API v2. https://docs.stripe.com/api-v2-overview 7	Live doc; examples show API versions current to doc	2026-10-10	Provider contract documentation	Supported. Keep exact API-version scope. Do not turn 24 h/30 d into a universal Stripe rule.
Exactly-once Pub/Sub semantics suppress acknowledged-message redelivery under defined conditions	Google Cloud, Pub/Sub → Exactly-once delivery. https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery 5	Last updated 2026-10-09 UTC 5	2026-10-10	Provider documentation	Supported but highly scoped. Qualify.
Pub/Sub exactly-once guarantee is regional for subscribers	Google Cloud, Exactly-once delivery → Regional considerations. same URL 5	2026-10-09	2026-10-10	Provider documentation	Supported. Keep. Region is a service topology condition, not CZ-vs-US customer nationality.
Pub/Sub exactly-once applies only to pull subscriptions; publish-side duplicates can still produce multiple copies; latency is higher	Google Cloud, Exactly-once delivery → Exactly-once delivery and push subscriptions / Things to know. same URL 5	2026-10-09	2026-10-10	Provider documentation	Supported. Keep. Strong counterevidence to end-to-end “exactly once” shorthand.
SQS documentation distinguishes Standard at-least-once from FIFO “exactly-once message processing”	AWS, What is Amazon SQS? → Benefits → Durability. https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html 14	Live doc; no publication date shown	2026-10-10	Provider documentation	Supported as AWS terminology. Qualify heavily before relating it to external side effects.
Retrying transient faults assumes repeated operations are safe from additional side effects	AWS Builders Library, Making retries safe with idempotent APIs → Retrying and side effects. https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/ 3	Live article; publication date not exposed in inspected excerpt	2026-10-10	Attributable provider engineering practice	Supported. Keep. Not an independent experiment.
Atomic linkage between idempotency token and mutation prevents token/effect divergence; retention is necessarily finite/domain-specific	AWS Builders Library, Retries and semantic equivalence / Late arriving requests / Same client request ID, different intent. same URL 3	Live article	2026-10-10	Attributable provider engineering practice	Supported as reported Amazon design. Qualify when generalized.
Exponential backoff is appropriate for throttling/transient faults; retrying non-idempotent actions is risky	AWS Prescriptive Guidance, Retry with backoff → Intent / Applicability / Issues. https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/retry-backoff.html 8	Live guidance	2026-10-10	Provider architecture guidance	Supported. Keep.
AWS uses timeouts, retries and backoff/jitter to tolerate transient failures without magnifying overload	AWS Builder Center, Timeouts, retries, and backoff with jitter. https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter 15	Published 2026-06-12; modified 2026-06-15	2026-10-10	Attributable provider engineering practice	Supported. Keep, but do not claim measured universal impact.
Transactional outbox addresses database+message dual-write inconsistency; duplicates remain possible	AWS Prescriptive Guidance, Transactional outbox → Intent / Motivation / Issues and considerations. https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html 9	Live guidance	2026-10-10	Provider architecture guidance	Supported. Keep. Particularly relevant to warehouse/downstream handoff.
Compensation is distinct from forward recovery/retry	AWS Prescriptive Guidance, Saga patterns. https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/saga-patterns.html 11	Live guidance	2026-10-10	Provider architecture guidance	Supported. Keep. Business-specific implementation still required.
Outbox events can carry event ID separately from aggregate/business identity	Debezium, Outbox Event Router → Basic outbox table. https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html 10	Live “stable” documentation; inspected example identifies connector 3.7.0.Final	2026-10-10	Primary project documentation	Supported. Keep, without treating the example connector version as the whole doc’s version.
CloudEvents distinguishes unique event identity (source + id) from event subject and permits extension attributes	CNCF CloudEvents spec, Required Attributes: id/source; subject; Extension Context Attributes. https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md 2	Live main; document labels itself 1.0.3-wip	2026-10-10	Primary specification repository	Supported. Qualify: causation_id/hop_count in this dossier are custom editorial fields.
SQLite UPSERT can update or no-op on uniqueness conflict	SQLite, UPSERT → Description. https://www.sqlite.org/lang_upsert.html 12	Live documentation	2026-10-10	Primary software documentation	Supported. Keep only as harness implementation evidence, not a distributed-systems guarantee.
Retry-enabled event functions can loop on permanent errors; defined end conditions are recommended	Firebase, Retry asynchronous functions → Semantics / Best practices. https://firebase.google.com/docs/functions/retries 1	Live documentation	2026-10-10	Provider documentation	Supported. Keep. Product-specific retry windows should not be generalized.
A platform can show successful/200 execution logging despite an underlying function error under documented conditions	Firebase, Retry asynchronous functions, discussion immediately before retry configuration. same URL 1	Live documentation	2026-10-10	Provider documentation	Supported, highly product-specific. Keep as counterexample, not universal workflow behavior.
Publisher pressure can accumulate requests in memory until deadline failure; flow control can bound outstanding work	Google Cloud Pub/Sub, Flow control → About flow control. https://docs.cloud.google.com/pubsub/docs/flow-control-messages 13	Live documentation	2026-10-10	Provider documentation	Supported. Keep. Strong evidence for explicit backpressure rather than unbounded memory queues.
Idempotency-Key draft models a unique client key, non-reuse across different payloads, and explicit expiry policy	IETF HTTPAPI WG, draft-ietf-httpapi-idempotency-key-header-07 §§2.2–2.5. https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-idempotency-key-header 6	Published 2025-10-15; expired 2026-04-18 6	2026-10-10	Standards-track Internet-Draft, expired	Conditional / not normative current standard. Qualify prominently. Useful design vocabulary; do not call it an RFC.

Evidence status of the central draft claims

Candidate claim	Status	Advice
“Two automations trigger each other. Stop the infinite circle with retries.”	Contradicted as general advice. Retries can perpetuate permanent failures; a termination condition plus ownership/causation guard is needed. 1	Rewrite around causation, meaningful change, one-writer ownership and a fuse.
“The API is rate-limited, so failed requests can simply be retried.”	Conditional. Backoff is supported for transient throttling, but accepted work needs durability and deadlines; uncontrolled retry can worsen overload. 8	Qualify. Ask what may wait, what must survive, and when escalation beats another retry.
“Cancellation arrived first, so it wins.”	Unresolved without business revision semantics. Arrival order is not trustworthy. 4	Rewrite: newest authoritative source revision wins; retrieve current source state when revision is unavailable.
“We have exactly-once delivery, therefore customers cannot be charged/notified twice.”	Contradicted as an end-to-end inference. Provider guarantee boundaries are narrower, and publish-side duplicates may remain. 5	Remove or rewrite as a product-scoped transport claim.
“The workflow is green, so recovery worked.”	Contradicted as sufficient evidence. Provider docs contain concrete cases where acknowledgement/execution success is decoupled from downstream completion or even internal error reporting. 4	Replace with business-state and side-effect invariants.
“Store event IDs and duplicates are solved.”	Contradicted as complete advice. Separate Event objects can represent the same object/event semantics; event identity differs from business-operation identity. 4	Rewrite to distinguish event dedupe from business dedupe.
“Idempotency keys make retries safe forever.”	Contradicted. Provider key validity can be finite; Stripe documents different v1/v2 windows. 7	Remove. Retention/replay horizon is part of the design.

Contradictions, limits and reusable article directions

Where the recommendation should reverse

The deterministic recovery architecture is not automatically better than a manual process.

Very low volume + high consequence. If a team performs a handful of unusual refunds, contract terminations, physical shipments or irreversible communications, human verification before the side effect may be cheaper and safer than building sophisticated autonomous recovery. Provider idempotency can still be used underneath the manual action where available. AWS itself notes that the complexity of robust idempotent service contracts is not right for every solution. 3

No authoritative business revision. If the source gives events that cannot be ordered and there is no reliable current-state lookup, automatic conflict resolution is fundamentally under-informed. The system should reduce autonomy rather than invent an ordering rule. Stripe’s warning about timestamp ordering illustrates the problem. 4

No idempotency and no external reconciliation interface. For a timeout after an irreversible side effect, the system may literally be unable to distinguish “did not happen” from “happened but response vanished.” In that case an automatic blind retry is not evidence-based recovery; human review, serialization or a better provider interface is warranted. AWS’s lost-response/idempotent-token discussion is the relevant documented design precedent. 3

Ordering costs more than it solves. Google documents throughput/latency trade-offs around ordered exactly-once Pub/Sub processing and notes higher latency for exactly-once subscriptions. Where the business object itself carries a source revision, accepting out-of-order transport and reconciling state can be simpler than enforcing total queue ordering. This latter conclusion is an architectural inference, not a Google recommendation. 5

AI sees too few exceptions to repay its upkeep. Under the illustrative cost model above, the AI layer adds 30 first-year maintenance/build hours before it saves anything. At low exception volume, deterministic runbooks and good audit queries are the simpler system. The numeric comparison is synthetic, not empirically measured.

Contradictions and unresolved gaps

The evidence does not support a universal “exactly once is impossible” slogan. Some providers make legitimate, useful exactly-once claims inside clearly defined boundaries. Google Pub/Sub defines exactly-once semantics under specific subscription/region requirements; AWS calls FIFO queue processing exactly-once. The correct criticism is narrower: those contracts must not be silently extended beyond their documented boundary to an end-to-end business outcome. 5

Stripe’s “return 2xx quickly” advice and a durable-before-acknowledgement design are not actually contradictory, but there is an engineering tension: durable acceptance has to be quick enough not to turn persistence latency into webhook timeouts. The inspected Stripe documentation requires a quick response before complex work but does not prescribe an inbox implementation. The durable-inbox recommendation is therefore an inference and should be presented as such. 4

Transactional outbox does not create magical exactly-once behavior. AWS explicitly notes that the event processor can still publish duplicates and recommends idempotent consumers. Outbox solves a particular atomicity gap; it does not remove every other one. 9

Dead-letter storage is not synonymous with recovery. A dead letter without an owner, alert, inspection path and safe replay mechanism is merely durable failure. AWS documents dead-letter constructs, but the ownership/runbook requirement in this dossier is editorial operating design. 14

Compensation is not rollback in the database sense. Sending a refund does not make an accidental charge historically nonexistent; a corrective email does not un-send the first one. Saga guidance supports the conceptual distinction between forward retry and compensating transactions, but the exact compensating business action must be domain-defined. 11

The current IETF Idempotency-Key material inspected for this research had expired as an Internet-Draft by the snapshot date and explicitly describes itself as work in progress. It is therefore supporting vocabulary rather than a current standards authority. 6

No production tests were performed against Stripe, Google Cloud, AWS, Debezium, or any commerce source. Provider documentation was inspected; the only firsthand execution was the synthetic SQLite harness.

No evidence was found in the inspected sources for a universal optimal deduplication-retention period. Retention must be derived from actual provider retry/resend windows, source replay behavior, business correction horizon and storage/governance constraints. Stripe’s differing replay windows demonstrate why one fixed folklore number would be unsafe. 4

Smallest evaluation before building

Before implementing the full workflow, run a fixture suite against the intended design with these acceptance tests:

Injection	Required observable result
Same delivery twice	Only one durable application
Same event under new delivery ID	Event deduped; both attempts auditable
Different event IDs, same business revision/intent	One business side effect
CANCEL r3 before CONFIRM r2	Final state remains CANCELLED r3
Crash after durable accept	Work resumes from inbox
Crash after local state update	Outbox intent still exists
Timeout after remote side effect	Reconciliation prevents blind duplication
429 burst	Work remains durable; retry pressure backs off
Permanent validation failure	Stops retrying and becomes owned dead letter
Source replay beyond provider idempotency TTL	Local business operation guard still determines safety
Automation A → B → A	Causation guard or meaningful-change rule terminates chain
New legitimate higher revision after cancellation	New revision is processed according to domain rules
Reconciliation mismatch	Incident is visible even if every workflow execution was green

These are editorial acceptance criteria, with the underlying failure modes grounded in provider documentation on retries, ordering, idempotency, dual writes, throttling and exactly-once scope. 4

Candidate article angles

Angle	Core thesis	Evidence strength
Green Is Not Done	Judge automation by source state + side-effect evidence + reconciliation, not the run icon	Strong: Stripe, Firebase, Pub/Sub 4
The Five IDs Behind Safe Automation	Delivery ID, event ID, business ID, source revision and operation ID solve different problems	Strong for event/business distinction; operation model is editorial synthesis 2
Exactly Once, Until You Leave the Box	Broker-level guarantees should not be presented as end-to-end business guarantees	Strong provider counterevidence 5
A Timeout Is an Unknown Result, Not a Failure	Reconcile before repeating payments/messages after ambiguous responses	Strong engineering-practice support 3
Cancellation Arrived First. Should You Believe It?	Arrival order is weaker than source revision/current-state reconciliation	Strong webhook evidence; source-revision design editorial 4
Your Retry Queue Is a Business Priority Queue	Rate limits require durability, deadlines and backpressure, not “retry=true”	Strong provider evidence 8
Two Automations Walk Into a Loop	Stop reciprocal triggers through ownership, causation and meaningful transitions	Retry-loop risk documented; specific causation design editorial 1
Idempotency Expires	Provider key retention and your business replay horizon are different design variables	Particularly concrete with Stripe v1/v2 + webhook replay windows 4
Dead Letter Is Not a Recovery Plan	Failed work is only recovered when an owner can inspect, decide, repair and reconcile	DLQ mechanism documented; operational conclusion editorial 14

Useful reusable assets

The complete Python/SQLite fixture above is the principal portable test asset. It was actually executed locally during this research; it uses no private data and no external system.

A second useful asset is a minimal event contract:

{
  "delivery_id": "del-17",
  "source": "shop.example",
  "event_id": "evt-91",
  "business_entity": {
    "type": "order",
    "id": "ORD-42"
  },
  "source_revision": 3,
  "intent": "CANCELLED",
  "operation_key": "notify_cancel:ORD-42:r3",
  "origin": "source",
  "causation_id": "cause-7",
  "hop_count": 0
}

source/event id map naturally to CloudEvents concepts; business_entity, source_revision, operation_key, causation_id and hop_count are application-level design fields in this dossier. CloudEvents explicitly allows extension attributes but assigns no standard meaning to these custom names. 2

A minimal recovery record should make this question answerable without reading application logs:

{
  "operation_key": "notify_cancel:ORD-42:r3",
  "business_state": "CANCELLED",
  "source_revision": 3,
  "accepted": true,
  "attempts": 2,
  "external_result": "verified",
  "external_reference": "provider-object-123",
  "terminal_disposition": "done",
  "compensation_required": false,
  "reconciled_at": "2026-10-10T12:00:00Z"
}

That schema is an editorial asset, not a provider standard.

Freshness triggers

Re-check this dossier whenever Stripe changes webhook retry/manual-resend windows, webhook ordering guidance, or v1/v2 idempotency behavior; those are live provider contracts and the current API v1/v2 retention differences are operationally material. 4

Re-check Pub/Sub if exactly-once changes its pull-only constraint, regional condition, publish-side duplicate behavior, or latency trade-off. The inspected page was updated as recently as October 9, 2026, one day before this snapshot. 5

Re-check the real order source whenever it changes event IDs, webhook replay rules, source revision/version semantics or current-object retrieval behavior. The synthetic design depends critically on the source revision being authoritative.

Re-run the duplicate/late/timeout/quota/loop fixture suite after changes to queue provider, database transaction boundaries, workflow engine, API version, downstream provider, retry policy or operation-key retention.

Re-evaluate the AI triage layer after changes to model/tool permissions. The safety boundary assumed here is that AI does not authorize payment, message sending or production writes.

Retire or simplify the custom automation when a native provider integration can satisfy the same observable outcome and recovery evidence at lower lifecycle burden, when event volume no longer justifies upkeep, or when the team can no longer name an owner for reconciliation and recovery. The central design test remains the same: not whether the machinery is green, but whether the business state can be proven correct after the machinery fails.

Portable citation index

1. [Retry asynchronous functions  |  Cloud Functions for Firebase](https://firebase.google.com/docs/functions/retries) 2. [spec/cloudevents/spec.md at main · cloudevents/spec · GitHub](https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md) 3. [references-details-empty](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) 4. [docs.stripe.com](https://docs.stripe.com/webhooks) 7. [docs.stripe.com](https://docs.stripe.com/api-v2-overview) 5. [Exactly-once delivery  |  Pub/Sub  |  Google Cloud Documentation](https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery) 6. [draft-ietf-httpapi-idempotency-key-header-07](https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-idempotency-key-header) 8. [Retry with backoff pattern - AWS Prescriptive Guidance](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/retry-backoff.html) 9. [Transactional outbox pattern - AWS Prescriptive Guidance](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html) 11. [Saga patterns - AWS Prescriptive Guidance](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/saga-patterns.html) 14. [What is Amazon Simple Queue Service? - Amazon Simple Queue Service](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html) 10. [Outbox Event Router :: Debezium Documentation](https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html) 12. [UPSERT](https://www.sqlite.org/lang_upsert.html) 13. [Flow control  |  Pub/Sub  |  Google Cloud Documentation](https://cloud.google.com/pubsub/docs/flow-control-messages) 15. [Timeouts, retries, and backoff with jitter | AWS Builder Center](https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter)

Import verification and asset availability

Genuine Deep Research completed in 9 minutes and was captured on 2026-10-10 after the first due time. Original export, exact independent prompt, canonical chat and citation URLs are retained. Complete inline examples/code were preserved; no missing sandbox download target is claimed as a delivered asset. Report code, test outcomes and execution assertions were not independently run or verified during this import. Delivery is not independent claim verification; Evidence status remains Captured.

Retries, Duplicates and Idempotency: Evidence Dossier for Outcome-First Recovery Design

Research snapshot: 10 October 2026
Perspective: Decision-maker — data/operations lead choosing control depth, ownership and staged rollout
Scope: Small team operating an event/webhook-driven workflow into a cloud analytical warehouse and downstream business automation
Evidence boundary: Public primary sources and synthetic fixtures only. No private systems, customer results or invented operational experience are used.

Executive answer and uncertainties

Decision

Recovery should be designed and reported in terms of verified business outcomes, not “green” workflow runs.

A successful webhook invocation, queue acknowledgment, workflow completion or warehouse write is useful operational evidence, but none of those facts alone establishes that the intended business action happened correctly. The classic end-to-end argument is unusually direct on this point: lower-level delivery acknowledgments cannot necessarily establish whether the target application actually performed the requested action; the required acknowledgement may have to come from the application that knows the intended result. HTTP makes a related distinction: 202 Accepted explicitly means a request was accepted for processing, not that processing completed. 1

That distinction matters because current provider guarantees deliberately stop at different boundaries. Stripe documents duplicate and unordered webhook delivery; Google Pub/Sub’s “exactly-once” guarantee has subscription, client-mode and regional boundaries; BigQuery’s exactly-once semantics concern writes to an application-created Storage Write API stream; and Amazon SQS FIFO deduplication applies to queue submissions within its deduplication interval. None of these claims, individually, proves that an order was canceled, a customer was charged only once, a message was sent exactly once, or downstream systems agree on the final business state. 2

For the scoped order workflow, I would therefore use this hierarchy:

Evidence level	What it proves	What it does not prove
Transport accepted	The webhook/queue/API accepted something	The business operation happened
Durable work accepted	The event is committed to storage/queue from which it can be recovered	The event is current or should cause an action
Business state applied	The authoritative revision was reflected in the local order state	An external side effect succeeded
External effect confirmed	The provider reports the intended payment/message/etc. operation	All participating systems agree
Reconciled business outcome	Source state, warehouse state and required external effects satisfy the business invariant	Nothing stronger is implied than the invariant actually checked

This hierarchy is an editorial design model, not a vendor standard. Its rationale follows the end-to-end principle and the documented limits of provider acknowledgments. 1

The recommended control pattern is:

Accept durably → classify identity and version → apply only meaningful current transitions → execute side effects with operation-level idempotency where available → verify the effect → reconcile end-to-end → quarantine ambiguity for a named human owner.

A “green run” remains useful for debugging and availability. It should not be the terminal success metric.

What is strongly supported

Several parts of this recommendation rest on unusually consistent primary evidence.

Providers explicitly document retries, duplicates and ambiguous failure. Stripe says webhook events can arrive more than once and not in generated order; its live-mode webhook delivery can retry for up to three days. AWS’s design guidance points out that after a timeout the caller can be unable to tell whether a resource was created, making blind retries potentially duplicative. 2

Idempotency also has boundaries. AWS recommends caller-provided request identifiers because identical parameters do not necessarily mean identical intent, and says recording the idempotency token plus associated mutations must form an atomic operation. Stripe API v1 retains the result of an idempotent operation once endpoint execution begins and documents that keys can be pruned after at least 24 hours; reusing a pruned key can start a new operation. 3

Provider “exactly-once” capabilities are useful but scoped. Pub/Sub limits exactly-once delivery to pull subscriptions and says the guarantee is regional; it also warns that publish-side duplicate publications can still yield multiple copies with different message IDs. BigQuery explicitly advises teams to consider whether exactly-once stream-offset management is necessary when upstream is at-least-once or post-ingestion duplicates are easy to detect. 4

What remains uncertain

The evidence base is much stronger on mechanisms and semantics than on business effect sizes. I did not find a primary empirical study establishing a general ROI, duplicate-error reduction percentage, recovery-time improvement or staffing threshold for small teams adopting idempotent event-processing architecture. The primary material is dominated by standards, provider documentation, provider-reported engineering practice and foundational systems-design research.

There is also no universal answer to how much control is worthwhile. AWS explicitly acknowledges that the stronger idempotency contract it describes adds cost and complexity and is not appropriate for every solution. BigQuery similarly documents a simpler default stream when exact offsets are unnecessary. 3

That makes the decision conditional:

Situation	Default choice
Low volume, visible exceptions, cheap manual correction	Manual/native baseline
Repeated process, clear state model, important or expensive side effects	Deterministic automation with durable ledger and reconciliation
High exception-review burden after deterministic controls are working	Add bounded AI triage
No reliable business identity or authoritative source version	Do not increase autonomy yet
Irreversible/high-consequence effect with weak provider idempotency or observability	Human approval or stronger deterministic control
“Exactly-once” feature adds complexity but duplicate repair is cheap	Prefer simpler at-least-once + dedup/reconciliation

For Czech versus US operation, the retry/idempotency mechanics in scope are properties of the protocols and providers, not country-specific architecture. This dossier therefore finds no reason to design different technical retry logic merely because a team is in Czechia versus the US. Privacy, retention, financial and consumer obligations can differ and may change how long audit data is retained or who may execute recovery, but those legal questions are outside this technical evidence scope.

Bounded process, ownership and baseline

The recognizable process

The bounded synthetic process is an e-commerce order feed whose source system emits order creation, update and cancellation events. Those events feed a cloud analytical warehouse; selected meaningful transitions can also trigger downstream business automation.

The process intentionally contains the failure patterns in the research scope.

Element	Bounded definition
Trigger	HTTP webhook/event delivery from the order source
Business entity	order_id, for example O-1042
Transport/message identity	source + event_id
Source version	Monotonic revision, for example 1, 2, 3
Operation intent	Separate value such as set_order_state, send_cancel_notice, create_refund
Inputs	Event ID, order ID, revision, event type/state, source timestamp, payload, origin/causation metadata
Business owner	Order Operations/Product owner — defines what “correctly completed” means
Technical owner	Data/Automation owner — owns ingestion, recovery controls, replay and observability
Recovery owner	Named on-call/operator who can quarantine, reconcile, retry or escalate
Warehouse outcome	Latest materialized order state equals the authoritative source revision
Downstream outcome	Every required external action is confirmed once at the business-operation level, or is explicitly compensated/waived
Manual fallback	Freeze outbound action, inspect source + ledger + provider reference, repair/replay under an audit record

CloudEvents is useful for illustrating why these identities should remain separate. Its specification defines event identity using source plus id; an occurrence can result in more than one event, while subject may identify the business object the event concerns. Event identity is therefore not synonymous with order identity. 5

Stripe’s duplicate guidance exposes the same distinction in practical provider terms: logging an event ID handles repeated delivery of one Event object, but Stripe notes that separate Event objects can in some cases represent duplicate underlying activity and recommends looking at the object ID together with event type. 2

Proposed state model

The following state machine is an editorial design, not a provider-defined protocol:

WEBHOOK_RECEIVED
      |
      v
DURABLY_ACCEPTED
      |
      v
VALIDATED
      |
      +---- invalid / contradictory ----> QUARANTINED
      |
      v
BUSINESS_STATE_APPLIED
      |
      v
WAREHOUSE_COMMITTED
      |
      +---- no downstream action required ---> RECONCILE
      |
      v
OUTBOUND_OPERATION_PENDING
      |
      +---- known failure -------------> RETRYABLE / QUARANTINED
      |
      +---- unknown outcome -----------> VERIFY_EXTERNAL_EFFECT
      |
      v
EXTERNAL_EFFECT_CONFIRMED
      |
      v
RECONCILED
      |
      v
VERIFIED_COMPLETE

DURABLY_ACCEPTED deserves to be a separate state from WEBHOOK_RECEIVED. Stripe recommends returning a successful HTTP response quickly, before complex downstream logic, so webhook endpoints do not unnecessarily hold delivery open. But a transport response is not itself durable storage. The safe editorial interpretation is: perform the minimum durable inbox/queue commit first, then return 2xx, and process complex work asynchronously. Otherwise, an application can acknowledge receipt and crash before preserving the work, turning a superficially successful webhook into a lost business operation. Stripe’s guidance supports asynchronous processing but does not prescribe this particular internal transaction boundary. 2

Observable completion

For synthetic order O-1042, suppose the source’s authoritative latest revision is:

{
  "order_id": "O-1042",
  "revision": 3,
  "state": "CANCELED"
}

The business outcome is not “workflow run 91842 is green.”

A suitable outcome invariant might instead be:

source(O-1042).revision = 3
AND source(O-1042).state = CANCELED
AND warehouse(O-1042).revision = 3
AND warehouse(O-1042).state = CANCELED
AND no downstream operation exists whose intent conflicts with cancellation
AND every required cancellation-related operation is either:
    CONFIRMED,
    COMPENSATED,
    or HUMAN_WAIVED with actor + reason

This is synthetic, but the architectural principle is supported by the end-to-end paper’s distinction between intermediate delivery acknowledgement and confirmation that the target application actually performed the requested operation. 1

The useful top-level metrics then become things such as verified_complete / accepted_business_operations, age of the oldest unreconciled accepted item, number of unresolved external-outcome ambiguities, number of stale-event suppressions, number of pending compensations and time from detection to verified repair. Thresholds for those measures are business choices, not universal reliability constants.

Simplest manual/native baseline

The baseline should be deliberately boring:

The source system remains the order system of record. A native export, scheduled warehouse load or source connector populates analytical data. An operator uses the source administration UI and provider dashboards/logs to inspect cancellations or discrepancies. Any customer message, payment repair or correction requiring judgment is performed manually. A recurring source-versus-warehouse report exposes missing or inconsistent orders.

This approach has weak latency and can consume human time, but it avoids a custom retry engine, deduplication store, operation ledger, loop controller and AI layer. For low volume or rapidly changing business rules, that can be the rational design. AWS explicitly cautions that stronger idempotency contracts add implementation complexity and are not right for all solutions, while BigQuery advises that exactly-once offsets need not be used where simpler duplicate detection is adequate. 3

The key point is that the manual baseline must still define the same outcome. Comparing a manual process’s verified business completion to an automated workflow’s “green runs” would be an apples-to-oranges evaluation.

Evidence synthesis around recovery design

Transport delivery and business completion are different claims. RFC 9110 defines 202 Accepted as acceptance for processing rather than completed processing. Saltzer, Reed and Clark’s original end-to-end design paper goes further: for a requested application action, the meaningful acknowledgment may be one generated at the application endpoint indicating whether it actually did the work. Their paper explicitly covers duplicate message detection, sequencing, guaranteed delivery and recovery, while warning that lower-layer mechanisms can be useful performance enhancements without replacing end-to-end correctness checks. 6

This is the strongest conceptual evidence behind the topic decision. “Green run” is an intermediate acknowledgment unless its success condition itself includes verification of the business invariant.

Retries are safe only relative to an operation’s semantics. AWS describes the classic timeout ambiguity: a resource-creation request may have succeeded even when the caller received no response, so blindly creating again can produce two resources. Its recommended API pattern is a caller-generated request identifier that states intent. AWS specifically rejects the idea that equal request parameters always imply duplicate intent, because two identical EC2 creation requests might legitimately mean “create two instances.” 3

That is why deduplication keys should not be casually derived from an entire payload hash. A payload hash can detect byte-level repetition, but business intent should be represented explicitly. In this dossier the useful distinction is:

message identity      = source + event_id
business identity     = order_id
source version        = revision
operation intent      = send_cancel_notice
business operation ID = order_id + revision + operation_intent
provider retry key    = idempotency key for that business operation

The first protects against transport redelivery. The fifth protects against repeating the same business side effect. They solve different problems. AWS’s warning about identical parameters with different intent directly supports that separation. 3

Idempotency keys are not eternal memory. Stripe API v1 documents that an idempotency result is stored once endpoint execution begins, including a resulting 500, and that a key can be removed after it is at least 24 hours old; a reused key after pruning can initiate a new request. Stripe also compares parameters and rejects a reused key whose parameters differ. AWS says retention requirements vary by service and describes an EC2 strategy based on resource lifetime plus an interval covering plausible late requests. 7

The practical implication is important: the application’s own business-operation ledger should live for the full replay and reconciliation horizon, which may be substantially longer than a provider’s idempotency-key cache. That is an editorial inference from the documented finite provider windows.

The idempotency boundary must be atomic where possible. AWS explicitly says the operation combining the recording of the idempotency token and its associated server-side mutations needs ACID properties; otherwise one can record the token without performing the work, or perform the work without recording the token. 3

For a local database transaction, that can often be done directly. For a transaction spanning the local database and an external message/payment API, a single ACID transaction is normally not available. The practical substitute is a durable operation ledger/outbox, stable operation identity, provider-supported idempotency where available, external-effect verification and compensation/reconciliation for ambiguity. This is an architectural interpretation, not an AWS guarantee.

“Exactly-once” must always be followed by “exactly once where?” SQS standard queues document at-least-once delivery and tell consumers to be idempotent. SQS FIFO suppresses duplicate sends inside its documented five-minute deduplication interval. Google Pub/Sub’s exactly-once delivery concerns deliveries on supported pull subscriptions, with a same-region requirement; the documentation explicitly notes publish-side duplicates and additional latency/quota considerations. BigQuery can provide exactly-once write semantics using offsets on application-created streams, but its documentation recommends considering the simpler default stream when duplicates can be detected elsewhere. 8

So an architecture can simultaneously have:

exactly-once Pub/Sub delivery
+ exactly-once BigQuery stream writes
+ a duplicate customer email

without either provider violating its documented contract. The email call lies beyond both guarantees. That is precisely why end-to-end business verification is required. 4

Out-of-order events require business versions or reconciliation, not arrival timestamps. Stripe explicitly says webhook event ordering is not guaranteed. When required related objects have not yet arrived, Stripe advises retrieving them rather than assuming webhook arrival order expresses business order. 2

Accordingly, “cancellation arrived before creation” should not be solved with “whichever webhook arrived last wins.” If the source provides an authoritative monotonic revision, the higher revision should win for that entity. If it does not, an irreversible action should be based on a fresh source-of-truth read or held for reconciliation. That latter rule is an editorial safety design, conditional on the source API’s semantics.

Rate limiting is both a retry problem and a business-priority problem. RFC 6585 defines 429 Too Many Requests and allows the server to include Retry-After; the RFC intentionally does not define how a server counts requests. RFC 9110 defines the Retry-After field as a time or delay after which a subsequent request should be attempted. 9

Backoff therefore answers “when should I try again?” It does not answer “which accepted business operation is allowed to wait?” Google Pub/Sub’s flow-control documentation gives the lower-level mechanism: bound outstanding messages/bytes so publishing does not outrun available throughput. It explicitly distinguishes transient spikes from persistent capacity problems. 10

The business layer must add deadlines and priority. A warehouse enrichment job might tolerate an hour. A cancellation intended to prevent fulfillment might not. Those deadlines are process requirements, not properties of HTTP 429.

Retries can worsen dependency overload. AWS’s Builders’ Library warns that retries consume additional downstream capacity and can amplify overload; it recommends bounded retries, retrying at an appropriate layer, and jitter to spread synchronized retries. AWS describes token-bucket retry throttling in its SDKs and notes that side-effecting operations need idempotency if they are to be safely retried. This is provider-reported engineering practice, not a universal experimental law. 11

A dead-letter queue is a recovery inbox, not a business-completion state. Google Pub/Sub retries unacknowledged work and can forward messages to a dead-letter topic after an approximately configured number of attempts. Google explicitly says that maximum attempt count is best effort and may be exceeded or undershot; the dead-letter subscription exists for analysis and debugging. 12

A dashboard showing “main queue empty” while the dead-letter queue contains a canceled order is therefore not green in business terms.

Compensation is not magical rollback. Microsoft’s current compensating-transaction guidance says a failed multi-step eventually consistent operation may require recorded undo actions, but compensation is application-specific, can itself fail and sometimes requires manual intervention. It also cautions that compensation need not simply reverse operations in exact reverse order. 13

That distinction is especially important for messages. A duplicate monetary reservation may have a refund or release operation. A duplicate customer email cannot literally be unsent; “compensation” might mean a corrective message or human contact. Recovery design must model the actual business effect.

Feedback-loop prevention is a separate control from duplicate delivery. GitHub provides a concrete provider-specific example: workflow events generated using the repository’s GITHUB_TOKEN normally do not create new workflow runs, with documented exceptions, specifically to prevent recursive workflow runs. That proves that platform-level recursion suppression is a real design mechanism, but it does not establish a general solution for arbitrary business automations. 14

For arbitrary workflows, provenance such as origin, causation_id, root_correlation_id, semantic no-op detection and a finite hop budget are safer than assuming “event ID deduplication” will stop a loop: each iteration in a loop can be a perfectly new event.

Failure modes, worked timelines and safe repair

The following examples are synthetic and reasoned. They are not reports of customer incidents or provider tests.

Synthetic end-to-end order

Assume the authoritative source history is:

revision 1  CREATED
revision 2  UPDATED
revision 3  CANCELED

The observable desired final state is:

O-1042 = revision 3 / CANCELED
warehouse agrees
no fulfillment-start operation remains pending
required cancellation message is confirmed or explicitly waived
all accepted events are processed, quarantined or superseded

Duplicate delivery

10:00:00  evt-100 / O-1042 / rev=1 / CREATED arrives
10:00:01  inbox insert commits
10:00:02  state becomes CREATED r1
10:00:03  webhook delivery acknowledged

10:00:15  evt-100 / O-1042 / rev=1 / CREATED arrives again
10:00:15  UNIQUE(source,event_id) finds existing record
10:00:15  return success; no state transition; no side effect

Stripe explicitly says webhook endpoints can receive the same Event more than once and recommends remembering processed event IDs. Standard SQS similarly documents possible redelivery and recommends idempotent consumers. 2

Safe repair criterion: there is one accepted business transition and zero additional side effects. A duplicate-delivery counter increments; the order outcome is unchanged.

Different messages representing the same business operation

10:00  event A / O-1042 / rev=3 / CANCELED
10:01  event B / O-1042 / rev=3 / CANCELED

event A != event B, so transport-ID deduplication alone does not suppress B.

Business operation identity:

(order_id=O-1042, revision=3, intent=set_order_state)

does suppress the repeated semantic transition.

This distinction is consistent with Stripe’s warning that distinct Event objects can sometimes describe the same underlying object activity and AWS’s warning that request semantics—not superficial parameter identity—should determine idempotency. 2

Safe repair criterion: the second event remains auditable but produces no second business transition.

Cancellation arrives before creation

10:00:00  evt-cancel / O-1042 / revision=3 / CANCELED arrives
10:00:01  local latest := CANCELED r3

10:02:00  evt-create / O-1042 / revision=1 / CREATED arrives late
10:02:01  revision 1 < current revision 3
10:02:01  record event as STALE
10:02:01  do not regress order to CREATED

Stripe establishes the relevant premise: webhook delivery order is not guaranteed. 2

Safe repair criterion: the materialized business state remains revision 3. The late event is retained for audit/replay evidence but causes no outbound action.

If the source has no trustworthy monotonic revision, the recommendation weakens. Before an irreversible effect, query the source-of-truth object. If that lookup is unavailable or ambiguous, quarantine instead of guessing.

This directly answers the proposed article claim “The cancellation arrived first, then the confirmation. Which state should the automation use?”: use the authoritative business version/current source state, not arrival order. The advice is conditional on having trustworthy source-version semantics.

Acknowledged but lost

Unsafe ordering:

10:00:00.000  source POSTs event
10:00:00.020  receiver returns HTTP 200
10:00:00.025  process crashes
10:00:00.030  inbox write never happened

The provider may reasonably treat the delivery as complete because it received a successful response. The business work disappeared.

Safer ordering:

10:00:00.000  source POSTs event
10:00:00.020  INSERT inbox(...); COMMIT
10:00:00.030  receiver returns HTTP 200
10:00:01.000  asynchronous worker starts processing

Stripe recommends quick successful webhook responses before complex work; Saltzer et al. show why intermediate delivery acknowledgment is not equivalent to application completion. The “commit before 200” sequencing here is an editorial design derived from those constraints. 2

Safe repair criterion: every successful webhook acknowledgement corresponds to either a durable accepted-work record or a deliberately rejected event. There is no 2xx path that can silently discard accepted work.

Timeout after an external side effect

Consider a synthetic payment operation:

operation_id = O-1042:r2:create_payment
idempotency_key = O-1042:r2:create_payment
amount = EUR 100.00

Timeline:

10:00:00  local operation ledger -> PENDING
10:00:01  POST create-payment with idempotency key
10:00:02  provider performs side effect
10:00:03  response is lost / client times out
10:00:04  local system knows neither SUCCESS nor FAILURE

The wrong response is:

"Timeout -> send a new create-payment request with a new key."

That creates new intent.

Where provider semantics support it, the correct path is to retry/lookup using the original operation identity, recover the provider reference and confirm outcome. AWS’s retry-safety discussion centers on exactly this “request may have succeeded although the response was lost” ambiguity. Stripe supports idempotent POSTs, though its API v1 semantics include the important detail that once execution begins it returns the saved result for the same key, including saved 500 responses. 3

If the provider cannot prove whether the effect occurred, the state is not FAILED; it is UNKNOWN_OUTCOME.

Safe repair criterion:

provider confirms action exists -> mark CONFIRMED, do not resend
provider confirms action absent -> retry same business operation safely
provider remains ambiguous -> human/reconciliation path
duplicate effect discovered -> compensate according to business rules

For a non-idempotent message API, the operation ledger can prevent your own worker from intentionally sending twice, but a timeout after the provider accepted the message may still leave irreducible ambiguity. That should be exposed explicitly rather than hidden behind a green retry.

Rate-limited queue

Synthetic timeline:

12:00:00  cancellation job C accepted; business deadline 12:05
12:00:01  analytics enrichment A accepted; deadline 18:00
12:00:02  external API returns 429 Retry-After: 60
12:00:03  both jobs remain in durable queue

12:01:02  retry window opens + jitter
           C dispatched first because its deadline is nearer
           A remains queued

12:04:00  C still cannot complete
           alert/escalate before business deadline

12:15:00  A can still safely wait

RFC 6585 establishes the meaning of 429; RFC 9110 establishes Retry-After. Google Pub/Sub documents flow control for bounding outstanding requests/messages rather than allowing publication pressure to grow without bound. 9

Safe repair criterion: accepted work remains durable while throttled; retry traffic does not exceed the chosen backpressure policy; jobs whose business deadlines cannot be met are escalated rather than silently aging.

This answers the proposed article claim “The system limits the number of requests. What can wait and what must not be lost?”: transport retry policy tells the system when it may try again; the business process must separately define durability, priority and deadline.

Useful measurements are:

oldest_accepted_work_age
accepted_work_past_deadline
queue_depth by priority
retry_attempt_rate
429 rate
time_to_verified_completion

A universal “maximum queue age = X minutes” would be invented. The limit has to come from the business consequence.

Two automations trigger each other

Synthetic bad loop:

A: order.updated  -> writes CRM.status
B: CRM.status     -> writes order.tag
A: order.updated  -> writes CRM.status
B: CRM.status     -> writes order.tag
...

Every event can have a unique message ID. Pure transport deduplication therefore does nothing.

Proposed loop metadata:

{
  "origin": "automation-B",
  "causation_id": "event-7821",
  "root_correlation_id": "root-551",
  "hop_count": 3
}

Proposed controls, in priority order:

1. Semantic no-op: do not write a value that already has the desired business meaning.
2. Origin filter: an automation ignores changes produced solely by itself where appropriate.
3. Causation/root tracking: identify that apparently new events belong to the same chain.
4. Hop/time budget: quarantine a chain that still oscillates rather than running indefinitely.
5. Human resolution: decide which system owns the contested field.

GitHub demonstrates one provider-specific implementation of recursion suppression by generally preventing events produced using GITHUB_TOKEN from triggering another workflow run. It should be treated as an example, not as a portable guarantee for unrelated automation platforms. 14

Safe repair criterion: terminating a loop must leave a known business state. “We stopped after ten hops” is protection against resource consumption; it is not proof the correct system won.

This answers “Two automations trigger each other. How do you stop the infinite loop?”: first remove semantic self-triggering, then carry provenance, and use a hop budget as a circuit breaker—not as the primary correctness rule.

Replay and race conditions

Suppose a historical ingestion replay sends revisions 1, 2 and 3 again.

A safe replay should normally be permitted to reconstruct analytical data without replaying historical customer-facing effects. A useful explicit attribute is:

processing_mode = LIVE | REPLAY | REPAIR

and downstream business-action rules can require LIVE or an individually approved REPAIR.

BigQuery documents offsets as one way to obtain exactly-once stream writes, but also explicitly recommends asking whether offsets are necessary when duplicates can be identified after ingestion. 15

A race between two workers processing revision 2 and revision 3 requires the business-state write to be guarded atomically, for example conceptually:

UPDATE order_state
SET revision = 3, state = 'CANCELED'
WHERE order_id = 'O-1042'
  AND revision < 3;

The exact SQL depends on storage technology. The principle is that an older writer must not be able to overwrite a newer accepted version.

Safe repair criterion: replaying any accepted historical event sequence leaves the same latest business state and creates no unintended new outbound operation.

Dependency unavailable and dead-letter handling

A dependency outage should move work through bounded retries and then into an explicitly owned recovery state rather than disappearing. Google’s Pub/Sub dead-letter documentation is particularly useful here because it warns that the configured delivery-attempt limit is approximate, and describes the dead-letter subscription as a place for analysis/debugging. 12

Proposed terminal categories:

VERIFIED_COMPLETE
SUPERSEDED
COMPENSATED
HUMAN_WAIVED
QUARANTINED_REQUIRES_OWNER

DEAD_LETTERED should not be a business terminal category; it describes where the technical message currently resides.

Automation variants, lifecycle economics and staged rollout

Same process, three operating models

Dimension	Manual/native	Deterministic automation	Bounded AI triage
Source of correctness	Human checks source/provider state	Explicit state machine + ledgers + reconciliation	Same deterministic foundation
Side effects	Human performs them	Deterministic executor performs allowed operations	AI cannot send messages, authorize payment or write production state
Duplicate protection	Human recognition/native provider controls	Message dedup + business-operation idempotency	Unchanged
Out-of-order handling	Human inspects current source state	Source revision/current-state reconciliation	AI may summarize why item is ambiguous
Recovery	Human from start	Automated for known-safe classes; human for ambiguity	AI proposes classification/repair plan; human decides
Main upkeep	Operator time/training	Code, schemas, provider contracts, runbooks, monitoring	All deterministic upkeep + model/evaluation/prompt/version upkeep
False-positive consequence	Human investigates unnecessarily	Depends on automated rule	Additional human review; no direct business write
Best fit	Low scale/high variability	Repeated, well-specified process	High enough exception-review burden to justify extra layer

The recommendation is not to start with an agent and make recovery deterministic later. Idempotency, durable work, source versioning and operation identity are correctness mechanisms that an AI classifier does not eliminate.

Lifecycle maintenance deserves first-class treatment

BigQuery’s documentation gives one concrete illustration of provider/API drift: schema updates can take time to propagate to Storage Write API connections, and configurations that allow unknown fields to be dropped can lead to unintended data loss. This is a provider-specific example of why “the workflow still runs” is not sufficient maintenance evidence. 15

The minimum recurring maintenance budget should therefore cover:

source event contract changes
schema evolution
idempotency-window changes
API-version/provider behavior changes
retry/quota changes
credential/auth changes
reconciliation drift
runbook testing
manual-override testing
dependency retirement/replacement
AI evaluation/version drift, if AI is present

The provider documents also change over time: for example, the Pub/Sub exactly-once and flow-control pages inspected here were last updated on 9 October 2026, one day before this research snapshot. 4

Synthetic economics

The following numbers are not observed customer economics. They are a transparent decision model showing how to evaluate whether additional automation is worthwhile.

Assumptions:

N = business orders per month
exception rate before automation = 3% of orders
manual review time = 4 minutes / exception
loaded human cost = €60 / hour

deterministic automation:
  initial build = 60 engineering hours
  evaluation horizon = 12 months
  amortized build = 5 hours/month
  recurring maintenance/recovery = 8 hours/month
  platform cost = €150/month
  deterministic rules eliminate 85% of manual exception reviews
  residual review rate = 15% × 3% = 0.45% of orders

AI triage incremental:
  model/service cost = €100/month
  evaluation/calibration/maintenance = 4 hours/month
  time saved on each residual human review = 2 minutes

Manual cost per exception:

4\text{ min} / 60 \times €60/\text{h} = €4

Manual monthly recovery cost:

C_{manual} = 0.03N \times €4 = €0.12N

Deterministic fixed monthly cost:

(5 + 8)\text{ h} \times €60/\text{h} + €150
= €930

Residual human cost:

0.0045N \times €4 = €0.018N

Therefore:

C_{deterministic} = €930 + €0.018N

Synthetic break-even against manual:

€0.12N = €930 + €0.018N

N = \frac{€930}{€0.102/\text{order}}
\approx 9{,}118\text{ orders/month}

This is not a recommendation that 9,118 is a universal threshold. Change any assumption—labor cost, build effort, exception frequency, consequence of one duplicate payment—and the result changes.

AI adds:

€100 + 4\text{ h}\times€60/\text{h} = €340/month

Saving two minutes per residual review is:

2/60\text{ h}\times€60/\text{h}=€2/\text{review}

There are 0.0045N residual reviews, so:

AI\ savings = €0.009N

AI triage breaks even against deterministic-only operation when:

€0.009N = €340

N \approx 37{,}778\text{ orders/month}

Equivalent formulation:

€340 / €2 = 170\text{ residual reviews/month}

Under these synthetic assumptions, adding AI at 10,000 orders/month would be economically worse even though AI might be technically impressive.

Synthetic monthly order volume	Manual	Deterministic	Deterministic + AI triage
3,000	€360	€984	€1,297
10,000	€1,200	€1,110	€1,360
30,000	€3,600	€1,470	€1,540
50,000	€6,000	€1,830	€1,720

At the illustrative 10,000 orders/month, recovery/control cost per verified order—assuming all 10,000 are ultimately reconciled—would be:

manual          €1,200 / 10,000 = €0.120
deterministic   €1,110 / 10,000 = €0.111
AI-assisted     €1,360 / 10,000 = €0.136

The model deliberately excludes the monetary consequence of an unrecoverable duplicate payment, missed cancellation, customer complaint or compliance failure because no defensible generic value was found. Where a single error is expensive, an expected-loss term should be added:

C_{total}
=
C_{build}
+
C_{maintenance}
+
C_{infra}
+
C_{human}
+
\sum_i P(failure_i)\times Impact_i

Those probabilities and impacts must come from the team’s own incidents, shadow testing or business estimates—not from this dossier.

Proposed staged rollout

I would stage this process in four control-depth steps.

Baseline instrumentation first. Run the manual/native process long enough to count accepted orders, exceptions, stale events, duplicates, reconciliation mismatches, manual recovery time and business deadlines. Do not infer automation ROI before there is a denominator.

Deterministic shadow mode second. Ingest events durably, compute the intended state and operation ledger, but do not execute customer-facing side effects. Compare the calculated state against the source/native outcome.

Controlled deterministic side effects third. Enable only operations with explicit business identity, understood retry semantics and a tested recovery path. High-consequence or ambiguous operations remain human-approved.

Bounded AI triage last. Give the AI read-only access to quarantined event history, source snapshots and runbook documentation. It may propose “stale,” “probable duplicate,” “source lookup required” or a human repair sequence. It may not authorize a payment, send a customer message or directly write production state.

Proposed go/no-go criteria are design choices rather than industry thresholds: every seeded duplicate/out-of-order test should converge to the intended business state; replay must not regenerate live side effects; timeout ambiguity must enter UNKNOWN_OUTCOME rather than automatically become FAILED; every dead-letter item must have a named owner; and reconciliation must detect deliberately seeded source-versus-warehouse mismatches.

A sensible stop/retirement criterion is equally important. Retire or simplify the automation when the source offers a reliable native integration that makes custom recovery redundant, when business rules change faster than safe automation can be maintained, when monthly maintenance is persistently greater than verified operational benefit, or when authoritative state/version information becomes unavailable.

Smallest workable implementation and portable test asset

Minimum architecture

The smallest deterministic design I would consider production-worthy for the scoped process is:

                     +--------------------+
Webhook ------------>| durable event inbox|----+
                     +--------------------+    |
                                                v
                                         +-------------+
                                         | state reducer|
                                         +-------------+
                                           |         |
                                      valid|         |ambiguous
                                           v         v
                                  +------------+   QUARANTINE
                                  | order state |       |
                                  +------------+        v
                                       |           HUMAN OWNER
                                       v
                                  +-----------+
                                  | warehouse |
                                  +-----------+
                                       |
                         meaningful live operation?
                                |              |
                               no             yes
                                |              v
                                |       +----------------+
                                |       |operation ledger|
                                |       +----------------+
                                |              |
                                |              v
                                |       external provider
                                |              |
                                +-------+------+
                                        v
                                  RECONCILIATION
                                        |
                                        v
                                VERIFIED_COMPLETE

Its minimum persistence model is three conceptual tables:

event_inbox
  source
  event_id
  order_id
  revision
  event_type
  payload
  received_at
  processing_status
  UNIQUE(source, event_id)

order_state
  order_id PRIMARY KEY
  source_revision
  business_state
  source_updated_at
  locally_updated_at

operation_ledger
  order_id
  source_revision
  operation_intent
  idempotency_key
  status
  provider_reference
  last_error
  attempts
  updated_at
  UNIQUE(order_id, source_revision, operation_intent)

The first uniqueness constraint addresses transport duplicates. The last addresses repeated execution of the same business intent. They deliberately do not share an identity.

For external providers, the ledger should persist longer than the provider’s relevant retry/idempotency horizon. Stripe API v1’s documented ability to prune keys after at least 24 hours is a concrete reason not to treat the provider’s cache as the application’s permanent business ledger. 7

The corresponding recovery ownership should be explicit:

State	Automatic action	Human responsibility
Duplicate transport event	Suppress	None unless abnormal rate
Stale lower revision	Retain, no-op	None normally
Same revision, contradictory state	Quarantine	Check authoritative source
External outcome known successful	Mark confirmed	None
External outcome known absent	Safe retry	None if retry policy allows
External outcome ambiguous	Stop blind retries	Investigate provider/source
DLQ/quarantine beyond deadline	Alert	Repair, compensate or waive
Feedback-loop budget exhausted	Disable chain/quarantine	Determine correct owner/state

The observability dashboard should put business state before orchestration state. A useful proposed top row is:

accepted business operations
verified complete
unreconciled
past business deadline
unknown external outcome
compensation pending
quarantined without owner
oldest unreconciled age

Then, underneath it, show technical diagnostics such as webhook success rate, workflow failure rate, retries, 429s, dedup hits and queue depth.

Portable reducer fixture

The following is the complete synthetic test asset used in this research. It tests transport duplicates, a separate event expressing the same business transition, late revisions and same-revision conflicting state.

Execution status: executed locally in this research session using Python. Assertions passed.
Not tested: live Stripe, AWS, Google Cloud or other provider APIs; actual network failures; database transaction isolation; concurrency between real worker processes.

from dataclasses import dataclass
from typing import Dict, Set, Tuple


@dataclass(frozen=True)
class Event:
    event_id: str
    order_id: str
    revision: int
    state: str
    intent: str


class Reducer:
    def __init__(self) -> None:
        self.current: Dict[str, Tuple[int, str]] = {}
        self.seen_events: Set[str] = set()
        self.seen_transitions: Set[Tuple[str, int, str, str]] = set()

    def apply(self, e: Event) -> str:
        # Transport identity: same delivery/event ID.
        if e.event_id in self.seen_events:
            return "transport_duplicate"
        self.seen_events.add(e.event_id)

        # Business identity: same entity revision + semantic operation.
        business_key = (e.order_id, e.revision, e.state, e.intent)
        if business_key in self.seen_transitions:
            return "business_duplicate"

        current = self.current.get(e.order_id)
        if current is not None:
            current_rev, current_state = current

            if e.revision < current_rev:
                return "stale"

            if e.revision == current_rev:
                if e.state == current_state:
                    self.seen_transitions.add(business_key)
                    return "business_duplicate"

                return "conflict_quarantine"

        # Synthetic assumption:
        # source revision is authoritative and monotonically increasing.
        self.current[e.order_id] = (e.revision, e.state)
        self.seen_transitions.add(business_key)
        return "applied"


fixtures = [
    Event(
        "evt-cancel-r3",
        "O-1042",
        3,
        "CANCELED",
        "set_order_state",
    ),
    Event(
        "evt-create-r1",
        "O-1042",
        1,
        "CREATED",
        "set_order_state",
    ),
    Event(
        "evt-cancel-r3",
        "O-1042",
        3,
        "CANCELED",
        "set_order_state",
    ),
    Event(
        "evt-cancel-r3-copy",
        "O-1042",
        3,
        "CANCELED",
        "set_order_state",
    ),
    Event(
        "evt-update-r2",
        "O-1042",
        2,
        "UPDATED",
        "set_order_state",
    ),
    Event(
        "evt-conflict-r3",
        "O-1042",
        3,
        "UPDATED",
        "set_order_state",
    ),
]

expected = [
    "applied",
    "stale",
    "transport_duplicate",
    "business_duplicate",
    "stale",
    "conflict_quarantine",
]

reducer = Reducer()
actual = [reducer.apply(e) for e in fixtures]

assert actual == expected, (actual, expected)
assert reducer.current["O-1042"] == (3, "CANCELED")

print(list(zip([e.event_id for e in fixtures], actual)))
print(reducer.current)

Executed result:

[
  ('evt-cancel-r3', 'applied'),
  ('evt-create-r1', 'stale'),
  ('evt-cancel-r3', 'transport_duplicate'),
  ('evt-cancel-r3-copy', 'business_duplicate'),
  ('evt-update-r2', 'stale'),
  ('evt-conflict-r3', 'conflict_quarantine')
]

{'O-1042': (3, 'CANCELED')}

The fixture proves only the behavior of this small in-memory reducer. It does not prove database atomicity, provider idempotency or end-to-end correctness. In particular, the assumption that source revision is authoritative is deliberately visible in the code; removing that assumption requires source lookup/reconciliation rather than blindly applying the reducer.

Claim-to-source dossier and evidence quality

The table distinguishes standards, provider documentation, provider-reported design practice, original research/design literature and editorial inference. “Supported” means the source directly supports the bounded claim, not that it establishes a universal business effect.

Claim or draft target	Primary source and exact scope	Source date / inspected	Evidence type	Assessment	Editorial action
A successful intermediate run/ack does not prove the application did the requested business work.	Saltzer, Reed & Clark, End-to-End Arguments in System Design, “Delivery guarantees” and conclusions. DOI 10.1145/357401.357402; https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf 1	Nov 1984 / 2026-10-10	Original systems-design paper	Supported	Keep. This is the core conceptual claim.
HTTP acceptance can be weaker than completion.	RFC 9110 §15.3.3, 202 Accepted; https://www.rfc-editor.org/rfc/rfc9110.html 6	Jun 2022 / 2026-10-10	Internet standard	Supported	Keep, but do not imply every workflow uses HTTP 202.
429 tells you rate limiting occurred; retry timing can be signaled with Retry-After, but business priority is separate.	RFC 6585 §4, 429 Too Many Requests; https://www.rfc-editor.org/rfc/rfc6585.html; RFC 9110 §10.2.3 Retry-After. 9	Apr 2012 / Jun 2022; inspected 2026-10-10	Internet standards	Supported / conditional	Keep transport claim; qualify prioritization as application design.
Webhook delivery may duplicate and arrive out of order.	Stripe Webhooks, “Event ordering,” “Handle duplicate events,” “Automatic retries”; https://docs.stripe.com/webhooks 2	Living documentation, publication date not shown / 2026-10-10	Provider documentation	Supported for Stripe	Keep as concrete example; do not universalize Stripe’s retry schedule.
Transport event ID and underlying business identity can differ.	Stripe “Handle duplicate events” says separate Event objects can represent duplicate underlying activity; CloudEvents id, source, subject; https://docs.stripe.com/webhooks; https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md 2	Stripe living; CloudEvents main showed 1.0.3-wip / 2026-10-10	Provider docs + open specification	Supported	Keep. Strong basis for separate message/business identity.
Hashing identical parameters is not a universally safe way to infer same intent.	AWS Builders’ Library, “Reducing client complexity with idempotent API design”; https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/ 3	Undated living page / 2026-10-10	Attributable provider engineering practice	Contradicts simplistic dedup	Rewrite advice around caller/business intent.
Recording an idempotency token and the associated mutation requires an atomic boundary.	Same AWS article, lines describing ACID/all-or-nothing idempotency session + mutation. https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/ 3	Undated living page / 2026-10-10	Provider design practice	Supported	Keep, noting external side effects require another pattern.
Idempotency keys have retention semantics; do not assume forever.	Stripe API Reference, “Idempotent requests,” API v1 behavior; https://docs.stripe.com/api/idempotent_requests; AWS “Late arriving requests.” 7	Living pages / 2026-10-10	Provider documentation/practice	Supported, provider-specific	Keep and qualify. Local ledger retention must be an explicit design choice.
Standard SQS can redeliver messages; consumers should be idempotent.	AWS SQS Developer Guide, “Amazon SQS standard queues and at-least-once delivery”; https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html 8	Living documentation / 2026-10-10	Provider documentation	Supported for SQS standard	Keep as concrete at-least-once example.
SQS FIFO “exactly-once processing” is not an end-to-end business exactly-once guarantee.	AWS FIFO deduplication docs describe five-minute send deduplication interval; https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html 16	Living documentation / 2026-10-10	Provider documentation	Supported as queue-level scope; broader claim contradicted	Qualify every “exactly once” statement by boundary.
Pub/Sub exactly-once has explicit boundaries and can still see publish-side duplicate copies.	Google Cloud Pub/Sub “Exactly-once delivery,” regional considerations, push-subscription limitation and “Things to know”; https://cloud.google.com/pubsub/docs/exactly-once-delivery 4	Last updated 2026-10-09 / 2026-10-10	Provider documentation	Supported	Keep. Strong counterexample to end-to-end interpretation.
Exactly-once storage writes may be unnecessary when duplicates can be detected elsewhere.	BigQuery Storage Write API best practices, “Manage stream offsets to achieve exactly-once semantics”; https://cloud.google.com/bigquery/docs/write-api-best-practices 15	Last updated 2026-09-30 / 2026-10-10	Provider documentation	Supported	Keep. Important counterweight to maximal control.
Backpressure should bound outstanding work when publishing outruns capacity.	Pub/Sub “Flow control,” “About flow control”; https://cloud.google.com/pubsub/docs/flow-control-messages 10	Last updated 2026-10-09 / 2026-10-10	Provider documentation	Supported for Pub/Sub client behavior	Keep, but business deadlines remain editorial.
A configured dead-letter attempt count should not be treated as mathematically exact.	Pub/Sub “Dead-letter topics,” “How maximum delivery attempts are calculated”; https://cloud.google.com/pubsub/docs/dead-letter-topics 12	Living documentation; inspected 2026-10-10	Provider documentation	Contradicts exact-count assumption	Rewrite DLQ logic around ownership/recovery, not exact attempt count.
Retries can increase pressure on a failing dependency; jitter/backoff reduce synchronization.	AWS Builders’ Library, Timeouts, retries, and backoff with jitter; https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/ 11	Undated living article / 2026-10-10	Provider-reported engineering practice	Supported as AWS practice	Keep, qualify against universal effect-size claims.
Compensation can fail and can require manual recovery.	Microsoft Azure Architecture Center, “Compensating Transaction pattern”; https://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction 13	2026-04-20 / 2026-10-10	Provider architecture guidance	Supported	Keep. Do not present compensation as perfect rollback.
“Dvě automatizace se spouštějí navzájem. Jak zastavit nekonečné kolečko?” / Two automations trigger each other.	GitHub Actions GITHUB_TOKEN documentation documents provider-specific recursion prevention; https://docs.github.com/en/actions/concepts/security/github_token 14	Living documentation / 2026-10-10	Provider documentation	Conditional	Rewrite: provenance + semantic no-op + bounded loop; GitHub only as concrete example.
“Systém omezuje počet požadavků. Co může počkat a co se nesmí ztratit?” / What can wait under a quota?	RFC 6585 429, RFC 9110 Retry-After, Pub/Sub flow control. 9	2012–2026 / 2026-10-10	Standards + provider docs	Supported premise; business answer conditional	Keep distinction between retry timing, durable acceptance and business deadline.
“Nejdřív přišlo zrušení, potom potvrzení. Který stav má automatizace použít?” / Cancellation arrived first; which state wins?	Stripe “Event ordering” demonstrates delivery order is not guaranteed. https://docs.stripe.com/webhooks 2	Living documentation / 2026-10-10	Provider documentation	Supported premise; source-version solution conditional	Keep after rewrite: authoritative revision/current source state wins, not arrival time.
Exactly-once is not a sufficiently precise universal description of all stream correctness properties.	Trofimov et al., Delivery, consistency, and determinism: rethinking guarantees in distributed stream processing, arXiv 1907.06250; https://arxiv.org/abs/1907.06250 17	2019-07-14 / 2026-10-10	Original research paper	Supported within distributed stream-processing scope	Qualify; do not extrapolate its performance results to business workflows.
A green workflow-run count is an adequate end-to-end business KPI.	Contradicted by end-to-end acknowledgement argument and bounded provider guarantees above. 1	Multiple	Synthesis / interpretation	Contradicted	Remove as primary success criterion. Retain only as technical diagnostic.

Evidence-type caution

The AWS Builders’ Library sources are valuable because they are attributable descriptions by AWS engineers of how Amazon/AWS has approached retries and idempotent APIs. They are reported engineering practice, not randomized empirical evidence that the same designs produce a specified improvement in a small Czech or US team. 3

The provider documentation establishes what those provider features promise as documented on the inspection date. It does not establish customer ROI.

The 1984 Saltzer/Reed/Clark paper supplies a design argument rather than a modern cloud benchmark. Its relevance is conceptual: it explicitly analyzes duplicate suppression, message sequencing, delivery acknowledgments and why application-level verification may remain necessary even with lower-level reliability. The authors also caution that the end-to-end argument is a guideline, not an absolute rule, and that lower layers may usefully add reliability for performance. 1

Contradictions, gaps, article angles and freshness triggers

Where the recommendation reverses

Do not build the full recovery architecture just because distributed-systems literature says failures happen. At small enough volume, a native source report plus a human exception queue may be cheaper, clearer and safer. AWS’s own idempotency article says rigorous contracts carry cost and complexity and are not right for every solution. 3

Do not enable every “exactly-once” option by default. BigQuery explicitly recommends considering its simpler default stream when upstream is already at-least-once or duplicates are easy to detect. Pub/Sub documents additional requirements and significantly higher publish-to-subscribe latency for exactly-once subscriptions. 15

Do not interpret the end-to-end argument as “lower-layer reliability is useless.” Saltzer, Reed and Clark explicitly allow lower-level reliability as a performance enhancement and say the correct placement depends on the application. 1

Do not make an AI agent the cure for ambiguous state. An AI classifier cannot manufacture an authoritative source revision, provider transaction record or idempotency contract. In this scope its leverage is triage: compressing context for the human recovery owner after deterministic controls have identified an exception. The economics above show a plausible synthetic case where AI increases, rather than decreases, lifecycle cost.

Do not automatically retry an irreversible side effect after every timeout. A timeout can mean “the action happened and the response was lost.” AWS’s motivating example for idempotency is exactly this ambiguity. 3

Do not use arrival time as business ordering where providers disclaim delivery order. That advice reverses only if the source explicitly defines arrival order as authoritative—which the Stripe webhook example does not. 2

Do not assume compensation restores the prior world. Some effects are irreversible, and Microsoft’s guidance explicitly says compensations are business-specific, can fail and sometimes require manual intervention. 13

Unresolved gaps and inaccessible material

The original 1987 Garcia-Molina and Salem Sagas paper was identified at DOI 10.1145/38713.38742. The ACM full-text endpoint returned an access error in this research environment, and a Cornell-hosted PDF mirror also failed to fetch. I therefore did not rely on the full paper for substantive claims about compensation. Compensation claims above instead use accessible current Microsoft primary documentation. The DOI is reported for provenance, not as evidence silently substituted from secondary summaries.

No live Stripe, AWS, Google Cloud, email-provider or payment-provider experiments were performed. The only executed test was the local deterministic Python fixture shown in full above.

No primary evidence located in this pass establishes a universal rate for duplicate webhook delivery, out-of-order events, recovery labor, AI triage accuracy, financial loss from duplicate effects or ROI from idempotency controls. Any article should avoid turning documented possibility into invented frequency.

The outbound messaging example is intentionally provider-neutral and synthetic. Actual production advice should be re-evaluated against the chosen provider’s current idempotency, message-status and retention semantics.

Likewise, a revision field is a synthetic assumption in the worked order model. Some real sources offer sequence numbers, object versions, ETags or only current-object retrieval; some may provide no total ordering. The reducer should not be transplanted without verifying those source semantics.

There is no evidence in the technical primary sources inspected that the core retry/deduplication architecture should differ categorically between Czech and US teams. Regulatory, contract and retention requirements may change controls around the architecture, but such differences were intentionally not forced into this dossier.

Candidate distinct article angles

Angle	Core argument	Strongest evidence hook	Main qualification
Your automation is green. The customer was still charged twice.	Technical run success is weaker than business success.	End-to-end acknowledgment; timeout ambiguity; provider idempotency boundaries. 1	Use synthetic payment example, never imply an observed customer incident.
Exactly once — exactly where?	Provider exactly-once guarantees end at product boundaries.	Pub/Sub regional/pull scope, BigQuery write-offset scope, SQS FIFO window. 4	Avoid saying exactly-once features are useless.
The cancellation arrived before the order creation. Now what?	Arrival order is not business chronology.	Stripe explicitly disclaims event ordering. 2	Revision-based solution only works with trustworthy source versioning.
Two automations keep triggering each other. Deduplication won’t save you.	A feedback loop can consist entirely of unique messages.	GitHub’s explicit recursion protection as provider example. 14	Provenance/hop-budget architecture is editorial, not a universal platform feature.
429 is not just a retry error. It is a prioritization decision.	Backoff controls traffic; the business must decide what may wait.	RFC 429/Retry-After + Pub/Sub flow control. 9	Deadlines and priorities are process-specific.
An idempotency key is not a magic duplicate remover.	Message identity, operation intent and key lifetime differ.	AWS client-token semantics; Stripe key lifetime. 3	Provider contracts differ materially.
The cheapest recovery system may be a human with a good reconciliation report.	Control depth should follow consequence, volume and maintenance cost.	AWS complexity caveat; BigQuery simpler-default recommendation. 3	Economics must use the team’s own volumes and incident costs.
AI belongs in the exception desk before it belongs in the payment path.	AI can reduce interpretation work without becoming the authority for irreversible effects.	This dossier’s synthetic lifecycle model; deterministic evidence base above.	No empirical AI ROI or accuracy result is claimed here.

Optional reusable assets

The most useful companion asset would be an outcome-first recovery worksheet with five columns:

Business operation
Authoritative identity/version
What can duplicate?
How is external success verified?
What is the safe repair if success is unknown?

A second useful asset is an event timeline debugger showing, per root correlation ID:

received_at
source_event_time
event_id
business_entity_id
source_revision
operation_intent
origin
causation_id
hop_count
processing_state
provider_reference
reconciliation_result

A third is a recovery readiness matrix that fault-injects the exact synthetic cases in this dossier: duplicate delivery, second event for the same business transition, cancellation-before-creation, acknowledgment-before-crash, timeout-after-side-effect, 429/quota backlog, dependency outage, historical replay, concurrent revision race and two-automation feedback loop.

No separate downloadable file is referenced; the complete executed reducer fixture is included inline above.

Freshness triggers

This dossier should be rechecked rather than treated as timeless when any of the following changes:

A source provider changes webhook ordering, retry duration, event identity or API-version semantics; Stripe changes idempotency-key retention behavior; Pub/Sub changes its exactly-once regional/push limitations or latency/quota characteristics; SQS changes FIFO deduplication scope; BigQuery changes Storage Write API offset or schema-evolution behavior; a downstream payment/message provider changes idempotency or lookup support; the order source introduces/removes a trustworthy revision field; the system moves to multi-region subscribers; retry volume begins hitting new quotas; a previously reversible business action becomes irreversible; or an AI model/tool version changes enough to invalidate its triage evaluation. Current Pub/Sub documentation in particular was updated as recently as 9 October 2026, illustrating why provider guarantees deserve explicit inspection dates. 4

Bottom-line decision: measure recovery success as “the intended business state was reached, external effects are known, and discrepancies are reconciled”. Treat green workflow runs, successful message deliveries and provider-level exactly-once guarantees as valuable subordinate evidence—not as substitutes for that outcome. The smallest useful implementation is not an agentic recovery system; it is a durable inbox, explicit business identity/version, operation ledger, bounded retries, reconciliation, a human-owned quarantine path and outcome-level observability. Only after that deterministic foundation is demonstrably carrying enough exception-review cost should bounded AI triage be added.

Portable citation index

1. [web.mit.edu](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf) 2. [docs.stripe.com](https://docs.stripe.com/webhooks) 7. [docs.stripe.com](https://docs.stripe.com/api/idempotent_requests) 3. [references-details-empty](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) 11. [Timeouts, retries, and backoff with jitter | AWS Builder Center](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/) 4. [Exactly-once delivery  |  Pub/Sub  |  Google Cloud Documentation](https://cloud.google.com/pubsub/docs/exactly-once-delivery) 10. [Flow control  |  Pub/Sub  |  Google Cloud Documentation](https://cloud.google.com/pubsub/docs/flow-control-messages) 12. [Dead-letter topics  |  Pub/Sub  |  Google Cloud Documentation](https://cloud.google.com/pubsub/docs/dead-letter-topics) 15. [BigQuery Storage Write API (gRPC) best practices  |  Google Cloud Documentation](https://cloud.google.com/bigquery/docs/write-api-best-practices) 5. [spec/cloudevents/spec.md at main · cloudevents/spec · GitHub](https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md) 6. [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110.html) 9. [www.rfc-editor.org](https://www.rfc-editor.org/rfc/rfc6585.html) 8. [Amazon SQS at-least-once delivery - Amazon Simple Queue Service](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html) 16. [Exactly-once processing in Amazon SQS - Amazon Simple Queue Service](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html) 13. [Compensating Transaction Pattern - Azure Architecture ...](https://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction?utm_source=chatgpt.com) 14. [GITHUB_TOKEN - GitHub Docs](https://docs.github.com/en/actions/concepts/security/github_token) 17. [Delivery, consistency, and determinism: rethinking guarantees in distributed stream processing](https://arxiv.org/abs/1907.06250?utm_source=chatgpt.com)

Import verification and asset availability

Genuine Deep Research completed in 9 minutes and was captured on 2026-10-10 after the first due time. Original export, exact independent prompt, canonical chat and citation URLs are retained. Complete inline examples/code were preserved; no missing sandbox download target is claimed as a delivered asset. Report code, test outcomes and execution assertions were not independently run or verified during this import. Delivery is not independent claim verification; Evidence status remains Captured.