# Retries, Duplicates, and Idempotency: Evidence Dossier **Research snapshot:** 2026-10-10 **Scope:** Small-team, event/webhook-driven order workflow feeding a cloud analytical warehouse and downstream business automation. Public sources only; examples and fixtures below are synthetic. **Decision supported:** **Explain recovery design through concrete business outcomes rather than green workflow-run status.** ## Executive answer and uncertainties The evidence strongly supports the decision. A successful HTTP delivery, queue acknowledgment, workflow execution, or even a provider-specific “exactly once” guarantee is **not equivalent to the business action having happened correctly**. This distinction is old and fundamental: Saltzer, Reed, and Clark’s end-to-end argument notes that an acknowledgment that a message reached a host does not establish that the application actually performed the requested action. [1](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf) Modern provider documentation makes the same boundary visible. Stripe labels a webhook delivery successful when the endpoint returns a successful HTTP status; separately, Stripe recommends asynchronous processing and warns that events can be duplicated and arrive out of order. [2](https://docs.stripe.com/webhooks) For the bounded order process in this dossier, **“done” should mean something like this**: > The authoritative order revision has been durably accepted; the canonical business state and warehouse agree with that revision; every required downstream business intent has a known outcome or an explicitly owned unresolved state; prohibited older actions did not occur; and the whole chain can be reconciled from an audit trail. A green run is useful diagnostic evidence. It is not that outcome. The key distinctions are: | Concept | What it actually answers | What it does **not** establish | |---|---|---| | **Retry safety** | “Can I make this attempt again without an unacceptable additional effect?” HTTP defines idempotency in terms of the intended server effect of repeated identical requests. [3](https://www.rfc-editor.org/rfc/rfc9110.html) | That another system has not already independently performed the same business intent. | | **Idempotency key** | “Are these attempts the same logical request?” Stripe, for example, allows callers to associate POST retries with a client-generated key. [4](https://docs.stripe.com/api/idempotent_requests) | Permanent deduplication. Provider key retention is finite and provider-specific. | | **Transport deduplication** | “Have I seen this delivery/message identity before?” SQS FIFO, Cloud Tasks, Shopify and Pub/Sub expose different mechanisms and windows. [5](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html) | That two different messages do not represent the same business action. | | **Business idempotency** | “Has this order/revision/intent already produced the required business effect?” | It cannot normally be inferred from a transport ID alone. Saltzer et al. specifically observed that application-generated duplicate requests can look like distinct network messages. [1](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf) | | **Exactly-once claim** | A guarantee over a specifically documented boundary. Pub/Sub, Kafka, SQS and BigQuery attach materially different scopes and conditions to such guarantees. [6](https://cloud.google.com/pubsub/docs/exactly-once-delivery) | End-to-end “customer was charged/notified exactly once” correctness across independent systems. | | **Compensation** | A subsequent business operation that offsets an already-completed effect when atomic rollback is impossible. | Erasure of every consequence: an email already read, shipment already dispatched, or financial transaction already settled may not be fully reversible. | **Most important practical conclusion:** use provider-level duplicate suppression and exactly-once features when they simplify the design, but keep the **business invariant above them**. Pub/Sub explicitly limits exactly-once delivery to pull subscriptions and a regional scope, while acknowledging that publish-side duplicates can still result in multiple messages; Kafka states that exactly-once processing for non-Kafka destinations generally needs cooperation from the destination system. [6](https://cloud.google.com/pubsub/docs/exactly-once-delivery) BigQuery similarly distinguishes a default at-least-once write stream from client-offset-based exactly-once streams. [7](https://cloud.google.com/bigquery/docs/write-api) There are important uncertainties. Provider documentation establishes specified behavior, not measured real-world reduction in incidents, maintenance effort, or duplicate financial effects. I found no strong primary comparative study quantifying the business ROI of manual versus deterministic versus AI-assisted recovery for a small team. The recommendation that AI stay outside irreversible authorization in this workflow is therefore **editorial risk containment**, not an empirically demonstrated effect size. Likewise, the dossier does not perform a Czech-versus-US compliance comparison. The technical failure semantics reviewed are primarily provider/service/configuration dependent; regional configuration can matter—for example Pub/Sub’s exactly-once scope is regional—but I found no reason in the reviewed technical evidence to make Czech versus US company location the main architecture distinction. [6](https://cloud.google.com/pubsub/docs/exactly-once-delivery) ## Bounded process and viable alternatives The process must be defined before the automation. **Synthetic bounded process: order-state propagation and recovery.** An e-commerce or order-management source emits creation, update and cancellation events. The workflow must preserve the authoritative state in an analytical warehouse and, for selected meaningful transitions, create downstream business actions such as stopping fulfillment, issuing a refund, or sending a customer notification. The synthetic system deliberately permits retries, replay, duplicates and out-of-order delivery because those are documented characteristics of common webhook and messaging systems. Stripe states that webhook endpoints can receive duplicate events and that event order is not guaranteed; Shopify similarly documents duplicate deliveries and recommends idempotent processing or explicit webhook-ID tracking. [2](https://docs.stripe.com/webhooks) **Trigger:** receipt of an authenticated webhook or event. **Minimum input model:** ```text delivery_id identity of this transport delivery event_id identity of the source event/action, if supplied order_id identity of the business entity source_revision authoritative monotonic revision / sequence, if available event_type created | updated | confirmed | cancelled | ... operation_intent semantic action derived locally, e.g. refund:cancellation:r12 source_timestamp informational timestamp received_at local receipt timestamp origin source / automation-A / automation-B / operator payload source fields required for state/reconciliation ``` These identities should not be collapsed. Shopify provides a particularly concrete example: it documents a webhook delivery identifier separately from an event identifier, with the latter usable to correlate deliveries arising from the same merchant action. [8](https://shopify.dev/docs/apps/build/webhooks/subscribe/https) Stripe goes one step further for some duplicate cases: separate Stripe `Event` objects can represent the same object/action, so its documentation advises using the object ID together with the event type when identifying those duplicates. [2](https://docs.stripe.com/webhooks) **Business owner:** Order Operations owns the meaning of `CONFIRMED`, `CANCELLED`, deadlines and acceptable compensation. **Technical owner:** the integration/data team owns durable acceptance, queueing, state reduction, provider contracts and observability. **Recovery owner:** a named operator owns reconciliation failures and uncertain external side effects. One person can wear several hats in a small team; the responsibilities should still be explicit. A useful synthetic business-state model is: ```text CREATED ──> CONFIRMED ──> CANCELLED │ │ └─────────────┴──────> CANCELLED Newer authoritative revision wins. An older event may be retained in the event ledger but must not regress canonical state. Same revision + contradictory state => quarantine, not arbitrary winner. ``` The **processing** state is different: ```text RECEIVED ↓ DURABLY_ACCEPTED ↓ STATE_REDUCED ├──> WAREHOUSE_PENDING ──> WAREHOUSE_OBSERVED └──> BUSINESS_INTENT_PENDING ──> EFFECT_OBSERVED └──> OUTCOME_UNCERTAIN ↓ RECONCILED ↓ COMPLETE ``` The separation matters. A `CANCELLED` order can have a processing state of `OUTCOME_UNCERTAIN`; pretending those are the same dimension is how green dashboards hide business failures. **Observable completion** for synthetic order `O-42`, revision `12 = CANCELLED`, could therefore be defined as: ```text source_current_revision(O-42) == 12 canonical_state(O-42) == CANCELLED @ revision 12 warehouse_state(O-42) == CANCELLED @ revision 12 required_refund_intent(O-42, rev12) has known final provider status required_cancellation_notice(O-42, rev12) has known accepted/sent status no older CONFIRMED event subsequently changed canonical state no second refund exists for the same cancellation intent reconciliation_status == matched ``` Notice the wording around a message: unless the delivery channel exposes stronger evidence, “provider accepted/sent it” is observable while “the human actually read it” may not be. **Manual/native fallback** is deliberately boring: an operator opens the authoritative order, verifies its current revision/state, checks the payment or messaging provider, compares the warehouse record, and performs or approves the missing deterministic repair. The audit record captures what the operator saw and why the repair was safe. The same process can then be implemented at three autonomy levels: | Variant | Same business outcome | Normal path | Recovery | Upkeep profile | Best fit | |---|---|---|---|---|---| | **Manual/native baseline** | Source, warehouse and required downstream action reconcile. | Native source UI/export/connectors; operator checks exceptions and executes actions. | Human compares authoritative source with downstream systems. | Lowest custom-code surface; recurring human work and slower response. | Low volume, rare exceptions, high consequence, or immature state semantics. | | **Deterministic automation — recommended default** | Same. | Durable inbox → revision-aware reducer → queue/outbox → warehouse/effect → reconciler. | Replay safe operations; query uncertain effects; compensate or escalate where needed. | More engineering: schemas, API changes, queue configuration, reconciliation and runbooks. | Stable process with enough repetition to justify maintenance. | | **Bounded AI triage assistant** | **Still the same.** | Deterministic pipeline remains authoritative. AI summarizes DLQ items, groups likely causes, proposes a repair and surfaces relevant audit records. | Human approves a deterministic recovery command. | Adds model/prompt/evaluation drift and review burden. | Exception diagnosis is cognitively expensive but authorization risk should remain bounded. | The AI variant **does not authorize a refund/payment, send a customer message, or make a production write** in this design. That restriction is a design recommendation rather than a provider-documented necessity or measured empirical result. The simplest baseline should win when it already meets the outcome. More autonomy is not inherently superior. ## Evidence synthesis around recovery outcomes **Acknowledge durable work, not wishful work.** Stripe recommends returning a successful response quickly and handling complex webhook processing asynchronously. Shopify likewise recommends queueing webhook work, especially for bursts, and separately recommends reconciliation for missed data. [2](https://docs.stripe.com/webhooks) The critical design inference is to combine “fast ACK” with **durable acceptance before ACK**: verify enough of the request to accept it, persist an immutable receipt or durable work item, then return success and perform expensive work asynchronously. Otherwise a crash after `200 OK` but before persistence can produce an acknowledged-but-lost business event. Stripe’s delivery UI explicitly treats a successful HTTP response as successful endpoint delivery, not proof that later accounting or business work happened. [2](https://docs.stripe.com/webhooks) This is precisely the distinction made by the end-to-end argument: application correctness requires evidence that the requested application action occurred, not just lower-level delivery. [1](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf) **Retries solve transient failures but introduce ambiguity and load.** HTTP itself permits clients to retry idempotent methods after communication failure because repeating their intended effect is defined as equivalent; it warns clients not to automatically retry non-idempotent methods unless the client knows the operation is actually safe or knows the original request was not applied. [3](https://www.rfc-editor.org/rfc/rfc9110.html) AWS’s retry guidance adds an operational warning: retries can amplify overload, and the caller can time out even when the remote side effect actually happened. It recommends bounded retries, exponential backoff and jitter, and avoiding retries independently at every stack layer. Its illustrative calculation shows that five layers each making up to three attempts can drive **3⁵ = 243** attempts against the deepest dependency—not an observed universal effect size, but a straightforward worst-case fan-out model. [9](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/) The planning formula is visible: \[ \text{maximum downstream attempts} = r^d \] where `r` is attempts made independently per layer and `d` is the number of retrying layers. For `r = 3`, `d = 5`, the result is `243`. The implication is not “never retry”; it is to decide **where retry ownership lives** and make retry budgets visible. **Idempotency-key lifetime is not business-memory lifetime.** Stripe API v1 documentation says idempotency keys can be removed after they are at least 24 hours old; reusing a pruned key can therefore result in a fresh request. [4](https://docs.stripe.com/api/idempotent_requests) SQS FIFO deduplication recognizes a send retry within a five-minute deduplication interval. [5](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html) Cloud Tasks remembers a task name for up to 24 hours after task deletion. [10](https://cloud.google.com/tasks/docs/dual-overview) Those are three materially different windows. Therefore a payment, cancellation or notification that must remain unique across week-old replay needs an application/business identity whose lifetime is derived from the process—not merely the provider’s short-term dedup token. A robust operation identity might be: ```text operation_id = SHA256(order_id + ":" + semantic_transition + ":" + source_revision) Example: O-42 : refund_due_to_cancellation : 12 ``` The transport delivery can change; the business intent remains the same. **“Exactly once” is useful only after asking “exactly once where?”** Google Pub/Sub’s exactly-once feature prevents redelivery after successful acknowledgment under documented conditions, but it is limited to pull subscriptions and regional operation. Its documentation also warns that multiple unique publish retries can still yield multiple messages even with exactly-once delivery enabled. [6](https://cloud.google.com/pubsub/docs/exactly-once-delivery) Apache Kafka 4.3 documents exactly-once read/process/write semantics for Kafka topics using transactions and read-committed consumers, but explicitly says exactly-once delivery into other destination systems generally requires cooperation from those systems. [11](https://kafka.apache.org/43/design/design/) BigQuery’s Storage Write API similarly provides exactly-once stream appends when the client uses committed streams with offsets, while the default stream remains at-least-once. [7](https://cloud.google.com/bigquery/docs/write-api) SQS FIFO’s “exactly-once processing” documentation specifically means that repeated sends within its deduplication window do not introduce duplicate queue messages. [5](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html) These claims are not contradictory once their boundaries are preserved. They become misleading only when translated into “therefore the customer cannot be charged twice.” **Out-of-order arrival should not define business truth.** Pub/Sub ordered delivery is conditional on the same ordering key and publishing region and does not establish a global order across keys; redelivery behavior can also cause later messages on an ordering key to be redelivered. [12](https://cloud.google.com/pubsub/docs/ordering) BigQuery’s CDC documentation directly recognizes the problem: ingest time is normally used to choose precedence between records with the same primary key, but where update order is not guaranteed Google provides `_CHANGE_SEQUENCE_NUMBER` so the producer can supply the intended logical order. [13](https://cloud.google.com/bigquery/docs/change-data-capture) BigQuery also maintains a two-day retention window specifically to support CDC `DELETE` operations arriving out of order. [13](https://cloud.google.com/bigquery/docs/change-data-capture) For the article-target question **“Cancellation arrived before confirmation. Which state should automation use?”**, the answer is: > Use the state attached to the newest **authoritative source revision**, not the event that happened to arrive last. If revision `12 = CANCELLED` arrives before revision `11 = CONFIRMED`, revision 12 remains current when revision 11 eventually appears. If the source does not expose a trustworthy monotonic revision, the safer recovery is to read current authoritative source state before an irreversible effect. If neither a version nor an authoritative current-state query exists, contradictory events should be quarantined; arrival timestamps alone do not manufacture certainty. **Rate limiting is a business-priority problem as well as a queue problem.** Cloud Tasks lets an operator bound dispatch rate and concurrency, applies rate limits to first attempts and retries, and supports configurable retry duration/backoff. [14](https://docs.cloud.google.com/tasks/docs/configuring-queues) BigQuery recommends buffering important records, retrying transient insertion failures, handling `429` quota exhaustion, and persisting records elsewhere if every row is crucial and an outage could outlast the available buffer. Conversely, its documentation explicitly allows abandoning data when missing it is acceptable. [7](https://cloud.google.com/bigquery/docs/write-api) That gives a cleaner answer to **“What can wait and what must not be lost?”**: ```text Can wait = work whose business deadline has not expired. Can be lost = work the business explicitly declares reconstructable or disposable. Must persist = work whose disappearance would violate the business invariant. Must lead = work with the shortest consequential business deadline. ``` For example, a warehouse enrichment calculated again from the source may safely wait or be reconstructed. A cancellation intended to stop imminent fulfillment may have a much harder deadline. The queue does not know that distinction until the application encodes it. A useful queue design is therefore based on `(loss tolerance, deadline)`, not simply “failed/succeeded”: ```text durable accepted work │ ├── deadline-critical queue │ └── bounded retry → reconcile/escalate before deadline │ ├── normal durable queue │ └── backoff/jitter → DLQ → human repair │ └── reconstructable/best-effort └── delay, batch, or discard under explicit policy ``` **Feedback loops should be stopped at the meaningful-transition boundary, not merely by counting invocations.** AWS Lambda documents a native recursion detector that stops the next invocation after approximately 16 invocations in a chain for supported service combinations, but it also says detection covers only particular AWS services/SDK paths; loops involving unsupported services can escape it. [15](https://docs.aws.amazon.com/lambda/latest/dg/invocation-recursion.html) AWS therefore provides a useful emergency fuse, not a universal semantic solution. For **“Two automations trigger each other. How do we stop the infinite loop?”**, the deterministic design is: ```text 1. Trigger only on a meaningful business transition, not every write. 2. Suppress no-op writes: new_value == current_value => do nothing. 3. Carry origin/causation metadata. 4. Suppress an echo generated by the same operation_id. 5. Keep a hop/time budget as an emergency circuit breaker. 6. Alert and quarantine when the circuit breaker fires. ``` If automation A writes `sync_status=CANCELLED` and automation B writes that same status back, neither write should qualify as a new business transition. A recursion counter is defense in depth. It should not be the primary definition of correctness. ## Worked failure dossier and portable test asset The following timeline is **synthetic**, not a reported customer case. It combines failure modes documented separately by webhook, queue and warehouse providers. Stripe documents retries, duplicates and unordered webhook delivery; Cloud Tasks documents configurable retry/rate behavior; BigQuery documents offset-based deduplication and source-order controls. [2](https://docs.stripe.com/webhooks) | Time | Input / failure | Unsafe implementation | Safe interpretation and repair criterion | |---|---|---|---| | `10:00:00` | `O-42 r10 CONFIRMED`, delivery `d1`, event `evt10` | Apply and immediately trigger every downstream automation. | Durably accept, apply revision 10, create only semantic intents implied by transition. | | `10:00:01` | Exact redelivery of `d1` | Execute again. | Same `delivery_id`: record duplicate and no-op. | | `10:00:03` | `O-42 r12 CANCELLED`, `d2/evt12` arrives before r11 | Arrival-order reducer sets cancelled—for now. | Apply because 12 > 10; create cancellation/refund intents identified by `(O-42, intent, r12)`. | | `10:00:04` | Refund API accepts request, client times out before response | “Workflow failed” → issue a new refund request with a new key. | Outcome is **unknown**, not failed. Query provider or retry the same valid idempotency identity. A communication timeout does not prove the first side effect failed. HTTP and AWS guidance both distinguish this ambiguity. [3](https://www.rfc-editor.org/rfc/rfc9110.html) | | `10:00:07` | Downstream says `429` | Tight loop retry. | Leave intent durable; retry according to bounded backoff/deadline policy. Retries themselves must obey rate limits. [14](https://docs.cloud.google.com/tasks/docs/configuring-queues) | | `10:00:09` | `O-42 r11 CONFIRMED`, `d3/evt11` arrives late | “Latest received wins” → resurrect CONFIRMED. | Store for audit, but do not apply because `11 < 12`. BigQuery exposes a source-supplied sequence mechanism for this class of problem. [13](https://cloud.google.com/bigquery/docs/change-data-capture) | | `10:02:00` | Replay sends `evt12` again under new delivery `d4` | Delivery ID differs, therefore treat as new cancellation. | Event identity reveals replay; even if event ID also changed, `(order, cancellation/refund intent, rev12)` should already exist. | | `10:03:00` | Automation A writes `sync_status=CANCELLED`; automation B mirrors it back | Each write triggers the other indefinitely. | Same semantic value + same causal operation => no transition, no new output. Emergency hop budget catches defects. AWS's provider-native detector demonstrates the value and limits of a separate recursion fuse. [15](https://docs.aws.amazon.com/lambda/latest/dg/invocation-recursion.html) | | `10:15:00` | Warehouse becomes available after backlog | Re-run everything from scratch including side effects. | Replay immutable events through state reduction; warehouse writes can be retried independently of already-observed payment/message effects. | | `10:30:00` | Reconciler runs | Sees all workflow runs green and closes incident. | Compare source r12, canonical r12, warehouse r12 and provider-side refund/message records. Only close on invariant match. Saltzer et al.'s end-to-end argument is the foundational rationale. [1](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf) | The recovery rule is consequently **state-dependent**, not “retry on red”: | Observed state | Safe action | |---|---| | No side effect attempted | Retry when dependency policy permits. | | Side effect returned a definitive non-application result | Retry same logical operation. | | Side effect may have succeeded but reply was lost | **Reconcile/query first**, or retry with the same provider idempotency identity while its documented guarantee still applies. | | Idempotency key window expired and provider outcome is unknown | Do not blindly create a new operation; query by business/reference identity or escalate. | | Duplicate delivery/event | Store audit evidence; no-op. | | Different events represent same business intent | Deduplicate by business operation identity. | | Older source revision | Store but do not apply to canonical state. | | Same source revision contains contradictory states | Quarantine for authoritative reconciliation. | | Wrong reversible effect definitely happened | Run an explicitly defined compensation. | | Wrong irreversible effect happened | Do not pretend rollback exists; contain consequences, record them, and require human recovery. | | Work has exhausted retry policy | DLQ/quarantine must preserve enough immutable input and context for deterministic replay. | ### Portable executed fixture The following Python fixture is complete and self-contained. It exercises duplicate delivery, replay under a different delivery ID, and an out-of-order older revision. **It was executed during this research session.** It is a synthetic model, not a provider integration test. ```python from dataclasses import dataclass from typing import Dict, List, Tuple @dataclass(frozen=True) class Event: delivery_id: str event_id: str order_id: str revision: int state: str @dataclass class Canonical: revision: int state: str class Reducer: def __init__(self) -> None: self.seen_deliveries = set() self.seen_events = set() self.orders: Dict[str, Canonical] = {} self.intents = set() self.audit: List[Tuple] = [] def process(self, event: Event) -> None: # Transport identity. if event.delivery_id in self.seen_deliveries: self.audit.append( (event.delivery_id, "duplicate_delivery_skipped") ) return self.seen_deliveries.add(event.delivery_id) # Source event identity is different from delivery identity. duplicate_event = event.event_id in self.seen_events self.seen_events.add(event.event_id) current = self.orders.get(event.order_id) # Business/source ordering beats arrival ordering. if current and event.revision < current.revision: self.audit.append( ( event.delivery_id, "stale_revision_stored_not_applied", event.revision, current.revision, ) ) return # Identical source revision should not repeatedly create effects. if current and event.revision == current.revision: if event.state == current.state: self.audit.append( ( event.delivery_id, "same_revision_same_state_noop", duplicate_event, ) ) return # Contradictory truth at the same revision is not safely resolvable # from arrival order. self.audit.append( ( event.delivery_id, "conflict_quarantine", event.revision, current.state, event.state, ) ) return self.orders[event.order_id] = Canonical( event.revision, event.state ) self.audit.append( ( event.delivery_id, "state_applied", event.revision, event.state, ) ) # Semantic/business identity: independent of delivery ID. if event.state == "CANCELLED": operation_id = ( event.order_id, "notify_cancellation", event.revision, ) if operation_id not in self.intents: self.intents.add(operation_id) self.audit.append( ( event.delivery_id, "intent_created", operation_id, ) ) events = [ Event("d1", "evt10", "O-42", 10, "CONFIRMED"), Event("d1", "evt10", "O-42", 10, "CONFIRMED"), # delivery duplicate Event("d2", "evt12", "O-42", 12, "CANCELLED"), # newer event arrives Event("d3", "evt11", "O-42", 11, "CONFIRMED"), # late older event Event("d4", "evt12", "O-42", 12, "CANCELLED"), # replay, new delivery ID ] reducer = Reducer() for event in events: reducer.process(event) print(reducer.orders) print(reducer.intents) for row in reducer.audit: print(row) ``` Executed result: ```text {'O-42': Canonical(revision=12, state='CANCELLED')} {('O-42', 'notify_cancellation', 12)} ('d1', 'state_applied', 10, 'CONFIRMED') ('d1', 'duplicate_delivery_skipped') ('d2', 'state_applied', 12, 'CANCELLED') ('d2', 'intent_created', ('O-42', 'notify_cancellation', 12)) ('d3', 'stale_revision_stored_not_applied', 11, 12) ('d4', 'same_revision_same_state_noop', True) ``` What this fixture **does not test**: an actual webhook signature, real queue durability, real Stripe idempotency behavior, BigQuery CDC, provider outages, concurrent workers, database transaction isolation, real rate limits, real message delivery, or an AI assistant. Those tests remain unperformed. A smallest workable production evaluation should therefore inject at least these cases before launch: ```text duplicate same delivery ID duplicate same source event with new delivery ID two different events representing same business action newer cancellation followed by older confirmation same revision with contradictory payload crash immediately before durable acceptance crash immediately after durable acceptance but before ACK crash after side effect but before recording response 429 / 503 dependency responses queue backlog beyond normal latency replay older than provider idempotency-key lifetime two concurrent workers for same order automation-A → automation-B → automation-A loop manual override while automated work is queued DLQ replay after source has advanced again ``` The acceptance assertion for every case should be a **business invariant**, not merely “test runner exited zero.” ## Lifecycle, observability, and smallest workable implementation The minimum deterministic architecture is smaller than a full orchestration platform: ```text Source webhook │ ▼ Authenticate / minimally validate │ ▼ ┌──────────────────────────────┐ │ Durable inbox / event ledger │ │ unique delivery/event fields │ └──────────────────────────────┘ │ durable acceptance ├───────────────> HTTP 2xx │ ▼ Revision-aware state reducer │ ├──> canonical order state │ ├──> warehouse work │ └──> business-operation outbox │ ▼ rate-limited worker │ ┌─────────┴──────────┐ ▼ ▼ external side effect outcome uncertain │ │ └─────────┬──────────┘ ▼ reconciler │ matched / repair ``` The important transaction boundary is not “everything across every SaaS API is atomic”—usually it is not. It is that the system should not acknowledge accepted work while having no durable record from which to recover. The external side effect then lives behind its own persistent operation identity and reconciliation state. This design is consistent with provider recommendations to ACK webhooks quickly and process them asynchronously, while adding the end-to-end correctness check that transport documentation cannot supply. [2](https://docs.stripe.com/webhooks) For warehouse ingestion, BigQuery documents both sides of the trade-off. When each record matters, Google recommends buffering before insertion, persisting data when necessary through prolonged unavailability, putting poison records aside for investigation, and using explicit offsets when exactly-once append semantics are required. If exactly-once behavior is unnecessary, the default stream has lower complexity and higher throughput characteristics. [7](https://cloud.google.com/bigquery/docs/write-api) That is a useful general lesson: **stronger guarantees have cost and complexity and should be bought where the outcome needs them.** The dashboard should therefore lead with: ```text accepted but not reconciled orders oldest unreconciled business deadline orders whose source revision != canonical revision orders whose canonical revision != warehouse revision business intents pending / uncertain / compensated duplicate delivery count duplicate business-intent suppression count stale-event count same-revision conflict count rate-limit / dependency-error count DLQ age and count feedback-loop fuse activations manual recoveries and reason codes ``` Provider/workflow run success belongs below those metrics. Pub/Sub itself documents higher latency for exactly-once subscriptions and throughput implications when exactly-once is combined with ordered delivery, illustrating why infrastructure metrics and business correctness metrics need separate treatment. [6](https://cloud.google.com/pubsub/docs/exactly-once-delivery) An audit record should contain enough information to answer an incident without reconstructing it from scattered logs: ```json { "delivery_id": "d4", "event_id": "evt12", "order_id": "O-42", "source_revision": 12, "source_state": "CANCELLED", "received_at": "2026-10-10T10:02:00Z", "operation_id": "O-42:refund_due_to_cancellation:12", "provider_idempotency_key": "O-42:refund_due_to_cancellation:12", "idempotency_key_created_at": "...", "idempotency_key_known_valid_until": "... or provider-specific/unknown", "attempt": 2, "provider_object_id": "...", "provider_status": "succeeded | declined | unknown", "canonical_decision": "stale | apply | no-op | quarantine", "reconciliation_status": "matched | mismatch | pending", "human_actor": null, "human_reason": null } ``` This also exposes an often-hidden maintenance burden: **provider guarantees themselves are configuration and version dependencies.** Stripe's idempotency semantics differ between API v1 and v2 on its current documentation page; Pub/Sub’s guarantee depends on subscription type and regional conditions; Cloud Tasks has configurable retry limits; Kafka exactly-once behavior depends on transactions and consumer isolation; BigQuery distinguishes stream modes. [4](https://docs.stripe.com/api/idempotent_requests) A lifecycle-cost model should therefore include more than build cost: \[ C_{\text{annual}} = \frac{C_{\text{build}}}{A} + C_{\text{platform}} + C_{\text{monitoring}} + C_{\text{schema/API change}} + C_{\text{reconciliation}} + C_{\text{incidents}} + C_{\text{human exceptions}} + C_{\text{retirement}} \] where `A` is the chosen amortization period in years. This is a **planning model, not an empirical cost estimate**; no numerical market benchmark was found in the reviewed primary evidence. The manual baseline remains preferable when: - volume and exception cost are below the lifecycle burden of automation; - the source/native integration already supplies adequate reconciliation; - the business cannot define a stable identity or authoritative ordering rule; - the external side effect is high-consequence but offers neither idempotency nor a reliable status lookup; - changes to the underlying process are frequent enough that the automation would continuously chase them. Conversely, deterministic automation becomes attractive when the transition rules, identities and source authority are stable and there is enough repeated work to amortize the reconciler and maintenance. There are also cases where the recommendation to maximize reliability **reverses**. Saltzer et al. explicitly use real-time voice as an example in which retransmission delay can be worse than a small loss, showing that reliability controls depend on the application’s actual requirements. [1](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf) Low-consequence telemetry may likewise rationally use at-most-once or best-effort behavior. Fully Kafka-contained processing can rely more heavily on Kafka transactions because the guarantee boundary encompasses both consumed offsets and produced Kafka records; that argument weakens as soon as the outcome moves into an independent destination. [11](https://kafka.apache.org/43/design/design/) Intentionally recursive Lambda designs are also possible—AWS permits recursion protection to be disabled—but AWS itself recommends caution and guardrails. [15](https://docs.aws.amazon.com/lambda/latest/dg/invocation-recursion.html) The smallest workable implementation I would choose for this order workflow is therefore: **one durable inbox, one canonical revision-aware order table, one durable table of semantic business intents, bounded queue workers, and one periodic reconciler.** Add provider-specific exactly-once features where cheap. Add a DLQ only together with a named repair path. Add AI only after deterministic repair primitives exist. That is materially more valuable than adding another orchestration layer whose principal achievement is turning the run green. ## Claim-to-primary-source matrix The matrix separates provider behavior, foundational argument, attributable engineering practice, and editorial interpretation. “Supported” means the cited primary source directly supports the bounded claim; it does not imply an independently measured business effect. | Claim target / exact scope | Primary source and exact section | Source date | Inspected | Evidence type | Assessment | Editorial action | |---|---|---:|---:|---|---|---| | Repeating an HTTP request is automatically safe only when its semantics are idempotent, or the client otherwise knows repetition is safe. | RFC 9110 §9.2.2, **Idempotent Methods**. [3](https://www.rfc-editor.org/rfc/rfc9110.html) URL: `https://www.rfc-editor.org/rfc/rfc9110.html#section-9.2.2` | 2022-06 | 2026-10-10 | Internet standard | **Supported** | **Keep.** Use as definition of retry safety, not business dedup. | | A lower-level successful delivery does not prove the requested application action completed; duplicate suppression and ordering can require application knowledge. | Saltzer, Reed & Clark, **End-to-End Arguments in System Design**, “Delivery guarantees,” “Duplicate message suppression,” “Guaranteeing FIFO message delivery.” [1](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf) DOI: `https://doi.org/10.1145/357401.357402`; manuscript: `https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf` | 1984-11 | 2026-10-10 | Peer-reviewed foundational systems paper | **Supported** | **Keep.** This is the strongest conceptual basis for “business outcome, not green run.” | | Stripe POST retries can be tied to an idempotency key; API v1 retains results once endpoint execution begins, and keys can be pruned after at least 24h. | Stripe API, **Idempotent requests**. [4](https://docs.stripe.com/api/idempotent_requests) URL: `https://docs.stripe.com/api/idempotent_requests` | Undated live documentation | 2026-10-10 | Provider documentation | **Supported within Stripe/API-version scope** | **Qualify.** Never imply infinite lifetime or universal provider semantics. | | A Stripe webhook may be delivered repeatedly and out of order; successful HTTP delivery is distinct from downstream business processing. | Stripe, **Webhooks**: event delivery behavior, automatic retries, event ordering, duplicate events, asynchronous handling. [2](https://docs.stripe.com/webhooks) URL: `https://docs.stripe.com/webhooks` | Undated live documentation | 2026-10-10 | Provider documentation | **Supported** | **Keep.** Excellent concrete example. | | Delivery identity and source-event identity are not necessarily the same thing. | Shopify, **Verify webhook deliveries**: duplicate deliveries and `Webhook-Id` versus `Event-Id`. [8](https://shopify.dev/docs/apps/build/webhooks/subscribe/https) URL: `https://shopify.dev/docs/apps/build/webhooks/verify-deliveries` | Undated live documentation | 2026-10-10 | Provider documentation | **Supported within Shopify scope** | **Keep, then generalize only as an architecture distinction.** | | SQS FIFO prevents a retry from introducing a duplicate message only within its defined deduplication mechanism/window; this is not an end-to-end business guarantee. | AWS SQS Developer Guide, **Exactly-once processing**. [5](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html) URL: `https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html` | Undated live documentation | 2026-10-10 | Provider documentation | **Supported/conditional** | **Rewrite any naked “exactly once” claim to include the queue boundary and five-minute dedup interval.** | | Pub/Sub exactly-once is conditional: pull subscriptions, regional scope; publish-side duplicates can still produce multiple messages. | Google Cloud Pub/Sub, **Exactly-once delivery**, including “Exactly-once delivery and push subscriptions” and “Things to know.” [6](https://cloud.google.com/pubsub/docs/exactly-once-delivery) URL: `https://cloud.google.com/pubsub/docs/exactly-once-delivery` | Last updated 2026-10-09 UTC | 2026-10-10 | Provider documentation | **Supported** | **Keep.** Strong counterexample to treating the phrase as universal. | | Provider ordering does not remove the need for business ordering; Pub/Sub ordering is per ordering key and has redelivery/latency implications. | Google Cloud Pub/Sub, **Order messages**. [12](https://cloud.google.com/pubsub/docs/ordering) URL: `https://cloud.google.com/pubsub/docs/ordering` | Last updated 2026-10-09 UTC | 2026-10-10 | Provider documentation | **Supported/conditional** | **Keep, qualified to Pub/Sub.** | | Rate limits and retries share capacity; retries should be explicitly bounded/backed off rather than hidden inside an unbounded workflow. | Google Cloud Tasks, **Configure queue routing, limits, and retries**, rate limits and retry parameters. [14](https://docs.cloud.google.com/tasks/docs/configuring-queues) URL: `https://cloud.google.com/tasks/docs/configuring-queues` | Last updated 2026-10-07 UTC | 2026-10-10 | Provider documentation | **Supported** | **Keep.** Business prioritization on top is editorial design. | | Cloud Tasks transport deduplication has a finite identity lifetime: task names are remembered for up to 24h after deletion. | Google Cloud Tasks, **Understand Cloud Tasks → Task deduplication**. [10](https://cloud.google.com/tasks/docs/dual-overview) URL: `https://cloud.google.com/tasks/docs/dual-overview` | Live documentation; page date not captured | 2026-10-10 | Provider documentation | **Supported** | **Keep as TTL counterexample.** | | BigQuery default-stream writes are at-least-once; committed streams plus caller-supplied offsets can provide exactly-once append behavior. Crucial data should be buffered/persisted through outages. | BigQuery, **Introduction to the Storage Write API**, stream types and “Handle unavailability” / reliability guidance. [7](https://cloud.google.com/bigquery/docs/write-api) URL: `https://cloud.google.com/bigquery/docs/write-api` | Last updated 2026-10-07 UTC | 2026-10-10 | Provider documentation | **Supported** | **Keep.** Do not translate stream-level dedup into business-level exact-once. | | When record arrival order is unreliable, BigQuery CDC can use a caller-supplied `_CHANGE_SEQUENCE_NUMBER`; ingest-time order is insufficient in such cases. | BigQuery, **Change data capture → Manage custom ordering**. [13](https://cloud.google.com/bigquery/docs/change-data-capture) URL: `https://cloud.google.com/bigquery/docs/change-data-capture` | Last updated 2026-10-07 UTC | 2026-10-10 | Provider documentation | **Supported** | **Keep.** Strong concrete support for “source version beats arrival order.” | | Out-of-order deletes are a real enough ingestion case that BigQuery CDC maintains a two-day delete-retention window. | BigQuery, **Change data capture → Limitations/DELETE behavior**. [13](https://cloud.google.com/bigquery/docs/change-data-capture) Same URL as above. | Last updated 2026-10-07 UTC | 2026-10-10 | Provider documentation | **Supported within BigQuery CDC** | **Keep as concrete example, not universal timing advice.** | | Native recursion protection can limit damage but is provider/path-specific and therefore cannot replace semantic loop prevention. | AWS Lambda, **Use Lambda recursive loop detection to prevent infinite loops**, “Understanding,” “Supported AWS services,” and “Responding.” [15](https://docs.aws.amazon.com/lambda/latest/dg/invocation-recursion.html) URL: `https://docs.aws.amazon.com/lambda/latest/dg/invocation-recursion.html` | Undated live documentation | 2026-10-10 | Provider documentation | **Supported/conditional** | **Keep with limitation.** “Approximately 16” is an emergency fuse, not a design target. | | Retries can amplify overload; backoff, jitter and centralized retry ownership reduce retry storms. | Marc Brooker/AWS, **Timeouts, retries, and backoff with jitter**. [9](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/) URL: `https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter` | Published 2026-06-12; modified 2026-06-15 | 2026-10-10 | Attributable primary company engineering guidance, not controlled empirical study | **Supported as reported practice/design guidance** | **Keep but label the 243× example as illustrative, not measured general effect.** | | Kafka exactly-once is strong inside its transactional boundary; writes to external systems need destination cooperation or coordinated state. | Apache Kafka 4.3, **Design → Message Delivery Semantics / Using Transactions**. [11](https://kafka.apache.org/43/design/design/) URL: `https://kafka.apache.org/43/design/design/` | Docs last modified 2026-05-22 | 2026-10-10 | Official open-source project documentation | **Supported** | **Keep.** Excellent “exactly once where?” evidence. | | “AI triage should not authorize payment/message/production writes” improves safety in this process. | No primary comparative evidence located for this exact bounded workflow. | — | 2026-10-10 | Editorial risk-control design | **Unresolved empirically** | **Keep only as explicit recommendation/hypothesis, not evidence-backed outcome claim.** | | A custom deterministic system will have lower total lifecycle cost than manual/native operation at a particular volume. | No appropriate primary quantitative evidence located. | — | 2026-10-10 | Evidence gap | **Unresolved** | **Do not claim.** Use a local cost model and measured exception data. | An original source on sagas/compensating transactions was identified as Garcia-Molina and Salem, *Sagas*, DOI `https://doi.org/10.1145/38713.38742`, 1987. The Princeton-hosted PDF fetch was inaccessible during this research session; I therefore did **not** rely on its detailed contents to establish any claim. The DOI/ACM metadata was accessible. [16](https://dl.acm.org/doi/10.1145/38713.38742?utm_source=chatgpt.com) ## Contradictions, unresolved gaps, article angles, and freshness triggers There is no fundamental contradiction between provider claims of exactly-once behavior and the recommendation to retain application-level business controls. There **is a terminology trap**. SQS FIFO uses exactly-once language around duplicate messages within its deduplication mechanism. [5](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html) Pub/Sub defines exactly-once delivery for a particular subscription mode and region while separately warning about publish-side duplicates. [6](https://cloud.google.com/pubsub/docs/exactly-once-delivery) Kafka can atomically coordinate offsets and output records inside Kafka, then explicitly requires external-system cooperation outside that boundary. [11](https://kafka.apache.org/43/design/design/) BigQuery can reject duplicate appends by offset within a committed write stream, while saying nothing about whether a payment API called by the same application executed twice. [7](https://cloud.google.com/bigquery/docs/write-api) So the defensible article language is not: > “Exactly once is impossible.” Nor: > “Exactly once solves duplicates.” It is: > **Exactly-once claims have a boundary. Name the boundary before using the claim to reason about a business outcome.** A second apparent tension is **fast acknowledgment versus durable acceptance**. Stripe and Shopify want webhook receivers to respond quickly rather than perform complex synchronous processing. [2](https://docs.stripe.com/webhooks) That should not be rewritten into “return 200 before preserving the event.” The stronger editorial design is to make the synchronous path tiny: authenticate, durably record accepted work, return success, process asynchronously. This is an inference from the combination of provider delivery semantics and the end-to-end argument rather than an exact vendor-prescribed transaction architecture. [1](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf) A third tension is **ordering versus throughput/availability**. Ordering is useful, but Pub/Sub documents restrictions and latency implications, while exactly-once plus ordered acknowledgment adds further throughput constraints. [6](https://cloud.google.com/pubsub/docs/exactly-once-delivery) Ordering should therefore not become an expensive substitute for a source revision when the business source already knows which state is newer. The largest unresolved evidence gaps are empirical rather than semantic. Public primary documentation is strong on provider behavior; it is weak on the questions leaders ultimately care about: how many duplicate charges a specific architecture prevents, what reconciliation costs a five-person company, when a DLQ pays for itself, or whether an AI triage assistant reduces mean recovery time without introducing dangerous advice. Those values should be measured locally rather than invented. The recommendation also weakens or reverses under several conditions. A single transactional local database update does not warrant a distributed recovery framework. Best-effort analytics may rationally tolerate loss. A mature native integration may be safer and cheaper than custom glue. Very latency-sensitive applications can prefer loss over delayed retries, as Saltzer et al.'s real-time voice example illustrates. [1](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf) And inside a single transactional Kafka boundary, built-in exactly-once semantics can replace a significant amount of custom deduplication logic. [11](https://kafka.apache.org/43/design/design/) **Candidate distinct article angles:** | Angle | Core reader payoff | |---|---| | **The workflow is green. The customer was charged twice.** | Makes the central distinction between execution telemetry and business correctness concrete. | | **Exactly once where?** | Explains SQS, Pub/Sub, Kafka and BigQuery as different scoped guarantees rather than arguing over the phrase abstractly. | | **The four identities an automation needs** | Delivery ID, source event ID, business entity ID and semantic operation ID; add source revision as the ordering dimension. | | **Cancellation arrived before confirmation** | A practical explanation of source revision versus arrival order, stale events and reconciliation. | | **Two automations are triggering each other** | Meaningful transitions, no-op suppression, causation metadata and an emergency hop fuse. | | **The API is rate-limiting us: what waits and what must survive?** | Frames backpressure through business deadline and loss tolerance rather than retry settings. | | **Retrying a failed payment may be the thing that duplicates it** | Timeout-after-side-effect ambiguity, status lookup, idempotency lifetime and compensation. | | **Where AI helps recovery—and where I would not let it click** | Deterministic correctness path plus bounded AI exception triage. | A useful compact editorial asset is the following “recovery sentence” template: ```text When [failure] happens, we preserve [durable evidence], decide current truth using [business identity + source revision], retry only [safe operation], verify success by observing [business outcome], and if that cannot be established by [deadline], [owner] performs [reconcile / compensate / quarantine / manual repair]. ``` That template forces an explanation such as: > When a refund request times out, we keep the refund intent and its idempotency identity, do not infer failure from the timeout, query or safely repeat the same operation, and close it only after the payment provider and order ledger reconcile. If the result remains ambiguous after the safe automated path, Order Operations owns manual reconciliation. That is substantially more informative than “the workflow retries three times.” **Freshness triggers for this dossier** should be provider-contract changes rather than an arbitrary calendar interval. Recheck the evidence when Stripe changes idempotency-key semantics or webhook retry behavior; Google changes Pub/Sub exactly-once/ordering, Cloud Tasks retention/retry behavior, or BigQuery Storage Write/CDC semantics; AWS changes SQS deduplication or Lambda recursion handling; Kafka changes transaction/exactly-once behavior; or the bounded business process gains a new irreversible side effect. The reviewed Google Pub/Sub pages were updated as recently as 2026-10-09 and the BigQuery/Cloud Tasks pages on 2026-10-07, one to three days before this snapshot, underscoring that provider semantics deserve version-aware maintenance. [6](https://cloud.google.com/pubsub/docs/exactly-once-delivery) **Bottom-line decision:** explain recovery in the vocabulary of **“Which real-world state is true? What irreversible action occurred? What remains uncertain? Who repairs it, and by when?”** Treat retries, idempotency keys, queues, exactly-once modes and green workflow runs as mechanisms and evidence underneath that contract—not as the contract itself. This conclusion is supported both by the foundational end-to-end systems argument and by the sharply bounded guarantees documented by current webhook, queue, streaming and warehouse providers. [1](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf) ## Portable citation index 1. [web.mit.edu](https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf) 2. [docs.stripe.com](https://docs.stripe.com/webhooks) 4. [docs.stripe.com](https://docs.stripe.com/api/idempotent_requests) 3. [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110.html) 5. [Exactly-once processing in Amazon SQS - Amazon Simple Queue Service](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html) 15. [Use Lambda recursive loop detection to prevent infinite loops - AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/invocation-recursion.html) 6. [Exactly-once delivery  |  Pub/Sub  |  Google Cloud Documentation](https://cloud.google.com/pubsub/docs/exactly-once-delivery) 7. [Introduction to the Storage Write API (gRPC)  |  BigQuery  |  Google Cloud Documentation](https://cloud.google.com/bigquery/docs/write-api) 10. [Understand Cloud Tasks  |  Google Cloud Documentation](https://cloud.google.com/tasks/docs/dual-overview) 12. [Order messages  |  Pub/Sub  |  Google Cloud Documentation](https://cloud.google.com/pubsub/docs/ordering) 13. [Stream table updates with change data capture ingestion  |  BigQuery  |  Google Cloud Documentation](https://cloud.google.com/bigquery/docs/change-data-capture) 8. [Verify webhook deliveries](https://shopify.dev/docs/apps/build/webhooks/subscribe/https) 9. [Timeouts, retries, and backoff with jitter | AWS Builder Center](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/) 11. [Design | Apache Kafka](https://kafka.apache.org/43/design/design/) 14. [Configure queue routing, limits, and retries  |  Cloud Tasks  |  Google Cloud Documentation](https://docs.cloud.google.com/tasks/docs/configuring-queues) 16. [Sagas | Proceedings of the 1987 ACM SIGMOD ...](https://dl.acm.org/doi/10.1145/38713.38742?utm_source=chatgpt.com) ## Import verification and asset availability Genuine Deep Research completed in 8 minutes and was captured on 2026-10-10 after the first due time. Original export, exact independent prompt, canonical chat and citation URLs are retained. Complete inline examples/code were preserved; no missing sandbox download target is claimed as a delivered asset. Report code, test outcomes and execution assertions were not independently run or verified during this import. Delivery is not independent claim verification; Evidence status remains Captured.