# Retries, Duplicates and Idempotency: Recovery Design as a Business Outcome **Practitioner evidence dossier · Research snapshot: October 10, 2026 · Scope: event/webhook-driven order workflow → analytical warehouse → downstream automation** ## Executive answer and uncertainties ### Decision **Yes: recovery should be explained and designed around concrete business outcomes, not a green workflow run.** A green execution can mean little more than “this invocation returned successfully under the platform’s semantics.” Google’s Firebase documentation gives an unusually direct example: under one retry configuration, a background function can report successful execution and show `200 OK` in logs even though the function encountered an error. Stripe, meanwhile, explicitly tells webhook handlers to return `2xx` **before** doing complex downstream work such as updating an accounting system. Google Pub/Sub’s exactly-once feature is scoped to particular subscription and regional conditions and can still expose multiple copies arising from publish-side duplicates. Taken together, these documented behaviors make “run succeeded” an inadequate proxy for “the business process is correct.” [1](https://firebase.google.com/docs/functions/retries) For the bounded order process in this dossier, the stronger completion test is: > **The authoritative order state is represented at the newest accepted source revision, every required side effect has a known terminal disposition, prohibited or duplicate side effects did not occur, and any unresolved accepted work is visible to an owner who can repair it.** That definition is an **editorial operational design**, not a provider guarantee. It deliberately connects transport mechanics to observable business evidence. The important distinction is therefore not “at-least-once versus exactly-once” in isolation. It is between five separate identities and promises: | Layer | Question it answers | Example | |---|---|---| | Transport delivery | “Have I seen this delivery attempt?” | `delivery_id=del-17` | | Producer event | “Have I seen this event occurrence?” | `source=shop`, `event_id=evt-91` | | Business entity | “Which thing changed?” | `order_id=ORD-42` | | Source revision | “Which version of that thing is authoritative?” | `revision=3` | | Operation intent | “Which external business action should happen at most once?” | `notify_cancel:ORD-42:r3` | This separation is supported by CloudEvents’ distinction between an event’s `source + id` and its `subject`, and by Debezium’s outbox model, which separately carries a unique event ID and an aggregate/business identity. CloudEvents says consumers may regard identical `source + id` pairs as duplicate events; it does **not** say that event identity is the same thing as business-object identity. [2](https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md) ### What the evidence supports strongly Retries are appropriate mainly for transient failures, and retries without an idempotent contract can reproduce side effects. AWS explicitly describes retries for network, server and rate-limit faults while warning that the retry approach assumes the operation can be repeated without unintended side effects; its concrete example is accidentally creating two EBS volumes. AWS Prescriptive Guidance likewise recommends backoff for temporary unavailability and `429` throttling and says retrying non-idempotent operations risks corrupting state. [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) Duplicate protection at the message layer is not sufficient for duplicate protection at the business layer. Stripe says webhook deliveries may be duplicated, recommends tracking event IDs, and additionally notes that distinct Event objects can sometimes describe the same underlying object/event type. Its webhook documentation also explicitly says event delivery order is not guaranteed and warns against using the event `created` timestamp to establish ordering. [4](https://docs.stripe.com/webhooks) An “exactly-once” transport feature is a bounded provider contract, not proof of an exactly-once business side effect. Google Pub/Sub limits exactly-once delivery to pull subscriptions and a single-region subscriber condition, notes significantly higher latency, and says publish-side duplication can still produce multiple message copies. AWS SQS, by contrast, describes FIFO queues as providing “exactly-once message processing”; the differing scopes themselves are a reason not to translate a queue-level label into a claim that a payment, message, shipment, or warehouse update occurred exactly once. [5](https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery) ### Important uncertainty The evidence base here is mainly **primary provider documentation, standards work, and attributable provider engineering practice**, because the question concerns contractual delivery and API semantics. This research did **not** identify or rely on a credible controlled empirical study establishing an effect size such as “idempotency lowers business incidents by X%.” No such effect size should be inferred from this dossier. No real commerce platform, warehouse, payment processor, message provider, or customer system was accessed. The implementation below is a **synthetic local SQLite execution**, not a production benchmark or customer result. The IETF `Idempotency-Key` work is useful conceptually but should not be treated as a finished standard in this snapshot. Draft `-07` was published October 15, 2025 and states that it expired April 18, 2026; the draft itself says Internet-Drafts are works in progress. Its useful model is that a client-generated key lets a resource recognize retries, that keys must not be reused for a different payload, and that expiry semantics belong to the resource’s documented contract. [6](https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-idempotency-key-header) No material Czech-versus-US difference appears in the technical retry mechanisms examined. The more relevant distinction in the inspected evidence is provider/product configuration—for example, Google’s exactly-once regional boundary—rather than whether the operating company is Czech or American. Any legal, retention, or sector-specific treatment of stored event payloads is outside this dossier’s evidence scope. [5](https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery) ## Bounded process and automation choices ### The recognizable process The process is intentionally narrow: **A small commerce team receives order events from a source application, reflects the latest valid order state in a cloud analytical warehouse, and triggers a small number of downstream business automations.** The synthetic source produces `CREATED`, `CONFIRMED`, and `CANCELLED` events. Events can be retried, replayed, duplicated under a different delivery attempt, or delivered out of order. Stripe documents all of the relevant provider-side phenomena in a real webhook context: automatic retry, manual resend, duplicate events, and lack of guaranteed event ordering. [4](https://docs.stripe.com/webhooks) The proposed process contract is: | Element | Bounded definition | |---|---| | Trigger | Signed webhook/event from the order source | | Core inputs | `source`, `delivery_id`, `event_id`, `order_id`, `source_revision`, `intent`, payload, occurrence time | | Recovery metadata | `operation_key`, `origin`, `causation_id`, optional `hop_count`, processing disposition | | Business owner | Commerce/operations owner responsible for the correctness of the order outcome | | Technical owner | Data/platform owner responsible for durable ingestion, retries, reconciliation and audit evidence | | Downstream state | Warehouse order projection plus durable side-effect intentions | | Possible side effects | Notification, CRM/ERP update, fulfilment instruction, payment operation where applicable | | Observable completion | Latest authoritative source revision represented; required side effects verified; no unresolved accepted work beyond the operating deadline | | Manual fallback | Place order automation on hold; retrieve current source state; compare revision/audit trail; repair warehouse; reconcile external actions; compensate where necessary; record resolution | The ownership division above is **recommended operating design**, not documented vendor practice. ### State rather than arrival order A useful synthetic transition sequence is: ```text source rev 1: CREATED source rev 2: CONFIRMED source rev 3: CANCELLED ``` Now deliberately deliver it as: ```text arrives first: rev 1 CREATED arrives second: rev 3 CANCELLED arrives third: rev 2 CONFIRMED ``` The correct final state is `CANCELLED @ revision 3`. The late revision-2 confirmation is stale and must not resurrect the order. That policy assumes the source provides a trustworthy monotonically comparable revision. Where it does not, a consumer should not manufacture certainty from arrival time. Stripe expressly says not to use its event `created` timestamp to determine order and recommends retrieving missing/current objects instead. [4](https://docs.stripe.com/webhooks) “Cancellation wins forever” is **not** a universal rule. If the domain permits a legitimate `REOPENED @ revision 4`, revision 4 should supersede revision 3. The invariant is **newest authoritative business revision**, not a hard-coded preference for one status. ### Three viable operating models | Model | How it works | Recovery characteristics | Where I would use it | |---|---|---|---| | **Manual/native baseline** | Source UI remains authoritative; operator reviews current order and performs or approves downstream updates; simple native export/connector refreshes analytics | Humans naturally reconcile context; few distributed failure modes, but slower and labor-intensive; audit quality depends on process discipline | Low volume, rare/high-consequence actions, unstable process | | **Deterministic automation** | Durable inbox → revision guard → warehouse projection + transactional outbox → idempotent/reconciled worker → audit/reconciliation | More engineering, but failure states become explicit and repeatable | Default choice once routine volume or latency justifies it | | **Bounded AI triage assistant** | Reads dead-letter/reconciliation cases and audit evidence, groups likely cause, drafts repair steps; deterministic system or human executes | Can reduce cognitive work on messy exceptions but adds model/evaluation/configuration upkeep | Only after deterministic state and recovery evidence exist | The AI variant in this design **cannot authorize payments, send customer messages, or write production state**. It does not decide the authoritative revision. It may summarize, classify and propose a repair; the deterministic rules and human operator remain the execution boundary. That is not merely conservatism. Idempotency correctness rests on stable identity, atomicity and business semantics. AWS’s account of idempotent APIs emphasizes that the idempotency token and related mutations need a sound atomic relationship, while transactional-outbox guidance exists specifically because a database mutation and message publication can otherwise split into inconsistent outcomes. These are poor places to substitute probabilistic inference for deterministic constraints. [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) ### Lifecycle burden There is no defensible generic market benchmark for the maintenance hours of this design in the sources examined. The following is therefore an **explicit synthetic planning model**, useful for deciding whether automation is worth owning. Let: - \(V\) = routine events per month - \(m\) = manual minutes per routine event - \(B_d\) = deterministic automation build hours - \(U_d\) = deterministic monthly upkeep hours - \(B_a\) = incremental AI-assistant build hours - \(U_a\) = incremental AI monthly upkeep hours - \(R\) = residual annual exception/recovery hours Then: \[ H_{manual} = \frac{12Vm}{60} \] \[ H_{det,Y1} = B_d + 12U_d + R \] \[ H_{AI,Y1} = B_d + B_a + 12(U_d + U_a) + R_{AI} \] **Illustrative assumptions only, not benchmarks:** \(V=500\), \(m=2\) minutes, \(B_d=40\) h, \(U_d=3\) h/month, \(B_a=12\) h, \(U_a=1.5\) h/month. That gives: - manual routine floor = \(500×12×2/60 = 200\) h/year; - deterministic floor before residual exception work = \(40 + 12×3 = 76\) h in year one; - deterministic + AI floor before residual exception work = \(40+12+12×(3+1.5)=106\) h in year one. On these assumptions, adding AI consumes another **30 h in year one**, so it only makes economic sense if its reduction in triage effort or other benefit exceeds that incremental burden. Those figures are a sensitivity example, not evidence about a real implementation. Maintenance should explicitly budget for source schema/revision changes, API-version changes, retry or idempotency-contract changes, new downstream actions, reconciliation drift, operating documentation and eventual retirement. Stripe’s current API documentation alone demonstrates why provider contract drift matters: its documented v1 and v2 idempotency semantics and retention windows differ. [7](https://docs.stripe.com/api-v2-overview) ## Failure and recovery evidence ### Retry safety is not the same thing as idempotency These terms are often collapsed into one bucket, which is exactly how duplicate side effects escape into production. | Mechanism | What it actually buys | What it does **not** buy | |---|---|---| | Retry | Another chance after a transient failure | Safety from duplicate effects | | Idempotent operation | Repetition produces the same intended state/effect under its contract | Infinite dedupe lifetime | | Idempotency key | Links repeated calls to one intended operation within provider rules | Business-state ordering; protection after key expiry | | Transport dedupe | Suppresses a repeated message/event identity | Two distinct events representing the same business intent | | Business-operation dedupe | Suppresses repeated semantic action such as `cancel-notice:order42:r3` | Correct source ordering by itself | | Exactly-once broker feature | Stronger delivery/processing semantics inside a defined product boundary | End-to-end exactly-once payments/messages/shipments | | Compensation | Repairs a business effect that already happened | Erasure of the historical fact that it happened | AWS’s retry guidance explicitly says transient errors such as throttling, temporary network faults and temporary service unavailability are appropriate retry candidates and warns that operations used with retry should be idempotent. Its Builders Library makes the same point using side-effect duplication as the failure case. [8](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/retry-backoff.html) The current Stripe API illustrates why an idempotency key also has a **lifetime and version scope**. Its `/v2` documentation says same-key requests qualify as idempotent replays within 30 days under the documented account/API conditions, while it describes API v1 replay as a 24-hour window. It also documents differing replay behavior after failures between v1 and v2. [7](https://docs.stripe.com/api-v2-overview) That matters operationally because Stripe’s webhook documentation separately permits CLI resends for events as old as 30 days. A system that can replay 30-day-old input but relies exclusively on an outbound API’s shorter deduplication window can reopen duplicate-side-effect risk. This is an **inference across two Stripe contracts**, not a claim that every Stripe integration necessarily has that bug. [4](https://docs.stripe.com/webhooks) The practical rule is therefore: > **Set local business-operation retention from the maximum replay/reconciliation horizon, not merely from the provider’s shortest convenient cache window.** For long-lived financially or operationally meaningful actions, retaining the durable business operation identity with the order can be safer than treating it as a disposable transport deduplication record. This is editorial design supported by documented finite provider idempotency windows and AWS’s discussion of retaining idempotency knowledge according to resource/request lifetime. [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) ### Durable acceptance closes a dangerous gap Stripe tells webhook consumers to verify the request and return `2xx` quickly, before complex downstream processing. It retries live webhook delivery for up to three days with exponential backoff when delivery fails. [4](https://docs.stripe.com/webhooks) Those instructions should **not** be interpreted as “acknowledge while the only copy is still in process memory.” The smallest robust interpretation for this process is: ```text receive → authenticate/validate enough to accept → durably insert minimal inbox envelope → return 2xx → process asynchronously ``` That ordering is an **editorial inference**, not a Stripe-prescribed architecture. It follows the same principle Google documents for exactly-once processing—maintain progress information until acknowledgement—and the same atomicity motivation behind the transactional-outbox pattern. [5](https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery) The resulting business distinction is useful: ```text HTTP received ≠ durable accepted durable accepted ≠ business state applied state applied ≠ external action completed external API returned ≠ business outcome necessarily verified workflow green ≠ process reconciled ``` ### The local transaction boundary should be explicit Suppose a handler updates `orders` and then publishes “order cancelled” as an unrelated second write. ```text UPDATE orders ... COMMIT publish notification intent ... CRASH ``` The order is cancelled locally but the downstream automation never learns about it. Reverse the order: ```text publish notification intent UPDATE orders ... CRASH ``` and the downstream action can happen for a business-state change that never committed. AWS describes this as the dual-write problem and presents the transactional outbox specifically to bind the database mutation and notification intent together. It also warns that the outbox publisher can itself emit duplicates, so downstream consumers still need idempotency. [9](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html) Debezium’s documented outbox structure reinforces the identity split: its default model includes a unique `id` that can be used for duplicate removal and a separate `aggregateid` for the business aggregate. [10](https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html) For a small team, the smallest workable transaction is therefore: ```text BEGIN claim source event compare source revision update local business projection insert semantic side-effect intent into outbox COMMIT ``` A separate worker may deliver the outbox repeatedly, because repetition is made safe at the **business operation** boundary. ### Timeout after an external side effect This is the classic dangerous sequence: ```text T0 worker sends "capture payment for order 42" T1 provider commits payment T2 response packet is lost T3 local worker sees timeout ``` At `T3`, “failed” is not a fact. The result is **unknown**. Blindly issuing a new payment operation can duplicate the charge. Reusing the provider-supported idempotency identity is safer within its documented contract; Stripe explicitly documents idempotency as protection against unintended repeated side effects. [7](https://docs.stripe.com/api-v2-overview) Where the external system supports lookup by a stable business/request reference, the recovery order should be: ```text 1. query/reconcile the previous operation 2. if completion is proven → record success locally 3. if non-completion is proven → retry 4. if outcome remains ambiguous → escalate rather than guessing ``` AWS’s idempotent-API discussion explains why semantically equivalent responses are valuable after a response is lost and why the request identifier must be stored atomically with the operation. [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) If the duplicate payment has already happened, idempotency no longer repairs history. The remedy becomes a **compensating business action**, such as refunding/reversing the extra transaction where the domain permits it. Saga guidance explicitly distinguishes forward recovery by retry from backward recovery through compensating transactions. [11](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/saga-patterns.html) The same distinction applies to messages. A second email cannot be “rolled back” after somebody has received it. The system can prevent another send, record the incident, or issue a corrective communication; compensation semantics are business-specific. ### Out-of-order events need source meaning, not queue optimism Stripe explicitly states that it does not guarantee webhook event delivery order and says not to infer ordering from its event timestamp. [4](https://docs.stripe.com/webhooks) For our synthetic order: ```text 10:00:00 order r1 CREATED 10:00:02 order r3 CANCELLED ← delivered now 10:00:05 order r2 CONFIRMED ← late ``` Safe state transition: ```text current r1 receive r3 → apply CANCELLED, current=r3 receive r2 → classify stale; preserve r3 ``` An ordinary `UPSERT` can provide the uniqueness/update primitive, but a correct implementation must add the revision predicate. SQLite documents that UPSERT can turn a uniqueness conflict into an update or no-op and can condition whether the update occurs; that is the mechanism used in the synthetic harness below. [12](https://www.sqlite.org/lang_upsert.html) If no trustworthy source revision/version exists, the safe repair criterion changes: fetch the authoritative current object from the source and reconcile local state rather than pretending local arrival order is authoritative. Stripe itself recommends retrieving missing objects when event order cannot be relied upon. [4](https://docs.stripe.com/webhooks) ### Rate limits are a queueing and deadline problem A `429` should usually mean **wait**, not “lose the business action” and not “hammer harder.” AWS’s retry guidance treats throttling as a transient condition suited to backoff and cautions that frequent retry can worsen service degradation. [8](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/retry-backoff.html) Google Pub/Sub’s flow-control documentation makes the backpressure problem concrete: when publish requests outpace client capacity, requests can accumulate in memory until they fail with `DEADLINE_EXCEEDED`; it recommends bounding outstanding messages/bytes and blocking or otherwise controlling publication pressure. [13](https://cloud.google.com/pubsub/docs/flow-control-messages) For business automation, each queued intention therefore needs two separate properties: **Durability:** should the work survive waiting? **Deadline:** how long is completing it still useful? For example, in this synthetic policy: ```text warehouse analytics refresh durable, can wait hours customer cancellation notice durable, high priority, escalate after 15 min fraud-review request durable, explicit business deadline obsolete "order confirmed" supersede if newer cancellation arrives first ``` The exact deadlines are business policy, not evidence from providers. Backpressure then means “accept no more active work than the downstream capacity can safely service while preserving already accepted work,” rather than using unbounded in-memory accumulation as an accidental queue. Google’s documented memory accumulation/failure mode is a concrete reason for that distinction. [13](https://cloud.google.com/pubsub/docs/flow-control-messages) ### Two automations triggering each other Consider: ```text Automation A: CRM status changed → update warehouse flag Automation B: warehouse flag changed → update CRM status A write from A wakes B. B writes back and wakes A. A wakes B... ``` Retries are not the correct cure. Retrying can make a permanent loop more persistent; Firebase explicitly warns that permanent bugs with retry enabled can remain in retry loops and recommends a defined termination condition. [1](https://firebase.google.com/docs/functions/retries) The smallest workable guard is a combination of: ```text one authoritative writer per business field where possible + origin/workflow identity + causation/correlation identity + "already handled this causation" ledger + optional hop/deadline ceiling + only emit when a meaningful value actually changed ``` CloudEvents formally provides event `source` and `id`, and permits extension attributes for additional correlation/identity metadata. `causation_id` and `hop_count` in this dossier are **editorial extensions**, not CloudEvents-standard attributes. [2](https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md) A hop ceiling is only a fuse. The actual fix is usually to remove the circular ownership relationship or make the second write non-triggering/semantically idempotent. ## Worked recovery cases and observable outcomes The timelines below are **synthetic fixtures reasoned from the documented behaviors above**. They are not reports of production incidents. ### Duplicate webhook ```text 09:00:00 delivery D1, event E7, order O42, revision 1 CREATED 09:00:01 apply O42@r1 09:00:03 delivery D2, event E7, order O42, revision 1 CREATED ``` Transport/event dedupe sees the same producer event and records the second arrival without applying it again. CloudEvents explicitly permits a retried duplicate to use the same event `id`; Stripe similarly recommends event-ID tracking for duplicated deliveries. [2](https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md) **Safe repair criterion:** one state application, one operation intention, both delivery attempts auditable. ### Two distinct events with the same business intent ```text E7 order O42 r3 CANCELLED E99 order O42 r3 CANCELLED ``` Event-ID dedupe alone sees two different events. The revision/state guard and semantic operation key: ```text notify_cancel:O42:r3 ``` still protect the downstream business action. This matters because Stripe explicitly documents cases where two separate Event objects can correspond to the same underlying object/event type; its suggested duplicate handling therefore goes beyond merely storing one event ID. [4](https://docs.stripe.com/webhooks) **Safe repair criterion:** both source events remain auditable, but only one meaningful cancellation transition and one cancellation-notification intent exist. ### Cancellation overtakes confirmation ```text received business version -------------------------------- CREATED r1 CANCELLED r3 CONFIRMED r2 ← late ``` **Safe state:** ```text O42 = CANCELLED @ r3 r2 disposition = stale ``` Stripe’s documented absence of delivery-order guarantees is the real-provider analogue. [4](https://docs.stripe.com/webhooks) **Safe repair criterion:** warehouse/source reconciliation agrees on revision 3; no downstream confirmation generated from stale revision 2. ### Acknowledged but work was lost Unsafe design: ```text receive webhook return 200 enqueue only in process memory process crashes ``` Synthetic outcome: source thinks delivery was acknowledged while no durable recovery record exists. Proposed safe boundary: ```text verify BEGIN INSERT durable inbox envelope COMMIT return 2xx ``` Then the heavier state transition occurs asynchronously. Stripe requires quick responses before complex logic, while Google’s exactly-once guidance explicitly tells consumers to preserve processing-progress information until acknowledgement; the durable-inbox sequencing is this dossier’s synthesis of those constraints. [4](https://docs.stripe.com/webhooks) **Safe repair criterion:** every acknowledged accepted event can be found in durable storage with a terminal or pending disposition. ### Timeout after payment/message side effect ```text T0 send operation with key K T1 provider performs operation K T2 response lost T3 local status = UNKNOWN, not FAILED T4 reconcile K / repeat safely under provider idempotency contract ``` Provider idempotency is the strongest option where supported. Stripe documents same-key protection with explicit API-version-specific retention semantics. [7](https://docs.stripe.com/api-v2-overview) **Safe repair criterion:** exactly one externally verified semantic operation is associated with K. If two exist, mark incident and use a defined compensation procedure rather than retrying again. ### Quota exhaustion ```text T0 cancellation action accepted durably T1 provider → 429 T2 action stays queued T3 exponential backoff + jitter T4 retry succeeds ``` The relevant success is **not** “the T1 workflow failed cleanly.” It is “the cancellation action remained durable during throttling and either completed before its business deadline or escalated visibly.” Backoff for rate-limit/transient failures and bounded flow control are directly supported by AWS and Google guidance. [8](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/retry-backoff.html) ### Feedback loop ```text cause C7 enters A A emits change tagged cause C7 B processes C7 and emits derived change tagged cause C7 A sees C7 again A's seen-causation guard blocks another execution ``` A defined end condition is consistent with Firebase’s retry-loop guidance; carrying the causation metadata is an editorial implementation using CloudEvents-compatible extension metadata. [1](https://firebase.google.com/docs/functions/retries) **Safe repair criterion:** loop terminates deterministically, legitimate unrelated changes still pass, and the audit trail shows which workflow stopped the repeated causation. ### What the dashboard should report A useful operational dashboard is outcome-oriented: | Signal | Why it is closer to business truth | |---|---| | `source_latest_revision - warehouse_latest_revision` | Detects stale analytical/business state | | Accepted events with no terminal disposition | Finds acknowledged-but-unfinished work | | Outbox age by business deadline | Shows work at risk even when workers are “healthy” | | Unknown external side effects | Exposes timeout-after-commit ambiguity | | Duplicate operation-key conflicts | Detects repeated business intent | | Dead-letter count and oldest age | Shows recovery obligations | | Reconciliation mismatches | Compares automation state to authoritative/external reality | | Compensation outstanding | Shows business effects not yet repaired | | Loop-block events | Shows circular automation rather than hiding it as successful runs | This dashboard is a **recommended evidence model**, not a provider feature. An especially useful invariant is: ```text accepted event → applied | duplicate | stale | superseded | dead-lettered | manually resolved ``` There should be no invisible eighth state called “probably fine.” ## Smallest workable implementation and recovery runbook ### Minimal architecture For a small team, I would not begin with a globally ordered event platform, distributed saga engine, or autonomous agent. The smallest implementation I would trust for this process is: ```text Webhook │ ▼ durable inbox / event claim │ ├── duplicate event → audit + stop │ ▼ source revision guard │ ├── stale → audit + stop │ ▼ atomic transaction ├── latest order projection └── durable outbox operation │ ▼ bounded worker retry/backoff reconcile unknown result │ ┌───────┴─────────┐ ▼ ▼ external verified dead letter │ │ ▼ ▼ DONE human recovery ``` The local state update and outbox write are one transaction because distributed dual writes can otherwise diverge; AWS explicitly documents this failure and also notes that duplicate outbox delivery remains possible, hence the idempotent consumer/operation boundary. [9](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html) ### Recovery runbook **For a duplicate:** establish whether it is the same transport delivery, the same producer event, or a distinct event expressing the same business revision/intent. Do not collapse those identities into one column. CloudEvents and Debezium both explicitly separate event IDs from broader source/aggregate identity. [2](https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md) **For stale/out-of-order state:** compare the source revision with current applied revision. If the source contract does not provide a safe revision, retrieve current source state before repair; do not sort arbitrary webhook timestamps and call that correctness. Stripe explicitly cautions against timestamp-based event ordering. [4](https://docs.stripe.com/webhooks) **For timeout after side effect:** mark result `UNKNOWN`; first reconcile using the external idempotency/business reference. Retry only when the operation’s contract makes the retry safe or non-completion is established. AWS’s idempotent API guidance describes this lost-response problem directly. [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) **For rate limiting:** retain the work, honor the provider’s documented throttling behavior where available, back off with jitter, enforce a business deadline, and escalate instead of silently dropping expired high-value work. AWS documents throttling as a retry/backoff use case; Google documents bounded flow control as protection from request accumulation. [8](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/retry-backoff.html) **For permanent failure:** stop automated retries and dead-letter it with reason, payload reference, operation key, owner and repair instructions. Firebase explicitly distinguishes transient retry candidates from permanent bugs/errors and recommends end conditions. [1](https://firebase.google.com/docs/functions/retries) **For partial completion:** decide between forward recovery and compensation based on business semantics. AWS saga guidance formally separates those two modes. [11](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/saga-patterns.html) **For feedback loops:** pause the offending route if necessary, identify the shared causation chain, determine the authoritative writer, then eliminate or suppress the circular transition. A retry limit alone only limits damage; it does not fix ownership. **For replay:** replay from the durable accepted record, never from an unaudited reconstruction. Preserve the original event identity and business revision while creating a distinct replay attempt identity. Confirm that the external idempotency horizon still covers the operation; Stripe demonstrates that such horizons can be finite and version-dependent. [7](https://docs.stripe.com/api-v2-overview) ### Complete portable Python/SQLite harness **Evidence classification: firsthand synthetic execution.** The following code uses only the Python standard library and SQLite. It does not contact real systems. It was executed during this research session. The executed fixtures verified: ```text initial create → applied same transport delivery replay → duplicate_delivery same source event, new delivery → duplicate_event revision 3 cancellation → applied late revision 2 confirmation → stale timeout after remote side effect → reconciled without second effect 429 → retained and later completed permanent synthetic failure → dead letter A → B → A feedback path → second A blocked final order → CANCELLED @ revision 3 external semantic effects → exactly 2 fixture effects: notify_created:r1 notify_cancel:r3 asserted invariants → PASS ``` This is not evidence that any cloud provider behaves exactly like the simulated `remote_effects` table. It tests the proposed **local recovery invariants**. ```python #!/usr/bin/env python3 """ Synthetic recovery harness: Python 3 standard library + SQLite only. Models: - delivery identity vs producer event identity - source revision ordering - durable event claim - atomic local projection + outbox - timeout after external side effect - reconciliation before retry - synthetic 429 with backoff - dead-letter handling - causation-based feedback-loop protection No network access or real provider systems are used. """ import json import random import sqlite3 DB = ":memory:" rng = random.Random(7) db = sqlite3.connect(DB) db.row_factory = sqlite3.Row db.executescript(""" PRAGMA foreign_keys=ON; CREATE TABLE deliveries( delivery_id TEXT PRIMARY KEY, source TEXT NOT NULL, event_id TEXT NOT NULL, order_id TEXT NOT NULL, revision INTEGER NOT NULL, intent TEXT NOT NULL, disposition TEXT NOT NULL, note TEXT ); CREATE TABLE claimed_events( source TEXT NOT NULL, event_id TEXT NOT NULL, PRIMARY KEY(source,event_id) ); CREATE TABLE orders( order_id TEXT PRIMARY KEY, revision INTEGER NOT NULL, state TEXT NOT NULL, source_event_id TEXT NOT NULL ); CREATE TABLE outbox( op_key TEXT PRIMARY KEY, order_id TEXT NOT NULL, revision INTEGER NOT NULL, action TEXT NOT NULL, status TEXT NOT NULL DEFAULT 'pending', attempts INTEGER NOT NULL DEFAULT 0, next_at REAL NOT NULL DEFAULT 0, deadline_at REAL NOT NULL, external_ref TEXT, last_error TEXT ); -- Simulates a queryable external provider keyed by our business operation key. -- In production this state would be remote, outside the local transaction. CREATE TABLE remote_effects( op_key TEXT PRIMARY KEY, external_ref TEXT NOT NULL ); CREATE TABLE dead_letters( op_key TEXT PRIMARY KEY, reason TEXT NOT NULL ); CREATE TABLE loop_seen( workflow TEXT NOT NULL, causation_id TEXT NOT NULL, PRIMARY KEY(workflow,causation_id) ); CREATE TABLE audit( seq INTEGER PRIMARY KEY AUTOINCREMENT, kind TEXT NOT NULL, ref TEXT NOT NULL, detail TEXT NOT NULL ); """) STATE_FOR = { "CREATED": "CREATED", "CONFIRMED": "CONFIRMED", "CANCELLED": "CANCELLED", } ACTION_FOR = { "CREATED": "notify_created", "CONFIRMED": "analytics_sync", "CANCELLED": "notify_cancel", } def audit(kind, ref, **detail): db.execute( "INSERT INTO audit(kind,ref,detail) VALUES(?,?,?)", (kind, ref, json.dumps(detail, sort_keys=True)), ) def ingest(e): """ Durably accept one delivery, claim its producer event, then apply only a newer authoritative source revision. """ try: with db: db.execute( """ INSERT INTO deliveries( delivery_id,source,event_id,order_id, revision,intent,disposition ) VALUES(?,?,?,?,?,?,?) """, ( e["delivery_id"], e["source"], e["event_id"], e["order_id"], e["revision"], e["intent"], "accepted", ), ) except sqlite3.IntegrityError: audit( "duplicate_delivery", e["delivery_id"], event_id=e["event_id"], ) db.commit() return "duplicate_delivery" with db: cur = db.execute( """ INSERT OR IGNORE INTO claimed_events(source,event_id) VALUES(?,?) """, (e["source"], e["event_id"]), ) if cur.rowcount == 0: db.execute( """ UPDATE deliveries SET disposition=?, note=? WHERE delivery_id=? """, ( "duplicate_event", "same source+event_id", e["delivery_id"], ), ) audit( "duplicate_event", e["delivery_id"], event_id=e["event_id"], ) return "duplicate_event" row = db.execute( """ SELECT revision,state FROM orders WHERE order_id=? """, (e["order_id"],), ).fetchone() if row and e["revision"] <= row["revision"]: same_semantics = ( e["revision"] == row["revision"] and STATE_FOR[e["intent"]] == row["state"] ) disposition = ( "semantic_duplicate" if same_semantics else "stale" ) db.execute( """ UPDATE deliveries SET disposition=?, note=? WHERE delivery_id=? """, ( disposition, f'current_revision={row["revision"]}', e["delivery_id"], ), ) audit( disposition, e["delivery_id"], order_id=e["order_id"], incoming_revision=e["revision"], current_revision=row["revision"], ) return disposition # Local business projection and side-effect intention are atomic. db.execute( """ INSERT INTO orders( order_id,revision,state,source_event_id ) VALUES(?,?,?,?) ON CONFLICT(order_id) DO UPDATE SET revision=excluded.revision, state=excluded.state, source_event_id=excluded.source_event_id """, ( e["order_id"], e["revision"], STATE_FOR[e["intent"]], e["event_id"], ), ) # Pending work from an older source revision may no longer be # meaningful after the newer business state arrives. db.execute( """ UPDATE outbox SET status='superseded', last_error='newer source revision' WHERE order_id=? AND revision r["deadline_at"]: with db: db.execute( """ UPDATE outbox SET status='dead' WHERE op_key=? """, (op_key,), ) db.execute( """ INSERT OR REPLACE INTO dead_letters VALUES(?,?) """, ( op_key, "deadline exceeded", ), ) audit( "dead_letter", op_key, reason="deadline exceeded", ) continue try: ref = remote_call(op_key) with db: db.execute( """ UPDATE outbox SET status='done', attempts=?, external_ref=?, last_error=NULL WHERE op_key=? """, ( attempt, ref, op_key, ), ) audit( "effect_done", op_key, attempt=attempt, external_ref=ref, ) except RateLimit as ex: # Bounded exponential backoff plus deterministic jitter. wait = ( min( 60.0, 2 ** (attempt - 1), ) + rng.random() ) with db: db.execute( """ UPDATE outbox SET status='retry', attempts=?, next_at=?, last_error=? WHERE op_key=? """, ( attempt, now + wait, str(ex), op_key, ), ) audit( "rate_limited", op_key, attempt=attempt, retry_in=round(wait, 3), ) except TimeoutError as ex: with db: db.execute( """ UPDATE outbox SET status='retry', attempts=?, next_at=?, last_error=? WHERE op_key=? """, ( attempt, now + 1.0, str(ex), op_key, ), ) audit( "unknown_result", op_key, attempt=attempt, ) except Permanent as ex: with db: db.execute( """ UPDATE outbox SET status='dead', attempts=?, last_error=? WHERE op_key=? """, ( attempt, str(ex), op_key, ), ) db.execute( """ INSERT OR REPLACE INTO dead_letters VALUES(?,?) """, ( op_key, str(ex), ), ) audit( "dead_letter", op_key, reason=str(ex), ) def feedback_step( workflow, causation_id, origin, hop, max_hops=4, ): """ Return metadata for the next synthetic event, or None when the feedback chain is blocked. """ if hop >= max_hops: audit( "loop_blocked_hop_limit", causation_id, workflow=workflow, hop=hop, ) db.commit() return None try: with db: db.execute( """ INSERT INTO loop_seen( workflow,causation_id ) VALUES(?,?) """, ( workflow, causation_id, ), ) audit( "loop_processed", causation_id, workflow=workflow, origin=origin, hop=hop, ) except sqlite3.IntegrityError: audit( "loop_blocked_seen", causation_id, workflow=workflow, hop=hop, ) db.commit() return None return { "origin": workflow, "causation_id": causation_id, "hop": hop + 1, } # ----------------------------- # Complete synthetic fixtures # ----------------------------- create = dict( delivery_id="del-1", source="shop", event_id="evt-create", order_id="ord-42", revision=1, intent="CREATED", ) # Same producer event, different transport attempt. create_retry = dict( create, delivery_id="del-1b", ) cancel = dict( delivery_id="del-3", source="shop", event_id="evt-cancel", order_id="ord-42", revision=3, intent="CANCELLED", ) late_confirm = dict( delivery_id="del-2", source="shop", event_id="evt-confirm", order_id="ord-42", revision=2, intent="CONFIRMED", ) print( "INGEST", ingest(create), ) create_key = ( "notify_created:ord-42:r1" ) fault_plan[create_key] = [ "timeout_after_commit", "success", ] # Remote effect happens, response is lost. run_worker(0) # Worker reconciles existing remote effect. # It does not produce a second effect. run_worker(2) # Exact same transport delivery. print( "DUPLICATE DELIVERY", ingest(create), ) # Same source event under another delivery. print( "DUPLICATE EVENT", ingest(create_retry), ) # Revision 3 deliberately arrives before revision 2. print( "CANCEL", ingest(cancel), ) print( "LATE CONFIRM", ingest(late_confirm), ) cancel_key = ( "notify_cancel:ord-42:r3" ) fault_plan[cancel_key] = [ "rate_limit", "success", ] # 429: work remains durable. run_worker(10) # Later retry succeeds. run_worker(20) # Explicit poison/dead-letter example. with db: db.execute( """ INSERT INTO outbox( op_key, order_id, revision, action, deadline_at ) VALUES( 'manual_export:ord-99:r1', 'ord-99', 1, 'manual_export', 100 ) """ ) fault_plan[ "manual_export:ord-99:r1" ] = [ "permanent", ] run_worker(30) # A -> B -> A. # The second visit to A under the same causation is blocked. m = feedback_step( "A", "cause-7", "source", 0, ) m = feedback_step( "B", m["causation_id"], m["origin"], m["hop"], ) m = feedback_step( "A", m["causation_id"], m["origin"], m["hop"], ) # ----------------------------- # Observable invariants # ----------------------------- order = dict( db.execute( """ SELECT * FROM orders WHERE order_id='ord-42' """ ).fetchone() ) effects = [ dict(x) for x in db.execute( """ SELECT * FROM remote_effects ORDER BY op_key """ ) ] dead = [ dict(x) for x in db.execute( """ SELECT * FROM dead_letters ORDER BY op_key """ ) ] outbox = [ dict(x) for x in db.execute( """ SELECT op_key, status, attempts, external_ref, last_error FROM outbox ORDER BY op_key """ ) ] loop_blocks = db.execute( """ SELECT count(*) FROM audit WHERE kind LIKE 'loop_blocked%' """ ).fetchone()[0] assert ( order["revision"] == 3 and order["state"] == "CANCELLED" ) assert sum( x["op_key"] == create_key for x in effects ) == 1 assert sum( x["op_key"] == cancel_key for x in effects ) == 1 assert db.execute( """ SELECT status FROM outbox WHERE op_key=? """, (create_key,), ).fetchone()[0] == "done" assert db.execute( """ SELECT status FROM outbox WHERE op_key=? """, (cancel_key,), ).fetchone()[0] == "done" assert len(dead) == 1 assert loop_blocks == 1 print( "FINAL ORDER", order, ) print( "REMOTE EFFECTS", json.dumps( effects, sort_keys=True, ), ) print( "OUTBOX", json.dumps( outbox, sort_keys=True, ), ) print( "DEAD LETTERS", json.dumps( dead, sort_keys=True, ), ) print( "LOOP BLOCKS", loop_blocks, ) print( "INVARIANTS PASS" ) ``` A limitation of this harness is deliberate: its simulated `remote_effects` table makes an external outcome queryable by operation key. Real providers differ. Where a real external system offers neither robust idempotency nor an authoritative way to query whether an ambiguous operation happened, full automatic recovery may be impossible without accepting duplicate risk. In that condition, the recommendation reverses toward serialization, human approval, or a provider/interface change. ## Claim-to-primary-source dossier All sources below are primary provider/standards/project material. “Documented design” means the source documents intended product behavior; it does **not** mean this research independently load-tested the behavior. | Claim / use | Primary source and exact scope | Source date | Inspected | Evidence type | Assessment / editorial advice | |---|---|---:|---:|---|---| | Stripe webhooks are retried, can be manually resent, and manual resend does not stop automatic retry | Stripe, **Webhooks → Event delivery behaviors → Automatic retries / Manual retries**. https://docs.stripe.com/webhooks [4](https://docs.stripe.com/webhooks) | Live doc; no publication date shown | 2026-10-10 | Provider documentation | **Supported. Keep**, but preserve live-vs-sandbox and resend-window scope. | | Stripe webhook delivery order is not guaranteed; event timestamp should not be used as ordering proof | Stripe, **Webhooks → Event ordering**. https://docs.stripe.com/webhooks [4](https://docs.stripe.com/webhooks) | Live doc; no publication date shown | 2026-10-10 | Provider documentation | **Supported. Keep.** Strong support for source-version/reconciliation design. | | Webhook handler should return `2xx` before complex downstream logic | Stripe, **Create a handler**. https://docs.stripe.com/webhooks [4](https://docs.stripe.com/webhooks) | Live doc; no publication date shown | 2026-10-10 | Provider documentation | **Supported, qualify.** Does not itself say to acknowledge before durable persistence. | | Stripe API idempotency semantics and retention differ between v1 and v2 | Stripe, **API v2 overview → Idempotency / Idempotency differences between API v1 and API v2**. https://docs.stripe.com/api-v2-overview [7](https://docs.stripe.com/api-v2-overview) | Live doc; examples show API versions current to doc | 2026-10-10 | Provider contract documentation | **Supported. Keep exact API-version scope.** Do not turn 24 h/30 d into a universal Stripe rule. | | Exactly-once Pub/Sub semantics suppress acknowledged-message redelivery under defined conditions | Google Cloud, **Pub/Sub → Exactly-once delivery**. https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery [5](https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery) | Last updated 2026-10-09 UTC [5](https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery) | 2026-10-10 | Provider documentation | **Supported but highly scoped. Qualify.** | | Pub/Sub exactly-once guarantee is regional for subscribers | Google Cloud, **Exactly-once delivery → Regional considerations**. same URL [5](https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery) | 2026-10-09 | 2026-10-10 | Provider documentation | **Supported. Keep.** Region is a service topology condition, not CZ-vs-US customer nationality. | | Pub/Sub exactly-once applies only to pull subscriptions; publish-side duplicates can still produce multiple copies; latency is higher | Google Cloud, **Exactly-once delivery → Exactly-once delivery and push subscriptions / Things to know**. same URL [5](https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery) | 2026-10-09 | 2026-10-10 | Provider documentation | **Supported. Keep.** Strong counterevidence to end-to-end “exactly once” shorthand. | | SQS documentation distinguishes Standard at-least-once from FIFO “exactly-once message processing” | AWS, **What is Amazon SQS? → Benefits → Durability**. https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html [14](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html) | Live doc; no publication date shown | 2026-10-10 | Provider documentation | **Supported as AWS terminology. Qualify heavily** before relating it to external side effects. | | Retrying transient faults assumes repeated operations are safe from additional side effects | AWS Builders Library, **Making retries safe with idempotent APIs → Retrying and side effects**. https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/ [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) | Live article; publication date not exposed in inspected excerpt | 2026-10-10 | Attributable provider engineering practice | **Supported. Keep.** Not an independent experiment. | | Atomic linkage between idempotency token and mutation prevents token/effect divergence; retention is necessarily finite/domain-specific | AWS Builders Library, **Retries and semantic equivalence / Late arriving requests / Same client request ID, different intent**. same URL [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) | Live article | 2026-10-10 | Attributable provider engineering practice | **Supported as reported Amazon design. Qualify** when generalized. | | Exponential backoff is appropriate for throttling/transient faults; retrying non-idempotent actions is risky | AWS Prescriptive Guidance, **Retry with backoff → Intent / Applicability / Issues**. https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/retry-backoff.html [8](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/retry-backoff.html) | Live guidance | 2026-10-10 | Provider architecture guidance | **Supported. Keep.** | | AWS uses timeouts, retries and backoff/jitter to tolerate transient failures without magnifying overload | AWS Builder Center, **Timeouts, retries, and backoff with jitter**. https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter [15](https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter) | Published 2026-06-12; modified 2026-06-15 | 2026-10-10 | Attributable provider engineering practice | **Supported. Keep**, but do not claim measured universal impact. | | Transactional outbox addresses database+message dual-write inconsistency; duplicates remain possible | AWS Prescriptive Guidance, **Transactional outbox → Intent / Motivation / Issues and considerations**. https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html [9](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html) | Live guidance | 2026-10-10 | Provider architecture guidance | **Supported. Keep.** Particularly relevant to warehouse/downstream handoff. | | Compensation is distinct from forward recovery/retry | AWS Prescriptive Guidance, **Saga patterns**. https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/saga-patterns.html [11](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/saga-patterns.html) | Live guidance | 2026-10-10 | Provider architecture guidance | **Supported. Keep.** Business-specific implementation still required. | | Outbox events can carry event ID separately from aggregate/business identity | Debezium, **Outbox Event Router → Basic outbox table**. https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html [10](https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html) | Live “stable” documentation; inspected example identifies connector `3.7.0.Final` | 2026-10-10 | Primary project documentation | **Supported. Keep**, without treating the example connector version as the whole doc’s version. | | CloudEvents distinguishes unique event identity (`source + id`) from event subject and permits extension attributes | CNCF CloudEvents spec, **Required Attributes: id/source; subject; Extension Context Attributes**. https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md [2](https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md) | Live `main`; document labels itself `1.0.3-wip` | 2026-10-10 | Primary specification repository | **Supported. Qualify:** `causation_id`/`hop_count` in this dossier are custom editorial fields. | | SQLite UPSERT can update or no-op on uniqueness conflict | SQLite, **UPSERT → Description**. https://www.sqlite.org/lang_upsert.html [12](https://www.sqlite.org/lang_upsert.html) | Live documentation | 2026-10-10 | Primary software documentation | **Supported. Keep** only as harness implementation evidence, not a distributed-systems guarantee. | | Retry-enabled event functions can loop on permanent errors; defined end conditions are recommended | Firebase, **Retry asynchronous functions → Semantics / Best practices**. https://firebase.google.com/docs/functions/retries [1](https://firebase.google.com/docs/functions/retries) | Live documentation | 2026-10-10 | Provider documentation | **Supported. Keep.** Product-specific retry windows should not be generalized. | | A platform can show successful/`200` execution logging despite an underlying function error under documented conditions | Firebase, **Retry asynchronous functions**, discussion immediately before retry configuration. same URL [1](https://firebase.google.com/docs/functions/retries) | Live documentation | 2026-10-10 | Provider documentation | **Supported, highly product-specific. Keep as counterexample**, not universal workflow behavior. | | Publisher pressure can accumulate requests in memory until deadline failure; flow control can bound outstanding work | Google Cloud Pub/Sub, **Flow control → About flow control**. https://docs.cloud.google.com/pubsub/docs/flow-control-messages [13](https://cloud.google.com/pubsub/docs/flow-control-messages) | Live documentation | 2026-10-10 | Provider documentation | **Supported. Keep.** Strong evidence for explicit backpressure rather than unbounded memory queues. | | `Idempotency-Key` draft models a unique client key, non-reuse across different payloads, and explicit expiry policy | IETF HTTPAPI WG, **draft-ietf-httpapi-idempotency-key-header-07 §§2.2–2.5**. https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-idempotency-key-header [6](https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-idempotency-key-header) | Published 2025-10-15; expired 2026-04-18 [6](https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-idempotency-key-header) | 2026-10-10 | Standards-track Internet-Draft, expired | **Conditional / not normative current standard. Qualify prominently.** Useful design vocabulary; do not call it an RFC. | ### Evidence status of the central draft claims | Candidate claim | Status | Advice | |---|---|---| | “Two automations trigger each other. Stop the infinite circle with retries.” | **Contradicted as general advice.** Retries can perpetuate permanent failures; a termination condition plus ownership/causation guard is needed. [1](https://firebase.google.com/docs/functions/retries) | **Rewrite** around causation, meaningful change, one-writer ownership and a fuse. | | “The API is rate-limited, so failed requests can simply be retried.” | **Conditional.** Backoff is supported for transient throttling, but accepted work needs durability and deadlines; uncontrolled retry can worsen overload. [8](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/retry-backoff.html) | **Qualify.** Ask what may wait, what must survive, and when escalation beats another retry. | | “Cancellation arrived first, so it wins.” | **Unresolved without business revision semantics.** Arrival order is not trustworthy. [4](https://docs.stripe.com/webhooks) | **Rewrite:** newest authoritative source revision wins; retrieve current source state when revision is unavailable. | | “We have exactly-once delivery, therefore customers cannot be charged/notified twice.” | **Contradicted as an end-to-end inference.** Provider guarantee boundaries are narrower, and publish-side duplicates may remain. [5](https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery) | **Remove or rewrite** as a product-scoped transport claim. | | “The workflow is green, so recovery worked.” | **Contradicted as sufficient evidence.** Provider docs contain concrete cases where acknowledgement/execution success is decoupled from downstream completion or even internal error reporting. [4](https://docs.stripe.com/webhooks) | **Replace** with business-state and side-effect invariants. | | “Store event IDs and duplicates are solved.” | **Contradicted as complete advice.** Separate Event objects can represent the same object/event semantics; event identity differs from business-operation identity. [4](https://docs.stripe.com/webhooks) | **Rewrite** to distinguish event dedupe from business dedupe. | | “Idempotency keys make retries safe forever.” | **Contradicted.** Provider key validity can be finite; Stripe documents different v1/v2 windows. [7](https://docs.stripe.com/api-v2-overview) | **Remove.** Retention/replay horizon is part of the design. | ## Contradictions, limits and reusable article directions ### Where the recommendation should reverse The deterministic recovery architecture is not automatically better than a manual process. **Very low volume + high consequence.** If a team performs a handful of unusual refunds, contract terminations, physical shipments or irreversible communications, human verification before the side effect may be cheaper and safer than building sophisticated autonomous recovery. Provider idempotency can still be used underneath the manual action where available. AWS itself notes that the complexity of robust idempotent service contracts is not right for every solution. [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) **No authoritative business revision.** If the source gives events that cannot be ordered and there is no reliable current-state lookup, automatic conflict resolution is fundamentally under-informed. The system should reduce autonomy rather than invent an ordering rule. Stripe’s warning about timestamp ordering illustrates the problem. [4](https://docs.stripe.com/webhooks) **No idempotency and no external reconciliation interface.** For a timeout after an irreversible side effect, the system may literally be unable to distinguish “did not happen” from “happened but response vanished.” In that case an automatic blind retry is not evidence-based recovery; human review, serialization or a better provider interface is warranted. AWS’s lost-response/idempotent-token discussion is the relevant documented design precedent. [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) **Ordering costs more than it solves.** Google documents throughput/latency trade-offs around ordered exactly-once Pub/Sub processing and notes higher latency for exactly-once subscriptions. Where the business object itself carries a source revision, accepting out-of-order transport and reconciling state can be simpler than enforcing total queue ordering. This latter conclusion is an architectural inference, not a Google recommendation. [5](https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery) **AI sees too few exceptions to repay its upkeep.** Under the illustrative cost model above, the AI layer adds 30 first-year maintenance/build hours before it saves anything. At low exception volume, deterministic runbooks and good audit queries are the simpler system. The numeric comparison is synthetic, not empirically measured. ### Contradictions and unresolved gaps The evidence does not support a universal “exactly once is impossible” slogan. Some providers make legitimate, useful exactly-once claims inside clearly defined boundaries. Google Pub/Sub defines exactly-once semantics under specific subscription/region requirements; AWS calls FIFO queue processing exactly-once. The correct criticism is narrower: **those contracts must not be silently extended beyond their documented boundary to an end-to-end business outcome.** [5](https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery) Stripe’s “return `2xx` quickly” advice and a durable-before-acknowledgement design are not actually contradictory, but there is an engineering tension: durable acceptance has to be quick enough not to turn persistence latency into webhook timeouts. The inspected Stripe documentation requires a quick response before complex work but does not prescribe an inbox implementation. The durable-inbox recommendation is therefore an inference and should be presented as such. [4](https://docs.stripe.com/webhooks) Transactional outbox does not create magical exactly-once behavior. AWS explicitly notes that the event processor can still publish duplicates and recommends idempotent consumers. Outbox solves a particular atomicity gap; it does not remove every other one. [9](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html) Dead-letter storage is not synonymous with recovery. A dead letter without an owner, alert, inspection path and safe replay mechanism is merely durable failure. AWS documents dead-letter constructs, but the ownership/runbook requirement in this dossier is editorial operating design. [14](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html) Compensation is not rollback in the database sense. Sending a refund does not make an accidental charge historically nonexistent; a corrective email does not un-send the first one. Saga guidance supports the conceptual distinction between forward retry and compensating transactions, but the exact compensating business action must be domain-defined. [11](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/saga-patterns.html) The current IETF Idempotency-Key material inspected for this research had expired as an Internet-Draft by the snapshot date and explicitly describes itself as work in progress. It is therefore supporting vocabulary rather than a current standards authority. [6](https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-idempotency-key-header) No production tests were performed against Stripe, Google Cloud, AWS, Debezium, or any commerce source. Provider documentation was inspected; the only firsthand execution was the synthetic SQLite harness. No evidence was found in the inspected sources for a universal optimal deduplication-retention period. Retention must be derived from actual provider retry/resend windows, source replay behavior, business correction horizon and storage/governance constraints. Stripe’s differing replay windows demonstrate why one fixed folklore number would be unsafe. [4](https://docs.stripe.com/webhooks) ### Smallest evaluation before building Before implementing the full workflow, run a fixture suite against the intended design with these acceptance tests: | Injection | Required observable result | |---|---| | Same delivery twice | Only one durable application | | Same event under new delivery ID | Event deduped; both attempts auditable | | Different event IDs, same business revision/intent | One business side effect | | `CANCEL r3` before `CONFIRM r2` | Final state remains `CANCELLED r3` | | Crash after durable accept | Work resumes from inbox | | Crash after local state update | Outbox intent still exists | | Timeout after remote side effect | Reconciliation prevents blind duplication | | `429` burst | Work remains durable; retry pressure backs off | | Permanent validation failure | Stops retrying and becomes owned dead letter | | Source replay beyond provider idempotency TTL | Local business operation guard still determines safety | | Automation A → B → A | Causation guard or meaningful-change rule terminates chain | | New legitimate higher revision after cancellation | New revision is processed according to domain rules | | Reconciliation mismatch | Incident is visible even if every workflow execution was green | These are **editorial acceptance criteria**, with the underlying failure modes grounded in provider documentation on retries, ordering, idempotency, dual writes, throttling and exactly-once scope. [4](https://docs.stripe.com/webhooks) ### Candidate article angles | Angle | Core thesis | Evidence strength | |---|---|---| | **Green Is Not Done** | Judge automation by source state + side-effect evidence + reconciliation, not the run icon | Strong: Stripe, Firebase, Pub/Sub [4](https://docs.stripe.com/webhooks) | | **The Five IDs Behind Safe Automation** | Delivery ID, event ID, business ID, source revision and operation ID solve different problems | Strong for event/business distinction; operation model is editorial synthesis [2](https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md) | | **Exactly Once, Until You Leave the Box** | Broker-level guarantees should not be presented as end-to-end business guarantees | Strong provider counterevidence [5](https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery) | | **A Timeout Is an Unknown Result, Not a Failure** | Reconcile before repeating payments/messages after ambiguous responses | Strong engineering-practice support [3](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) | | **Cancellation Arrived First. Should You Believe It?** | Arrival order is weaker than source revision/current-state reconciliation | Strong webhook evidence; source-revision design editorial [4](https://docs.stripe.com/webhooks) | | **Your Retry Queue Is a Business Priority Queue** | Rate limits require durability, deadlines and backpressure, not “retry=true” | Strong provider evidence [8](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/retry-backoff.html) | | **Two Automations Walk Into a Loop** | Stop reciprocal triggers through ownership, causation and meaningful transitions | Retry-loop risk documented; specific causation design editorial [1](https://firebase.google.com/docs/functions/retries) | | **Idempotency Expires** | Provider key retention and your business replay horizon are different design variables | Particularly concrete with Stripe v1/v2 + webhook replay windows [4](https://docs.stripe.com/webhooks) | | **Dead Letter Is Not a Recovery Plan** | Failed work is only recovered when an owner can inspect, decide, repair and reconcile | DLQ mechanism documented; operational conclusion editorial [14](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html) | ### Useful reusable assets The complete Python/SQLite fixture above is the principal portable test asset. It was actually executed locally during this research; it uses no private data and no external system. A second useful asset is a minimal event contract: ```json { "delivery_id": "del-17", "source": "shop.example", "event_id": "evt-91", "business_entity": { "type": "order", "id": "ORD-42" }, "source_revision": 3, "intent": "CANCELLED", "operation_key": "notify_cancel:ORD-42:r3", "origin": "source", "causation_id": "cause-7", "hop_count": 0 } ``` `source`/event `id` map naturally to CloudEvents concepts; `business_entity`, `source_revision`, `operation_key`, `causation_id` and `hop_count` are application-level design fields in this dossier. CloudEvents explicitly allows extension attributes but assigns no standard meaning to these custom names. [2](https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md) A minimal recovery record should make this question answerable without reading application logs: ```json { "operation_key": "notify_cancel:ORD-42:r3", "business_state": "CANCELLED", "source_revision": 3, "accepted": true, "attempts": 2, "external_result": "verified", "external_reference": "provider-object-123", "terminal_disposition": "done", "compensation_required": false, "reconciled_at": "2026-10-10T12:00:00Z" } ``` That schema is an editorial asset, not a provider standard. ### Freshness triggers Re-check this dossier whenever Stripe changes webhook retry/manual-resend windows, webhook ordering guidance, or v1/v2 idempotency behavior; those are live provider contracts and the current API v1/v2 retention differences are operationally material. [4](https://docs.stripe.com/webhooks) Re-check Pub/Sub if exactly-once changes its pull-only constraint, regional condition, publish-side duplicate behavior, or latency trade-off. The inspected page was updated as recently as **October 9, 2026**, one day before this snapshot. [5](https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery) Re-check the real order source whenever it changes event IDs, webhook replay rules, source revision/version semantics or current-object retrieval behavior. The synthetic design depends critically on the source revision being authoritative. Re-run the duplicate/late/timeout/quota/loop fixture suite after changes to queue provider, database transaction boundaries, workflow engine, API version, downstream provider, retry policy or operation-key retention. Re-evaluate the AI triage layer after changes to model/tool permissions. The safety boundary assumed here is that AI **does not** authorize payment, message sending or production writes. Retire or simplify the custom automation when a native provider integration can satisfy the same observable outcome and recovery evidence at lower lifecycle burden, when event volume no longer justifies upkeep, or when the team can no longer name an owner for reconciliation and recovery. The central design test remains the same: **not whether the machinery is green, but whether the business state can be proven correct after the machinery fails.** ## Portable citation index 1. [Retry asynchronous functions  |  Cloud Functions for Firebase](https://firebase.google.com/docs/functions/retries) 2. [spec/cloudevents/spec.md at main · cloudevents/spec · GitHub](https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md) 3. [references-details-empty](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) 4. [docs.stripe.com](https://docs.stripe.com/webhooks) 7. [docs.stripe.com](https://docs.stripe.com/api-v2-overview) 5. [Exactly-once delivery  |  Pub/Sub  |  Google Cloud Documentation](https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery) 6. [draft-ietf-httpapi-idempotency-key-header-07](https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-idempotency-key-header) 8. [Retry with backoff pattern - AWS Prescriptive Guidance](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/retry-backoff.html) 9. [Transactional outbox pattern - AWS Prescriptive Guidance](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html) 11. [Saga patterns - AWS Prescriptive Guidance](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/saga-patterns.html) 14. [What is Amazon Simple Queue Service? - Amazon Simple Queue Service](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html) 10. [Outbox Event Router :: Debezium Documentation](https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html) 12. [UPSERT](https://www.sqlite.org/lang_upsert.html) 13. [Flow control  |  Pub/Sub  |  Google Cloud Documentation](https://cloud.google.com/pubsub/docs/flow-control-messages) 15. [Timeouts, retries, and backoff with jitter | AWS Builder Center](https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter) ## Import verification and asset availability Genuine Deep Research completed in 9 minutes and was captured on 2026-10-10 after the first due time. Original export, exact independent prompt, canonical chat and citation URLs are retained. Complete inline examples/code were preserved; no missing sandbox download target is claimed as a delivered asset. Report code, test outcomes and execution assertions were not independently run or verified during this import. Delivery is not independent claim verification; Evidence status remains Captured.