# Retries, Duplicates and Idempotency: Evidence Dossier for Recovery Design

**Research snapshot:** 2026-10-10  
**Perspective:** Academic, with primary provider documentation where implementation semantics matter  
**Decision supported:** Explain recovery design through **observable business outcomes**, not workflow “green run” status  
**Active-work scope:** A small team operating an event/webhook-driven order workflow into a cloud analytical warehouse and downstream business automation  
**Evidence boundary:** Public sources only. No private systems, customer data, vendor tests, firsthand execution, or claimed customer outcomes. Worked timelines, fixtures, cost examples, and fault injections below are **synthetic and reasoned, not executed**.

## Concise answer and uncertainty

The strongest defensible recommendation is:

> **A recovery design is correct when the intended business state can be demonstrated after retries, duplicates, reordered events, partial failures and repair—not when every automation run is green.**

That distinction follows from several independent bodies of evidence.

Saltzer, Reed and Clark’s end-to-end argument is directly relevant: functions such as duplicate suppression, recovery and acknowledgement can exist in lower layers, but lower-layer correctness can be insufficient for the application’s end-to-end requirement. citeturn15view0turn15view1 HTTP makes a similarly narrow distinction: an idempotent method is one whose **intended server effect** is the same when repeated; it does not promise that every internal side effect or response is identical. citeturn17view14 Modern messaging and warehouse products reinforce the boundary problem. Pub/Sub exactly-once is scoped to supported pull subscriptions and a region; BigQuery exactly-once is obtained through stream offsets; Kafka’s adopted KIP scopes idempotence and transactions to particular producer/log and consume-transform-produce boundaries. citeturn18view3turn18view5turn17view5turn17view6turn24view0 None of those guarantees, by itself, proves that a customer was charged once, a shipment exists once, the latest order state won, or a cancellation was honored.

A useful design therefore needs at least four different identities:

| Identity | Example | What it answers |
|---|---|---|
| **Transport/message identity** | `source + event_id` | “Have I seen this delivered event before?” |
| **Business entity identity** | `order_id = O-7421` | “Which real-world thing is this about?” |
| **Source version** | `revision = 43` | “Which known state supersedes which?” |
| **Operation intent identity** | `fulfillment:create:O-7421:L1` | “Is this retry the same business action or a new action?” |

CloudEvents explicitly defines an event identity through the combination of `source` and `id`, and also notes that one occurrence can result in more than one event. That is a useful formal separation between an event record and the underlying business occurrence. citeturn24view2turn16search11 Stripe makes the same distinction operationally: duplicate delivery of one event can be detected by event ID, but two separate Event objects may describe the same relevant change, in which case Stripe recommends looking at the underlying object ID plus event type. citeturn17view2turn18view2 AWS likewise reports that byte-for-byte-equivalent requests are not necessarily the same intent; its preferred API pattern is a caller-supplied request identifier. citeturn18view7

The main uncertainties are not about whether duplicates and disorder occur—they are explicitly documented by major providers—but about **application semantics**. A generic infrastructure layer cannot decide whether a late confirmation should override a cancellation, whether two identical-looking resource creates are intentional, whether a sent message can be safely repeated, or what compensation means. Those rules have to come from the process owner and source-of-truth contract. Lamport's foundational work explains why receiving messages in a particular order is not itself proof of business causality, and modern systems such as Stripe explicitly decline to guarantee event delivery order. citeturn14view2turn20view2turn18view0

There is also no evidence in the researched sources that adding an AI agent improves this process’s recovery correctness. The defensible AI role here is therefore **bounded exception triage and explanation**, behind deterministic controls and human approval—not payment authorization, messaging, or production writes.

For Czech versus US use, the core distributed-systems conclusions do not materially differ. The relevant technical distinctions found here are provider, region and configuration boundaries—for example Pub/Sub’s same-region exactly-once condition—not Czech-versus-US engineering principles. citeturn18view3 Legal record-retention, accounting, payment-dispute and consumer-communication differences were outside this technical dossier and should not be inferred from it.

## Evidence synthesis around the decision

### Start with the business process, not the automation

A bounded process makes “recovery” concrete.

**Synthetic process:** maintain the current order state in an analytical warehouse and, when an order first becomes confirmed, create one downstream fulfillment case. If that order is subsequently cancelled, no active fulfillment case may remain. The source commerce/order system is authoritative.

| Process element | Bounded specification |
|---|---|
| **Trigger** | Source emits an order-created, order-updated or order-cancelled webhook/event. |
| **Inputs** | Transport event ID and source; `order_id`; source revision/version where available; order lifecycle; status; payload/schema version; occurrence time; received time. |
| **Business owner** | Order/operations lead owns the meaning of “confirmed,” “cancelled,” reopen rules and acceptable recovery delay. |
| **Technical owner** | Data/automation engineer owns durable ingestion, reducers, queues, observability and repair tooling. |
| **Exception owner** | Named operations person or rota owns unresolved discrepancies and external side effects whose result is unknown. |
| **Business state** | For this fixture: `UNKNOWN → CREATED → CONFIRMED → CANCELED`. `CANCELED` is terminal for lifecycle `L1`; intentional reopening creates lifecycle `L2`. This is a synthetic policy, not a universal order rule. |
| **Observable completion** | Authoritative source and warehouse agree on latest relevant source revision; required downstream action exists no more than once; obsolete downstream action is compensated; every durably accepted event has a terminal disposition or named unresolved owner. |
| **Manual fallback** | Operator reads the authoritative order, downstream object and operation ledger; chooses apply/ignore/compensate; records reason and repair ID. |

A green workflow run is deliberately absent from the completion definition. It is useful telemetry, but it is not the business invariant.

This is a direct application of the end-to-end principle rather than a claim that the 1984 paper studied SaaS automation: Saltzer et al. explicitly discuss duplicate message suppression, crash recovery and acknowledgement among their examples and argue that application requirements determine where correctness must ultimately be established. citeturn15view0turn15view1 The translation from that systems principle to “verify the order, shipment or payment outcome” is an **editorial design inference**, not an empirical effect measured by the paper.

### Three ways to operate the same process

The right comparison is not “automation versus nothing.” It is the **simplest viable baseline versus deterministic automation versus bounded AI assistance**.

| Dimension | Manual/native baseline | Deterministic automation | Bounded AI triage |
|---|---|---|---|
| Intake | Native order dashboard/webhook log; periodic review/export | Webhook → durable inbox | Same deterministic intake |
| State choice | Operator checks source | Reducer compares business identity + source revision | Same reducer |
| Duplicate handling | Operator notices/reconciles | Unique transport ID plus semantic operation ledger | Same controls |
| External effect | Operator creates/cancels fulfillment case | Deterministic outbox/worker | **AI forbidden to execute it** |
| Ambiguous order | Operator inspects source | Rule sends to exception queue | AI summarizes evidence/proposes repair |
| Rate limit | Operator waits | Durable queue + deadlines + backoff | AI may classify backlog, not bypass quota |
| Audit | Native histories + manual notes | Event, decision, operation and repair ledgers | AI recommendation stored as advisory evidence |
| Upkeep | Human process training | Schema, API, retries, runbooks, monitors | All deterministic upkeep **plus** model/prompt/evaluation upkeep |
| Best fit | Low volume, high ambiguity, inexpensive manual work | Repetitive semantics and meaningful consequence of duplicate/loss | High exception-reading burden where human remains accountable |

The manual/native approach is a serious alternative. For five consequential exceptions a month, a boring dashboard and checklist can be better engineering than an event platform. Automation adds software states, provider contracts, queues, credentials, schemas and on-call responsibilities. The Dataflow authors make a related, though much larger-scale, observation: correctness, latency and cost are competing dimensions, not a free bundle. citeturn23view2turn20view5

The AI variant is not a fourth reliability layer. It inherits the deterministic workflow’s safety envelope. A reasonable bounded task is:

> “Here are the source order, accepted events, warehouse row and downstream operation ledger. Explain the discrepancy, cite the conflicting revisions, and recommend `ignore-stale`, `re-fetch-source`, `retry-safe-operation`, `compensate`, or `human-review`.”

The assistant does **not** authorize a payment, send a customer message, cancel a shipment, or write production state. No primary evidence reviewed here establishes that an AI assistant improves recovery rates or lowers total lifecycle cost for this process, so any such benefit remains an evaluation hypothesis.

### Retry safety, idempotency and exactly-once are not synonyms

The terms are often collapsed in automation discussions, and that is dangerous.

**Retry safety** asks whether repeating an attempted operation after uncertainty is acceptable. HTTP explicitly permits automatic retry reasoning for idempotent methods after communication failure. citeturn17view14

**Idempotency** says repeated expression of the **same intent** has the same intended effect. It does not mean the request occurred only once. Stripe's API, for example, lets clients attach an idempotency key so a failed connection can be retried without creating the same object or update twice; for API v1, keys can be pruned once they are at least 24 hours old, after which reuse can be treated as a new request. Stripe also rejects same-key retries with mismatching parameters. citeturn17view0turn18view9 AWS's Builders Library similarly emphasizes caller-provided intent identifiers and warns that idempotency records cannot reasonably be retained forever. citeturn18view7turn18view8

**Deduplication** asks whether a receiver recognizes a delivery as one already seen. Cloud Tasks, for example, rejects a duplicate named task and remembers deleted task names for up to 24 hours; SQS FIFO suppresses duplicate sends within its documented five-minute deduplication interval. citeturn17view7turn17view11 Those windows are infrastructure features, not a universal lifetime for business identity.

**Exactly-once** only has meaning together with its boundary and assumptions. Pub/Sub’s feature applies to pull/StreamingPull and its guarantee is regional; it relies on Pub/Sub message identity and successful acknowledgements. citeturn18view3turn18view5 BigQuery's default Storage Write API stream is at-least-once, while application-created streams can obtain exactly-once write semantics through offsets; Google's documentation explicitly describes the added control and complexity. citeturn17view5turn17view6 Kafka's adopted KIP explains how producer IDs and sequence numbers prevent duplicate persistence to the Kafka log, and how transactions can make Kafka consume-transform-produce operations atomic; it also explicitly limits some consumer-side implications and describes those guarantees from particular Kafka viewpoints. citeturn24view0turn24view1

This creates an important counterexample to simplistic advice: **provider exactly-once features are useful**. The answer is not “never trust infrastructure.” The answer is “do not silently expand its guarantee.” A BigQuery offset can establish that a row was not appended twice to one write stream; it does not establish that the source sent only one business occurrence or that a customer was charged once. That conclusion is an inference from the documented guarantee boundary. citeturn17view5turn17view6

**Compensation** is different again. Some business actions cannot be transactionally rolled back with your database. The original Saga work is foundational here; the original SIGMOD DOI is `10.1145/38713.38742`, and the Princeton technical-report version is publicly surfaced as TR-070-87. The ACM page was inaccessible to this research session and the Princeton PDF could not be rendered reliably, so I do not rely on uninspected passages from it. citeturn21search1turn21search4turn21search19 Current Microsoft architecture documentation describes the resulting pattern explicitly: record how to undo completed steps, recognize that compensation itself can fail, make compensating steps retry-safe where possible, and allow manual intervention where automation cannot safely recover. citeturn21search15 This is provider guidance rather than empirical proof.

### Failure must be modeled before recovery can be trusted

Distributed execution creates uncertainty even under surprisingly strong assumptions. FLP's theorem is often abused, so its scope matters: Fischer, Lynch and Paterson prove a nontermination possibility for deterministic consensus in a **completely asynchronous** model with one crash, and the paper expressly assumes reliable exactly-once message delivery for the theorem’s model. citeturn23view1turn22view0 Their conclusion explicitly says the result does not mean distributed problems cannot be solved in practice; rather, practical systems introduce stronger assumptions or weaker requirements. citeturn22view1 It therefore does **not** prove that “exactly once is impossible.” It does demonstrate why words such as timeout, failure and committed outcome depend on system assumptions.

Lamport provides the complementary ordering result: in a distributed system, causality gives a partial order, and messages can be received in a different order from the order a naive central observer expects. citeturn14view2turn15view2 Stripe turns this from theory into a current webhook contract: it does not guarantee delivery in generation order and instructs consumers not to depend on that order. citeturn18view0 Pub/Sub's default contract is likewise at-least-once with no ordering guarantee, unless the relevant ordering feature is explicitly configured. citeturn17view4

For the bounded process, that yields this failure model:

| Failure | Unsafe naive behavior | Safe criterion |
|---|---|---|
| Same event delivered twice | Run action twice | Same transport ID has one terminal processing disposition |
| Two different events represent same business effect | Event-ID dedup misses it | Operation intent/business identity prevents duplicate effect |
| Cancellation arrives before older confirmation | “Last message processed wins” resurrects order | Higher authoritative source revision wins |
| No source revision exists | Guess from receive time | Fetch authoritative current state or escalate ambiguity |
| Receiver replies success before durable persistence | Event can vanish after accepted response | Persist accepted work before acknowledging provider |
| Receiver persists then response is lost | Provider retries | Duplicate intake returns success without repeating effect |
| External side effect succeeds but client times out | Blind retry can duplicate payment/message/shipment | Reuse stable idempotency key or inquire by business operation ID before retry |
| Two workers race | Both observe “not done” | Atomic unique operation claim / transactional update |
| Dependency is down | Drop or hot-loop retries | Durable backlog, bounded retry, deadline, owner |
| Provider returns 429 | Immediate retries worsen pressure | Honor `Retry-After` where provided; otherwise bounded backoff plus jitter |
| Full ingestion history is replayed | Old commands rerun | Reducer/version semantics plus durable operation history make replay safe |
| Human corrects production state | Automation immediately undoes correction | Override is a versioned/auditable state considered by automation |
| Compensation fails | Mark original workflow successful anyway | Compensation remains open with an owner until verified |
| Automation A triggers B triggers A | Infinite writes/cost | Trigger on meaningful transitions, provenance guard, bounded hops/circuit breaker |

The rate-limit behavior has direct standards and provider support. HTTP 429 means too many requests and **may**, not must, include `Retry-After`. citeturn17view15 Cloud Tasks rate limits apply both to initial attempts and retries, use queue dispatch controls, and retry failed tasks with exponential backoff; Google also warns that bursts can exceed a nominal steady dispatch rate and produce 429 or 503 responses. citeturn24view3 AWS reports from its own operational practice that correlated retries can recreate overload and recommends adding jitter to spread retry attempts. citeturn18view6turn17view13 This is provider-reported operational experience, not a controlled cross-vendor trial.

Feedback loops are also not hypothetical infrastructure behavior. AWS EventBridge documentation explicitly warns that a rule can respond to a change, make another change and thereby fire itself indefinitely. citeturn17view9 The business-safe solution should therefore not merely impose `max_runs=5`. It should stop defining **every record mutation** as a meaningful business transition.

## Worked recovery design and portable synthetic assets

### Worked example: one order, several identities

This section is **synthetic and unperformed**.

Order `O-7421`, lifecycle `L1`, has authoritative source revisions:

```text
revision 41: CREATED
revision 42: CONFIRMED
revision 43: CANCELED
revision 44: CANCELED + metadata-only update
```

Transport events:

```jsonl
{"source":"shop.example/orders","event_id":"evt-41","order_id":"O-7421","lifecycle":"L1","revision":41,"type":"order.created","status":"CREATED"}
{"source":"shop.example/orders","event_id":"evt-42","order_id":"O-7421","lifecycle":"L1","revision":42,"type":"order.updated","status":"CONFIRMED"}
{"source":"shop.example/orders","event_id":"evt-43","order_id":"O-7421","lifecycle":"L1","revision":43,"type":"order.canceled","status":"CANCELED"}
{"source":"shop.example/orders","event_id":"evt-43","order_id":"O-7421","lifecycle":"L1","revision":43,"type":"order.canceled","status":"CANCELED"}
{"source":"shop.example/orders","event_id":"evt-42","order_id":"O-7421","lifecycle":"L1","revision":42,"type":"order.updated","status":"CONFIRMED"}
{"source":"shop.example/orders","event_id":"evt-44","order_id":"O-7421","lifecycle":"L1","revision":44,"type":"order.updated","status":"CANCELED","note":"address metadata corrected"}
```

The transport key is:

```text
(shop.example/orders, evt-43)
```

The business entity key is:

```text
order = O-7421, lifecycle = L1
```

The source-state version is:

```text
revision = 43
```

The operation identities are intentionally different:

```text
fulfillment:create:O-7421:L1
fulfillment:cancel:O-7421:L1
```

That is crucial. `evt-42` and `fulfillment:create:O-7421:L1` answer different questions. The event could be redelivered with the same event ID, represented by another provider event ID, or followed by metadata-only events. The business action should still occur once.

CloudEvents’ `source + id` rule is useful for the first identity, but its specification does not claim that event identity is business-operation identity. citeturn16search11turn24view2 AWS's request-identifier experience supports making intent explicit rather than deriving intent merely from identical parameters. citeturn18view7

### Synthetic duplicate / late / cancel timeline

Assume the events are **received** in this order:

```text
10:00:00  evt-41 / rev 41 / CREATED
10:00:03  evt-43 / rev 43 / CANCELED
10:00:04  evt-43 / rev 43 / CANCELED    [transport duplicate]
10:00:09  evt-42 / rev 42 / CONFIRMED   [late]
10:00:15  evt-44 / rev 44 / CANCELED    [metadata only]
```

A dangerous reducer says:

```text
current_status = status_of_last_received_event
```

At `10:00:09`, that produces `CONFIRMED`: a green run and a wrong order.

The safe synthetic reducer is closer to:

```text
if event.revision < stored.revision:
    disposition = STALE_IGNORED

elif event.revision == stored.revision
     and payload_semantics_already_applied:
    disposition = DUPLICATE_OR_EQUIVALENT

else:
    atomically:
        update business state to event.revision
        record meaningful transition
        enqueue any transition-driven side effect
```

After the timeline, the required observable result is:

```text
warehouse.order_revision        = 44
warehouse.order_status          = CANCELED
active_fulfillment_cases        = 0
duplicate_fulfillment_creates   = 0
unresolved_accepted_events      = 0
```

`evt-44` advances source version but does not cause a second “cancel fulfillment” action because the relevant business state did not transition from active to cancelled.

This is also why event timestamps are a poor universal resolver. Stripe explicitly warns its consumers not to infer event ordering from its snapshot-event creation timestamp and tells them not to depend on webhook delivery order. citeturn18view0 Dataflow was designed for large-scale input where old data may arrive or be retracted and emphasizes explicit correctness/latency/cost trade-offs rather than assuming an eventually neat arrival sequence. citeturn23view2

### Durable acceptance and the transaction boundary

The smallest deterministic architecture I would recommend is:

```text
Source webhook
     |
     v
[verify/authenticate]
     |
     v
[durable inbox INSERT] ----> only then acknowledge receipt
     |
     v
[state reducer]
     |
     +---- atomic DB transaction ----+
     |                               |
     | update current order state    |
     | insert semantic outbox row    |
     +-------------------------------+
                     |
                     v
              [rate-limited queue]
                     |
                     v
             [side-effect worker]
                     |
          stable operation identity
                     |
          +----------+-----------+
          |                      |
          v                      v
   [known outcome]       [unknown outcome]
          |                      |
     record result      inquire/reconcile
                                 |
                          retry or human
```

Stripe requires webhook endpoints to return a successful status quickly, before lengthy business processing. citeturn18view1 The specific recommendation to put a **minimal durable write before that success response** is an editorial recovery design: otherwise “accepted” can mean “the remote sender will stop trying” while the local process has no recoverable record. Complex processing remains asynchronous, consistent with Stripe’s recommendation to queue webhook work. citeturn18view2

The local transaction boundary should include, where supported by the datastore:

```sql
BEGIN;

INSERT INTO accepted_event (...)
ON CONFLICT (event_source, event_id) DO NOTHING;

-- If genuinely new and newer than known source state:
UPDATE order_state
SET source_revision = :revision,
    status          = :status
WHERE order_id = :order_id
  AND source_revision < :revision;

-- Only on a meaningful transition:
INSERT INTO operation_outbox (
    operation_key,
    operation_type,
    order_id,
    status
)
VALUES (
    'fulfillment:create:O-7421:L1',
    'FULFILLMENT_CREATE',
    'O-7421',
    'PENDING'
)
ON CONFLICT (operation_key) DO NOTHING;

COMMIT;
```

This SQL is a **portable conceptual asset**, not tested against a named SQL engine. Syntax and concurrency semantics must be adapted and verified for the actual database.

The external fulfillment API is deliberately outside that transaction. A database transaction cannot magically roll back an already-created shipment at an independent provider. The operation ledger therefore needs a state such as:

```text
PENDING
IN_FLIGHT
SUCCEEDED
UNKNOWN_EXTERNAL_OUTCOME
COMPENSATION_REQUIRED
COMPENSATED
HUMAN_RESOLVED
```

A simple binary `SUCCESS/FAILURE` is inadequate.

### Timeout after an external side effect

Synthetic execution:

```text
10:01:00.000 worker calls create_fulfillment(op_key=fulfillment:create:O-7421:L1)
10:01:00.300 provider creates case FC-9981
10:01:00.320 response packet is lost
10:01:05.000 client times out
```

The worker **does not know** whether nothing happened or `FC-9981` exists.

Blind retry is safe only when the destination gives the required idempotent contract for the same business intent and the key is still valid. Stripe documents this pattern for API requests; AWS describes the same uncertainty and the value of stable caller request IDs. citeturn17view0turn18view7

Without such a contract, safe repair is:

```text
1. Query destination by operation/business identity if possible.
2. If exactly one matching effect exists -> record SUCCEEDED.
3. If none exists -> retry according to policy.
4. If multiple effects exist -> stop; compensate/escalate.
5. If destination cannot establish outcome -> HUMAN_REVIEW,
   unless business policy explicitly accepts duplicate risk.
```

This is particularly important for **payments and messages**. A payment call that timed out after a charge was accepted is not made safe merely because the surrounding workflow failed red. A second call with a different idempotency key is a new operation from an API perspective. Stripe's documented key-retention semantics illustrate why a provider key cannot be treated as permanent business memory. citeturn18view9 Likewise, deduplicating an inbound webhook does not itself deduplicate an unrelated outbound email, SMS, payment or shipment.

### Rate-limited queue: what may wait and what must not disappear

The queue should separate **durability** from **dispatch urgency**.

For this process, a synthetic business classification could be:

| Work | May wait? | May disappear? | Deadline behavior |
|---|---:|---:|---|
| Raw accepted event | Yes briefly | **No** | Persist before normal async work |
| Cancellation reconciliation | Limited | **No** | Escalate when deadline approaches |
| Payment/outbound-effect uncertainty | Limited | **No** | Stop blind retry; reconcile |
| Warehouse freshness update | Usually | No, unless periodic rebuild is authoritative | Backlog acceptable within freshness SLO |
| Nonessential enrichment | Yes | Potentially, by explicit business policy | Expire/cancel instead of infinite retry |
| AI exception summary | Yes | Yes; human can inspect raw evidence | Never block core recovery |

HTTP 429 alone does not say how valuable a request is; that prioritization is application policy. The standard says a 429 response may include `Retry-After`. citeturn17view15 Cloud Tasks provides dispatch-rate and concurrency limits and applies rate controls to retries as well as first attempts. citeturn24view3

A bounded retry calculation should be visible. One illustrative policy is:

```text
base delay b = 2 seconds
attempt index n = 0..7
cap C = 300 seconds

uncapped(n) = b * 2^n
capped(n)   = min(C, uncapped(n))
actual wait = jitter(0, capped(n))
```

So the caps before jitter are:

```text
2 s, 4 s, 8 s, 16 s, 32 s, 64 s, 128 s, 256 s
```

Total maximum nominal pre-jitter waiting if the full cap were used at every listed attempt:

```text
2 + 4 + 8 + 16 + 32 + 64 + 128 + 256 = 510 seconds
```

Those numbers are **synthetic policy inputs**, not provider recommendations. The evidence-backed principles are bounded retries, backoff for repeated failure, and jitter to avoid synchronized retry storms. citeturn18view6turn24view3

Do not let “max retries exhausted” become “finished.” The terminal state is instead something such as:

```text
DEAD_LETTER:
  business_deadline: 2026-10-10T10:15:00Z
  reason: downstream_429_exhausted
  source_revision: 43
  owner: order-ops
  repair_action: reconcile_current_source_before_replay
```

The **dead-letter queue is a recovery queue, not a success queue**. That is an editorial design rule, not a provider guarantee.

### Feedback-loop timeline

Synthetic loop:

```text
T0 Automation A:
   sees "order row changed"
   -> writes CRM field crm_synced=true

T1 Automation B:
   sees "CRM order changed"
   -> writes warehouse field crm_last_seen=T1

T2 Automation A:
   sees "order row changed"
   -> writes CRM again

T3 Automation B:
   ...
```

AWS documents the same failure shape in EventBridge: an event rule can cause a change that recreates the event that triggered the rule, resulting in an infinite loop. citeturn17view9

A safer design is:

```text
trigger_on =
    meaningful_business_transition
    AND origin != self
    AND transition_id not already applied
```

with metadata such as:

```json
{
  "correlation_id": "order-O-7421-L1",
  "causation_id": "evt-42",
  "origin": "order-sync-A",
  "semantic_transition": "CREATED->CONFIRMED",
  "hop_count": 1
}
```

Safe repair criteria are not merely “loop stopped.” They are:

```text
authoritative order state still correct
downstream state reconciled
unwanted loop-generated mutations identified
no legitimate business change discarded
trigger rule corrected
replay performed from known source state, not blindly from loop history
```

### Idempotency-key lifetime must match the failure horizon

Provider windows differ sharply. Stripe API v1 keys may be pruned after at least 24 hours; Cloud Tasks remembers deleted task names for up to 24 hours; SQS FIFO deduplication covers a documented five-minute interval. citeturn18view9turn17view7turn17view11 AWS's Builders Library explicitly discusses late requests and says retention needs vary by service and resource. citeturn18view8

Therefore:

```text
provider idempotency window ≠ business deduplication horizon
```

The local retention decision should be driven by:

```text
H_business =
max(
    automatic_retry_horizon,
    manual_resend_horizon,
    expected_late_delivery_horizon,
    supported_replay/backfill_horizon,
    external_outcome_reconciliation_horizon
)
```

subject to legitimate storage/privacy constraints.

Stripe itself illustrates why this can be much longer than an automatic retry window: its webhooks retry automatically for up to three days, while dashboard and CLI manual resend mechanisms can extend substantially beyond that. citeturn17view1 A 24-hour local dedup table would therefore be an unsafe universal assumption for such a webhook process.

### Reproducible fault-injection evaluation

This is a **proposed test protocol; it was not run**.

Known truth:

```yaml
order_id: O-7421
lifecycle: L1
authoritative_final_revision: 44
authoritative_final_status: CANCELED
expected_active_fulfillment_cases: 0
expected_ai_production_writes: 0
```

Run the same fixture under these injections:

| Injection | Mechanism | Required result |
|---|---|---|
| Duplicate | Deliver every event twice | Same final state; same semantic effect count |
| Out of order | Deliver `rev43` before `rev42` | Rev42 cannot restore confirmed state |
| Replay | Replay rev41–44 from scratch after completion | No duplicate downstream operation |
| Crash before durable acceptance | Kill receiver before inbox commit and before ack | Sender retry can recover event |
| **Negative control: ack before persistence** | Return success, then crash before write | Test should expose unrecoverable gap |
| Ack lost after persistence | Persist, then make response disappear | Redelivery safely deduplicated |
| Race | Process rev42 and rev43 concurrently | Final reducer state rev43/44 cancelled |
| External side effect + timeout | Destination commits, response lost | No second effect without outcome check/idempotency |
| Dependency outage | Fail destination for 15 minutes | Accepted work survives and ages visibly |
| Quota | Return HTTP 429 repeatedly | Backoff; deadline visible; no silent loss |
| Loop | Make A's write trigger B and vice versa | Guard/circuit stops loop; business state reconciled |
| Human override | Operator marks operation held | Automation does not silently overwrite override |
| Compensation failure | Cancel effect fails | Case stays unresolved with owner; not green-complete |

The key assertions should query **business truth**, for example:

```sql
-- Pseudocode: exact syntax engine-dependent.

-- Latest source revision materialized:
assert warehouse_order.revision = 44;
assert warehouse_order.status = 'CANCELED';

-- Business effect:
assert count(active_fulfillment
             where order_id='O-7421'
               and lifecycle='L1') = 0;

-- Recovery accountability:
assert count(accepted_events
             where terminal_disposition is null
               and age > recovery_deadline) = 0;
```

A test that merely asserts:

```text
workflow_runs.failed = 0
```

does not test the decision this dossier is intended to support.

## Practical implications, ownership and lifecycle cost

### The smallest workable implementation

For a small team, I would not begin with Kafka transactions, a workflow engine, a general saga platform, an agent framework and five observability products. Provider features can be excellent, but complexity itself has an upkeep cost. BigQuery explicitly characterizes application-created streams as offering additional control, including exactly-once behavior, at additional complexity. citeturn17view6

The smallest implementation worth evaluating is:

1. **One durable inbox table** keyed by transport identity.
2. **One current-state table** keyed by business entity/lifecycle with source revision.
3. **One operation/outbox table** keyed by stable business-operation intent.
4. **One rate-limited worker queue** with bounded retries.
5. **One reconciliation query/job** comparing source truth, warehouse state and unresolved side effects.
6. **One exception view/runbook** with a named owner.

No AI is needed to establish the core recovery properties.

The system can later adopt Pub/Sub exactly-once, BigQuery offsets, Kafka transactions, FIFO queues or provider idempotency where their guarantees reduce code. Pub/Sub can eliminate duplicates within its documented exactly-once boundary; BigQuery offsets can eliminate repeated writes within a write stream; Kafka transactions can make Kafka-side consume-transform-produce atomic. citeturn18view5turn17view5turn24view0 That should delete redundant custom machinery, not weaken the end-to-end invariant.

### Observability should distinguish processing health from outcome health

A useful dashboard would have two panels.

**Transport/processing signals:**

```text
webhook delivery attempts
accepted events
duplicate transport events
queue depth
oldest queue age
retry count
429/5xx rate
dead-letter count
worker failure rate
```

**Business/recovery signals:**

```text
orders behind authoritative source revision
confirmed orders missing required fulfillment
canceled orders with active fulfillment
operation intents with UNKNOWN_EXTERNAL_OUTCOME
uncompensated operations
accepted events past recovery deadline
manual overrides awaiting reconciliation
source-to-warehouse divergence age
```

Only the second group answers whether recovery achieved the business outcome. This separation is an application of the end-to-end principle and documented provider guarantee boundaries, not a published empirical finding that this exact dashboard improves a measured KPI. citeturn15view0turn17view6turn24view0

### Ownership cannot be delegated to the queue

A recovery design should state, before launch:

```text
Business invariant owner:
    decides what correct business state means

Automation owner:
    owns deterministic processing, retries and observability

Exception owner:
    resolves UNKNOWN_EXTERNAL_OUTCOME / semantic ambiguity

Provider owner:
    tracks API/schema/guarantee changes

Incident decision:
    who may pause processing, replay, compensate or retire automation
```

This matters because infrastructure can identify “retry exhausted” but cannot generically decide whether a cancelled hotel booking should trigger a refund, another booking, or human contact. Modern compensating-transaction guidance explicitly notes that compensation is application-specific and sometimes needs human intervention. citeturn21search15

### Lifecycle cost should be measured in upkeep, not just build hours

A reusable staff-hour model is:

\[
H_{year1}
=
H_{build}
+
12H_{monthly\ ops}
+
N_{changes}H_{change}
+
N_{incidents}H_{recovery}
\]

and for a mature year:

\[
H_{recurring}
=
12H_{monthly\ ops}
+
N_{changes}H_{change}
+
N_{incidents}H_{recovery}
\]

A more complete financial model is:

\[
C =
H \times C_{loaded\ hour}
+
C_{provider}
+
C_{incident\ impact}
\]

where all three terms must use a declared currency and period if actual monetary comparison is performed.

A deliberately synthetic sensitivity example:

```text
Manual/native:
  recurring reconciliation = 8 staff-hours/month
  build                    = 0 h
  first-year staff-hours   = 8 × 12 = 96 h

Deterministic:
  initial build            = 48 h
  routine upkeep           = 2 h/month = 24 h/year
  provider/schema changes  = 4/year × 3 h = 12 h
  recovery incidents       = 2/year × 2 h = 4 h
  year-one total           = 48 + 24 + 12 + 4 = 88 h

Bounded AI layer added:
  integration/evaluation   = 24 h
  additional upkeep        = 1 h/month = 12 h
  two evaluation changes   = 2 × 4 h = 8 h
  incremental year-one     = 44 h
```

Under **these invented assumptions only**, deterministic automation saves eight staff-hours in year one versus the manual process, while adding the AI layer is economically worse unless it saves more than:

\[
44/12 \approx 3.67\ \text{staff-hours/month}
\]

elsewhere.

Those numbers are **not estimates of a real implementation**. They show the calculation that a real pilot should fill with observed maintenance and recovery time.

Upkeep is not theoretical. Stripe webhook payload structure depends on API versioning, and BigQuery documents behavior that applications need to handle when table schemas change during streaming. citeturn18view0turn17view5 Kafka semantics are configuration- and version-sensitive; the adopted KIP itself was last updated in March 2026. citeturn24view1 Provider contracts therefore belong in the maintenance inventory.

A sensible retirement condition is also part of the design:

```text
Retire/simplify when:
- native integration now provides the required invariant;
- volume/value no longer justifies upkeep;
- source semantics become too ambiguous for deterministic operation;
- recovery effort exceeds labor saved;
- provider/API changes invalidate the safety contract.

Before shutdown:
- stop new intake deliberately;
- drain or explicitly dispose of accepted work;
- reconcile source and downstream truth;
- resolve outstanding compensation;
- retain operation evidence for the applicable recovery/record horizon;
- document which system becomes authoritative afterward.
```

These are editorial operational recommendations rather than findings from a controlled study.

## Claim-to-primary-source dossier

The table distinguishes what the evidence actually establishes from what an article should say. **Inspected date for every row: 2026-10-10.**

| Candidate claim | Primary source and exact scope | Source date | Evidence type | Assessment | Editorial action |
|---|---|---:|---|---|---|
| “A green lower-level run does not establish the application outcome.” | Saltzer, Reed & Clark, *End-to-End Arguments in System Design*, abstract/introduction/conclusion. Low-level duplicate suppression/recovery may be redundant or insufficient relative to application requirements. https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf DOI: `10.1145/357401.357402` citeturn15view0turn15view1 | Nov 1984 | Foundational design argument, not experiment | **Supported as interpretation** | **Keep, qualify:** translate principle to outcome verification; do not claim the paper studied workflow dashboards. |
| “Arrival order is not necessarily causal/business order.” | Lamport, *Time, Clocks, and the Ordering of Events in a Distributed System*, “The Partial Ordering” / logical clocks. https://lamport.azurewebsites.net/pubs/time-clocks.pdf DOI: `10.1145/359545.359563` citeturn14view2turn15view2turn20view2 | Jul 1978 | Formal/distributed-systems model | **Supported** | **Keep.** Do not stretch it into a claim that every SaaS webhook is unordered. |
| “Even reliable exactly-once messaging does not eliminate distributed coordination difficulty.” | Fischer, Lynch & Paterson, §§1 and 5. The model actually assumes correct exactly-once message delivery yet proves a nontermination possibility for asynchronous deterministic consensus with one crash. https://groups.csail.mit.edu/tds/papers/Lynch/jacm85.pdf DOI: `10.1145/3149.214121` citeturn23view1turn22view1 | Apr 1985 | Formal impossibility theorem | **Supported within strict assumptions** | **Qualify heavily.** Never rewrite as “exactly once is impossible.” |
| “Long-running distributed actions may require compensation rather than rollback.” | Garcia-Molina & Salem, *Sagas*. Princeton TR-070-87 surfaced at https://www.cs.princeton.edu/techreports/1987/070.pdf; proceedings DOI `10.1145/38713.38742`. Full primary-text inspection was impaired in this session. citeturn21search4turn21search19 | 1987 | Foundational transaction design | **Supported bibliographically; primary passage not fully inspected** | **Keep as foundational reference, qualify access gap.** Current implementation detail should rely on an inspected source. |
| “Out-of-order/late data is a first-class processing concern and entails correctness/latency/cost choices.” | Akidau et al., *The Dataflow Model*, Abstract and §1. https://research.google.com/pubs/archive/43864.pdf DOI: `10.14778/2824032.2824076` citeturn23view2turn20view5 | 2015 | System model plus reported real-world design experience | **Supported** | **Keep.** Do not infer small-team ROI from Google-scale experience. |
| “Stateful stream processing can provide strong framework-level fault tolerance.” | Akidau et al., *MillWheel*, Google Research publication/abstract. https://research.google.com/pubs/pub41378.html DOI: `10.14778/2536222.2536229` citeturn20view1 | 2013 | System paper / deployment report | **Supported within framework boundary** | **Qualify:** framework record processing is not an external payment guarantee. |
| “Transactional serverless workflows are technically feasible rather than merely theoretical.” | Zhang et al., *Beldi*, OSDI 2020, description/evaluation; tested applications over AWS Lambda. https://www.usenix.org/conference/osdi20/presentation/zhang-haoran citeturn20view4 | Nov 2020 | Academic system implementation and controlled evaluation | **Supported** | **Keep as feasibility evidence;** not evidence that small teams should adopt equivalent machinery. |
| “HTTP idempotency is about intended effect, not identical execution history.” | RFC 9110 §9.2.2, “Idempotent Methods.” https://www.rfc-editor.org/rfc/rfc9110.html#section-9.2.2 citeturn17view14 | Jun 2022 | Internet standard | **Supported** | **Keep.** |
| “Message identity should not be conflated with business identity.” | CloudEvents 1.0.3-wip main specification, Context Attributes `id`/`source`; `source + id` uniquely identifies distinct events; one occurrence may generate multiple events. https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md citeturn16search11turn24view2 | Current main snapshot | Open standard/specification | **Supported** | **Keep, qualify:** CloudEvents does not define your order ID or business revision. |
| “Webhook events can be duplicated.” | Stripe, “Handle duplicate events”; same event may arrive more than once; separate Events can also represent the relevant same object/type. https://docs.stripe.com/webhooks citeturn17view2turn18view2 | Current docs | Provider documentation | **Supported for Stripe** | **Keep provider-specific;** do not claim every provider behaves identically. |
| “Webhook event generation/delivery order cannot safely be assumed.” | Stripe, “Event ordering”; delivery order is not guaranteed. https://docs.stripe.com/webhooks citeturn18view0 | Current docs | Provider documentation | **Supported for Stripe** | **Keep.** Generalization should be phrased conditionally. |
| “Idempotency keys have a lifetime and same key should represent the same intent.” | Stripe API Reference, “Idempotent requests”; API v1 keys may be pruned ≥24h and parameter mismatch is rejected. https://docs.stripe.com/api/idempotent_requests citeturn18view9 | Current docs | Provider documentation | **Supported for Stripe API v1** | **Keep with API-version qualifier.** |
| “Content equality alone cannot establish duplicate business intent.” | AWS Builders Library, “Making retries safe with idempotent APIs,” client request identifiers, late requests and same ID/different intent sections. https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/ citeturn18view7turn18view8 | Current article | Attributable provider engineering practice | **Supported as AWS-reported practice** | **Keep but label reported practice, not theorem.** |
| “Exactly-once provider guarantees have explicit scope.” | Google Pub/Sub, “Exactly-once delivery”: pull/StreamingPull only, regional guarantee, acknowledgement requirements. https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery citeturn18view3turn18view5 | Current docs | Provider documentation | **Supported** | **Keep; central evidence.** |
| “A warehouse can offer exactly-once writes without proving end-to-end business correctness.” | BigQuery Storage Write API: default stream at-least-once; application-created streams + offsets exactly-once. https://docs.cloud.google.com/bigquery/docs/write-api-grpc citeturn17view5turn17view6 | Current docs | Provider documentation | **Documented guarantee; end-to-end limitation is inference** | **Keep, explicitly separate the two.** |
| “Kafka exactly-once has a defined transaction boundary.” | Apache Kafka KIP-98, “Motivation” and “Summary of Guarantees”; producer sequencing, cross-partition transactions and consume-transform-produce atomicity, with stated consumer caveats. https://cwiki.apache.org/confluence/spaces/KAFKA/pages/66854913/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging citeturn24view0turn24view1 | Last updated Mar 4 2026 | Adopted open-source design specification | **Supported** | **Keep; do not extrapolate to arbitrary external effects.** |
| “Default at-least-once delivery requires duplicate tolerance.” | AWS SQS standard queue documentation; copies can be redelivered and applications should be idempotent. https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html citeturn17view10 | Current docs | Provider documentation | **Supported for SQS Standard** | **Keep provider-specific.** |
| “SQS ‘exactly-once processing’ terminology has a finite deduplication scope.” | SQS FIFO documentation: retried `SendMessage` within five-minute deduplication interval does not introduce duplicates into queue. https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html citeturn17view11 | Current docs | Provider documentation | **Supported with scope** | **Rewrite headline claims:** “exactly-once within documented queue dedup semantics,” not business effect once. |
| “Queue throttling applies to retries too.” | Google Cloud Tasks, “Define rate limits” / “Set retry parameters.” https://docs.cloud.google.com/tasks/docs/configuring-queues citeturn24view3 | Current docs; page surfaced with 2026 update | Provider documentation | **Supported** | **Keep.** |
| “429 does not automatically mean Retry-After is present.” | RFC 6585 §4. https://www.rfc-editor.org/rfc/rfc6585.html#section-4 citeturn17view15 | Apr 2012 | Internet standard | **Supported** | **Keep;** say “honor if supplied,” not “always obey the header.” |
| “Jitter reduces synchronized retry pressure.” | AWS, *Timeouts, retries, and backoff with jitter*, “Jitter.” https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter citeturn18view6turn17view13 | Jun 12 2026; modified Jun 15 | Provider-reported engineering practice | **Supported as reported practice** | **Keep, qualify evidence type.** |
| “Two automations can trigger each other indefinitely.” | AWS EventBridge, “Avoid writing infinite loops.” https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-patterns-best-practices.html citeturn17view9 | Current docs | Provider documentation | **Supported** | **Keep.** The provenance/state-transition remedy is editorial design. |
| “Compensation may fail and sometimes requires a person.” | Microsoft Azure Architecture Center, “Compensating Transaction pattern.” https://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction citeturn21search15 | Apr 20 2026 | Provider architecture guidance | **Supported as guidance** | **Keep, label guidance rather than measured effect.** |

### Direct assessment of the three draft claim targets

**“Dvě automatizace se spouštějí navzájem. Jak zastavit nekonečné kolečko?”**  
**Status: supported, but the remedy needs qualification.** Event-driven infinite loops are explicitly documented by AWS. citeturn17view9 The evidence supports the existence of the failure mode. “Track origin, trigger only on a meaningful state transition, suppress an already-applied transition, and add a circuit breaker” is an editorial architecture proposal, not a provider-tested universal recipe. **Advice: keep the claim; qualify the solution.**

**“Systém omezuje počet požadavků. Co může počkat a co se nesmí ztratit?”**  
**Status: supported plus editorial business classification.** Standards and queue documentation support rate limiting, bounded dispatch and retry behavior. citeturn17view15turn24view3 They do **not** decide whether a cancellation is more important than a warehouse enrichment job. **Advice: keep, explicitly say that infrastructure controls rate while the process owner defines deadlines and loss tolerance.**

**“Nejdřív přišlo zrušení, potom potvrzení. Který stav má automatizace použít?”**  
**Status: strongly supported as a real class of problem; the answer is conditional.** Distributed messages do not inherently give a reliable global business order, and Stripe specifically does not guarantee webhook delivery order. citeturn14view2turn18view0 **Advice: rewrite any answer of “use the latest event” to “use the authoritative source version/business transition; if no trustworthy version exists, re-fetch authoritative state or escalate rather than guessing from arrival time.”** The precise cancellation-vs-confirmation rule remains domain-specific.

## Contradictions, limits and conditions where the recommendation reverses

The evidence does not support a maximalist “always build an idempotency platform” position.

### Cases where simpler recovery is better

**One transactional system.** If the state update and all meaningful effects occur inside one database transaction, an outbox/saga architecture can be unnecessary overhead. The additional machinery exists mainly when atomicity stops at a system boundary.

**Naturally idempotent overwrite.** A well-defined PUT that sets a resource to a desired final representation is explicitly within HTTP’s idempotent-method model. citeturn17view14 A complicated semantic operation ledger may add little if retries are already harmless and the result is directly verifiable.

**Provider guarantee covers the entire required boundary.** For a purely Kafka-internal transform, Kafka's transaction boundary can be the relevant business boundary. citeturn24view0 For a warehouse-only ingestion where duplicate rows are the sole concern, BigQuery stream offsets may resolve the exact problem. citeturn17view5turn17view6 In such cases, custom dedup code can be worse than using the provider correctly.

**Low-value, low-volume process.** If an operator spends one hour a month reconciling ten records and an error has trivial cost, manual/native operation can dominate an engineered recovery system on lifecycle cost. This is an economic inference, not an empirical threshold established by the literature reviewed here.

**Best-effort analytics with authoritative rebuild.** If an analytics table can safely be dropped and regenerated from an authoritative snapshot, elaborate event-level compensation may have little value. The recovery outcome is then “rebuild matches authoritative snapshot,” not “every event processed once.”

### Cases where automatic retry itself should stop

The recommendation becomes more conservative when an external effect is high-impact, irreversible, or unqueryable.

Suppose a payment/transfer/message/shipment endpoint:

```text
- may execute before the client observes timeout;
- has no usable idempotency contract;
- cannot be queried by your operation identity;
- has material duplicate cost.
```

Automatic retry can be the wrong recovery action. The safest state may be:

```text
UNKNOWN_EXTERNAL_OUTCOME -> reconcile/human
```

rather than:

```text
UNKNOWN_EXTERNAL_OUTCOME -> retry
```

AWS's discussion of lost responses and stable request tokens shows why outcome uncertainty exists; provider idempotency contracts are one mechanism for resolving it, not a universal property of APIs. citeturn18view7

### “Exactly once is impossible” should be removed

That phrase is contradicted by the source record if used without a boundary. Pub/Sub, BigQuery, Kafka and SQS all document forms of exactly-once or duplicate suppression under explicit conditions. citeturn18view5turn17view5turn24view0turn17view11 FLP does not establish the claim either: its theorem concerns deterministic consensus termination under a particular asynchronous fault model, and its message subsystem is assumed reliable and exactly-once. citeturn23view1

A better statement is:

> **Exactly once can be a real and valuable property inside a specified system boundary. The mistake is treating that boundary guarantee as proof that an externally visible business outcome occurred exactly once.**

### “Deduplicate by event ID” is insufficient as a universal rule

CloudEvents makes `source + id` a legitimate event-level duplicate identity, and Stripe recommends event IDs for duplicate receipts. citeturn16search11turn18view2 But Stripe also documents cases where different Event objects correspond to the relevant same object and event type. citeturn17view2 AWS likewise warns that apparently identical requests can represent different intended resource creates. citeturn18view7

Thus:

```text
event deduplication
≠ business occurrence deduplication
≠ operation idempotency
```

An article that merges those should be rewritten.

### “Last event wins” is not a general ordering strategy

Last **authoritative source revision** may win if the source contract gives a monotonic revision and the business process is a last-write state model. Last **arrival at your webhook handler** has no such general justification. Lamport establishes the distributed ordering problem formally; Stripe supplies a concrete current provider contract with unordered deliveries. citeturn14view2turn18view0

If the source exposes neither revision nor a reliably queryable current object, automatic conflict repair remains **unresolved**. A timestamp tie-breaker can produce deterministic behavior, but deterministic is not synonymous with correct.

### Evidence gaps

No source reviewed here establishes a cross-industry effect size such as “idempotency reduces incidents by X%.” None should be invented.

No controlled study reviewed here compares manual workflows, deterministic automation and AI-assisted recovery for a small Czech or US business.

No AI reliability study was included that would justify autonomous exception repair in this process.

The Saga primary source was identified through ACM/Princeton, but ACM returned access restrictions and the Princeton PDF could not be reliably rendered in the research tooling. Its bibliographic role is retained, while detailed implementation claims are grounded in inspected contemporary documentation instead. citeturn21search1turn21search4

The Dataflow PDF was successfully inspected for its abstract/introduction but a subsequent rendered page request failed after a redirect; claims here are therefore limited to the text that was actually accessible. citeturn22view2turn23view2

Provider documentation documents provider behavior, not independent business outcomes. AWS Builders Library material is attributable engineering practice and experience, not an independent controlled experiment. Stripe/Google/AWS docs establish API contracts, not causal evidence that a proposed architecture reduces a customer's financial loss.

The fault-injection plan, SQL, order fixture, cost calculations and repair timelines in this dossier are **unperformed synthetic assets**. They should not be described later as a test result.

## Reusable article angles, assets and freshness triggers

Several distinct articles can legitimately be supported from this dossier without turning them into the same “idempotency explainer.”

### Green workflow, wrong business

**Thesis:** “Finished successfully” describes execution, not necessarily recovery.

The narrative can start with the late-confirmation-after-cancellation fixture: every step returns 200, the workflow is green, yet a stale confirmation resurrects a cancelled order. The core intellectual backing is the end-to-end argument plus documented unordered webhook delivery. citeturn15view0turn18view0

The useful visual asset is:

```text
green transport
+ green workflow
+ green queue
≠ verified business invariant
```

followed by:

```text
source order = CANCELED rev43
warehouse    = CANCELED rev43+
fulfillment  = inactive
exceptions   = 0 past deadline
= observable recovery
```

### Exactly once — exactly where?

**Thesis:** Every “exactly once” claim needs a box drawn around it.

One comparison graphic could show:

```text
SQS FIFO       -> duplicate-send suppression window
Pub/Sub        -> supported message delivery/ack boundary
Kafka          -> producer/log + transactional Kafka boundary
BigQuery       -> stream/offset write boundary
Stripe key     -> one API operation intent during retained key lifetime
Business       -> source state + external side effects + compensation
```

The provider differences are documented and make this a strong anti-hand-waving article. citeturn17view11turn18view5turn24view0turn17view6turn18view9

### The cancellation arrived first

**Thesis:** Receive order is not business truth.

The article can distinguish:

```text
event ID
order ID
source revision
receive timestamp
event timestamp
operation intent
```

Lamport supplies the foundational ordering perspective, while Stripe provides a concrete webhook case. citeturn20view2turn18view0

### Retrying is load, not magic

**Thesis:** A retry consumes scarce capacity and can make an overloaded service worse.

Start from HTTP 429, move to a durable backlog and business deadlines, then use backoff/jitter rather than a hardcoded “retry three times.” RFC 6585, Cloud Tasks and AWS retry guidance provide primary support. citeturn17view15turn24view3turn18view6

The editorial business question is the stronger hook:

> Which work may become late, and which work may never silently disappear?

### Two automations are arguing with each other

**Thesis:** Triggering on “record changed” is often too weak; trigger on meaningful state transitions.

AWS's EventBridge warning gives a recognizable documented failure mode. citeturn17view9 The synthetic A↔B timeline then makes provenance, causation IDs and semantic transition guards concrete.

### Idempotency keys expire; business history may not

**Thesis:** Infrastructure deduplication windows are not the same as the period in which a duplicate business effect remains harmful.

Stripe's ≥24-hour key behavior, Cloud Tasks' up-to-24-hour task-name memory and SQS FIFO's five-minute deduplication interval make the point without claiming any one provider is deficient. citeturn18view9turn17view7turn17view11 They solve different problems with different scopes.

### Useful reusable assets from this dossier

The following assets are complete enough to reuse without an external download:

**Identity card**

```text
message identity   = source + transport event ID
business identity  = entity ID + lifecycle
source version     = authoritative revision/sequence
operation intent   = stable ID for one semantic side effect
```

**Business terminal dispositions**

```text
APPLIED
TRANSPORT_DUPLICATE
SEMANTIC_DUPLICATE
STALE_IGNORED
SUPERSEDED
SIDE_EFFECT_SUCCEEDED
COMPENSATED
HUMAN_RESOLVED
UNRESOLVED   <-- never count this as completed
```

**Recovery outcome checklist**

```text
[ ] Latest authoritative business state known
[ ] Warehouse agrees with authoritative source
[ ] Semantic side effect exists exactly as intended
[ ] Duplicate/obsolete effects absent or compensated
[ ] Every accepted item has a terminal disposition
[ ] Unknown external outcomes have a named owner
[ ] Human overrides are visible to automation
[ ] Replay has been tested against the same invariants
```

**Fault fixture**

The six-line JSONL `O-7421` fixture above is the portable test fixture. It is synthetic and **unperformed**.

**Fault matrix**

The twelve injection cases above constitute a portable evaluation specification. They are **unperformed**.

### Freshness triggers

Re-check this dossier rather than reusing provider claims unchanged when any of these occurs:

| Trigger | What to re-verify |
|---|---|
| Stripe API/webhook version change | Idempotency-key retention, POST/DELETE semantics, webhook retry/resend horizon, event ordering and duplicate guidance. citeturn18view9turn17view1 |
| Pub/Sub feature/config change | Subscription types supported by exactly-once, region restriction, ordering interaction, acknowledgement semantics. citeturn18view3turn18view5 |
| BigQuery ingestion architecture change | Default-stream semantics, application-created offsets/transactions, schema behavior. citeturn17view6 |
| Kafka major-version/config change | Current implementation defaults versus the adopted KIP’s original configuration text and guarantee boundary. The KIP was updated March 4, 2026. citeturn24view1 |
| Cloud Tasks queue change | Task-name deduplication retention, dispatch/burst/retry behavior. citeturn17view7turn24view3 |
| SQS FIFO change | Deduplication window and exact provider meaning of “exactly-once processing.” citeturn17view11 |
| Source adds/removes revision numbers | Conflict resolver and replay strategy; this can materially change whether automatic ordering is safe. |
| New external irreversible effect | Re-evaluate inquiry, idempotency and compensation before enabling retries. |
| AI role expands beyond advice | Require a separate evaluation of authorization boundaries, false repair recommendations, auditability and rollback before allowing any production write. |

The durable editorial conclusion survives those freshness checks:

> **Retries are a transport/recovery mechanism. Idempotency is an operation contract. Deduplication is recognition of repeated identity. Exactly-once is a scoped processing guarantee. Compensation is a business repair action. None of them, alone, is evidence that the business outcome happened.**

For leaders deciding what to automate, that yields a practical test: before discussing retry counts, queue products or agents, write down the authoritative business state, the externally observable effect, the acceptable exception state, and who repairs uncertainty. Then design recovery so those facts can still be demonstrated after the ugly timeline—not merely after the happy green run. citeturn15view0turn17view14turn24view0turn21search15