# Observability for Business Automations: Evidence Dossier

**Research question:** How can a team see that the expected result happened, not just that a job ran?  
**Perspective:** Academic, with provider behavior and practitioner evidence separated from research evidence  
**Research snapshot:** 2026-10-10  
**Active-work scope:** A bounded synthetic daily data-delivery process: an expected source export arrives, is preserved, ingested and transformed in an analytical warehouse, and a downstream business report should become available by a declared deadline.  
**Geographic relevance:** Czech and US contexts where materially different; the core control design is the same.

## Decision answer and evidence boundaries

**The strongest practical answer is to model “business completion” as a separate, durable state from “workflow execution.”** For every expected unit of work, create an identifiable case *independently of the workflow that is supposed to perform it*, give that case a business period, deadline, owner, and explicit acceptance criteria, and then verify the resulting business state end to end. A green scheduler is evidence about execution, not sufficient evidence about the intended result.

This distinction is not merely philosophical. Apache Airflow 3.3.2 defines a successful DAG run from terminal states of its leaf tasks; its documentation even warns that a leaf using `all_done` can make the DAG run successful despite a failure in the middle. citeturn22view0 Google SRE similarly distinguishes internal/white-box status from externally visible behavior and notes that an HTTP `200` can coexist with wrong content; detecting that class of error requires an end-to-end check. citeturn14view0 Huang et al.'s production experience with “gray failures” gives the deeper distributed-systems reason: a failure detector and the affected application can have different views of whether a system is healthy, which the authors call **differential observability**. citeturn22view1turn20view2

That leads to a recommended architecture:

\[
\text{Expected work}
\rightarrow
\text{observed stage evidence}
\rightarrow
\text{business acceptance predicate}
\rightarrow
\text{case state}
\rightarrow
\text{notification/action}
\]

rather than:

\[
\text{scheduler ran}
\rightarrow
\text{send “success” notification}
\]

The proposed architecture is an **interpretation and design recommendation**, not a controlled empirical finding. Its intellectual basis is consistent with Saltzer, Reed, and Clark's end-to-end argument: some functions can only be implemented completely and correctly with application-level knowledge at the endpoints; lower-level mechanisms may still improve performance or reliability, but cannot substitute for the end-level check. citeturn20view0turn20view1

For the bounded daily process, I would define two separate predicates:

```text
BUSINESS_COMPLETE(batch) =
    expected_batch_exists
    AND raw_input_preserved
    AND source_identity_matches_expected_batch
    AND required_input_completeness_passes
    AND warehouse_period = expected_business_period
    AND business_invariants_pass
    AND source_to_target_reconciliation_passes
    AND report_data_period = expected_business_period
    AND report_is_available_to_the_intended_access_path

ON_TIME(batch) =
    BUSINESS_COMPLETE(batch)
    AND verified_at <= declared_deadline
```

A recovered batch can therefore become `COMPLETED_LATE`; fixing it must **not rewrite history into an on-time success**.

The second crucial recommendation is to monitor **absence from an independent expectation**, not just failures emitted by the automation. If the scheduler never starts, there may be no job failure, trace, log entry, or webhook to inspect. The monitor therefore needs to know that `sales_daily / 2026-10-09` *ought to exist* and notice independently when its deadline passes without completion. Formal distributed-systems work on unreliable failure detectors is much more general than this business case, but it illustrates the underlying limitation: interpreting silence requires assumptions about time and failure detection rather than silence being self-evident proof of failure. citeturn13search39 In business automation, the practical assumptions are the declared deadline, timezone, calendar, grace period, and exception policy.

**Notifications should then be treated as a transport mechanism for actionable cases, not as the case record itself.** A delivered email, Slack message, page, or webhook does not prove that the underlying business state was corrected. Google SRE defines an alert as a notification intended for a human while treating investigation, mitigation, and root-cause work as subsequent activities; its incident-response guidance separately calls for defined roles and a working record of debugging and mitigation. citeturn14view0turn15view6

The best default for a small data team is therefore **deterministic expected-work reconciliation first, richer observability second, AI triage third**. Tracing is useful for explaining *where* a known case went wrong; Dapper was explicitly built to understand behavior and performance across large distributed systems. citeturn22view3 But a trace normally starts from something that happened. It does not inherently tell you that an expected batch never existed, nor that the final report contained the right business period.

There are important uncertainties. No finite test suite proves arbitrary semantic correctness: it proves only the properties encoded in the acceptance predicate. A source manifest and downstream table can agree while both are wrong. A synthetic report probe may exercise only part of the real user state; Google SRE explicitly cautions that artificial traffic can miss conditions affecting real users and can even dilute the real error signal. citeturn15view4 And I found no suitable controlled study establishing that LLM-based incident triage improves actual resolution for this specific small-team daily-data setting. AI is therefore treated below as a bounded hypothetical assistant, not as evidence of better operations.

## Process model and what counts as completion

The following is the **synthetic reference process** used throughout this dossier. Values are examples, not observations from a real system.

| Element | Synthetic reference design | Observable evidence |
|---|---|---|
| Business task | Publish the daily management report for business date \(D\) | One durable expected-work record for \(D\) |
| Trigger | One report is expected for each date in the process business calendar | Expected-work ledger, generated ahead of the workflow |
| Example deadline | `08:00 Europe/Prague` for the worked example | Stored deadline timestamp plus timezone/calendar identity |
| Inputs | Source export file(s), optional source manifest/control totals, transformation logic, report definition | Batch ID, business date, filenames, checksums, control totals, schema/version |
| Outcome owner | Business/report owner defines what constitutes acceptable delivery | Owner recorded with process definition |
| Technical owner | Data team operates ingestion, transformation and reconciliation | Named team/service ownership |
| Active case owner | Human or rotation responsible for an unresolved exception | Assignee + escalation policy |
| Main states | `EXPECTED → SOURCE_PRESERVED → INGESTED → TRANSFORMED → PUBLISHED → VERIFIED` | Timestamped evidence for each transition |
| Exception states | missing, late, stale, incomplete, duplicate, invariant failure, permission failure, publication failure | Case remains open even if some jobs are green |
| Terminal outcome | `VERIFIED_ON_TIME`, `VERIFIED_LATE`, `WAIVED`, or unresolved failure | Acceptance evidence and terminal case state |

The distinction between **business time** and **processing time** should be explicit. For this process, `business_date=2026-10-09` is domain/event time: the period the report claims to represent. `observed_at=2026-10-10T06:14:23Z` is processing/observation time: when a monitor happened to inspect it. Dataflow's event-time model was developed for unbounded, out-of-order processing and therefore exceeds the needs of a simple daily batch, but it is directly relevant to the conceptual mistake of equating “processed now” with “data for the expected period.” citeturn22view4 Airflow makes a related distinction between a DAG run's logical date/data interval and its actual execution time. citeturn22view0turn19view0

A practical acceptance record might therefore contain:

```text
process_id
case_id
business_date
source_partition
expected_at
deadline_at
deadline_timezone
business_calendar
source_contract_version
expected_manifest_id
outcome_owner
technical_owner
case_owner
current_state
first_failure_at
verified_at
resolution_evidence_uri_or_query_id
```

The **case identity should be business-oriented**, for example:

```text
case_id = process_id + business_date + source_partition
```

not merely `job_run_id`. Retrying a workflow can create several technical executions for one business case; conversely, two business dates can be simultaneously affected by one technical fault. Conflating execution identity and case identity makes both reconciliation and deduplication unreliable.

**Preservation** should establish that the input for \(D\) is recoverable and attributable, not that it is correct. A raw object checksum can prove that the bytes later examined are the bytes preserved; it cannot prove that the supplier exported the right business records. That stronger claim requires business identity, a manifest/control total, reconciliation, or another independent invariant.

**Freshness** should likewise not be reduced to “there are rows.” dbt's current freshness command can compare configured sources or models to freshness thresholds and produce warning/error status; dbt recommends scheduled checks and retaining longitudinal freshness results. citeturn18view7 That is useful native infrastructure, but a fresh table can still be incomplete, contain the wrong period, or represent the wrong business result. Freshness should therefore be one acceptance dimension rather than the acceptance criterion itself.

A useful set of synthetic calculations is:

\[
\text{freshness\_lag}
=
\text{checked\_at}
-
\text{actual\_data\_through\_timestamp}
\]

\[
\text{record\_delta}
=
\text{target\_records}
-
\text{expected\_source\_records}
\]

\[
\text{amount\_delta}
=
\text{target\_business\_total}
-
\text{expected\_business\_total}
\]

\[
\text{relative\_amount\_error}
=
\frac{
|\text{amount\_delta}|
}{
\max(|\text{expected\_business\_total}|,\epsilon)
}
\]

The tolerances are a business contract, not an observability-vendor default. An exact copy may warrant a zero count difference. A transformation that deliberately removes invalid records may instead require reconciling counts by documented categories. “Row count > 0” is rarely a meaningful substitute.

dbt data tests illustrate a useful deterministic implementation mechanism: tests can assert uniqueness, non-nullness, referential relationships, allowed values, or arbitrary organization-specific SQL business logic; a test passes when its query produces no failing rows. citeturn18view2 Stored failures can aid diagnosis, but dbt notes that a test's stored failure table replaces the preceding failures for that same test, so it should not automatically be treated as the durable incident ledger required for historical case management. citeturn18view3 dbt exposures can separately document a downstream dashboard/application and its owner/dependencies, which is useful metadata for tying transformed data to the actual consuming artifact. citeturn18view1

**The simplest viable baseline does not require an observability platform.** For a low-volume process, I would start with a manual reconciliation ledger:

1. Materialize the expected business dates in a spreadsheet or database ahead of time.
2. At the declared review time, an owner checks that the expected raw export exists and identifies \(D\).
3. Compare the source control total or manifest with warehouse results.
4. Confirm the report itself says/data-resolves to \(D\), not merely that the refresh job ran.
5. Record pass/fail, evidence, owner and corrective action.
6. Do not mark a failed case resolved until the same acceptance checks pass after recovery.

This baseline has weak scalability and depends on human discipline, but it establishes the **right semantics before automation**. A complicated monitoring system built before those semantics are clear merely automates uncertainty. Google SRE makes a closely related, although much more infrastructure-oriented, recommendation to keep critical monitoring simple and comprehensible and warns that monitoring itself can become a fragile maintenance burden. citeturn14view0

The process-design order I would use is therefore:

**Remove → standardize → clarify → automate → enrich.**

Remove steps and alerts for which nobody has a useful response. Standardize business-date/batch identity and preferably a source manifest. Clarify who owns the outcome, the deadline, acceptable tolerances, and what “resolved” means. Automate the same deterministic checklist. Only then add traces, anomaly detection, or AI where diagnosis rather than truth detection is the bottleneck. Google SRE's reported experience is consistent with that bias toward simpler alert rules: the organization describes limited success with complex learned/dependency-heavy monitoring and recommends keeping human-facing alert rules simple and representative of a clear failure. citeturn14view0

For Czech versus US implementations, there is no evidence-based reason to change the core pattern. The material differences are **calendar and time semantics**. Configure a named IANA timezone and an explicit business calendar for each process rather than assuming UTC dates or “Monday–Friday.” Airflow's current timezone documentation stores internal datetime information in UTC, supports timezone-dependent schedules, warns about ambiguous/nonexistent times around daylight-saving transitions, and notes that underlying timezone data may need updating when regulations change. citeturn18view6turn19view2 A Czech process may use `Europe/Prague`; a US process should use the appropriate local IANA zone rather than a generic national assumption. The same acceptance model applies.

## Evidence synthesis and viable alternatives

The central evidence is unusually coherent on one point: **observation should be made at the level of the property you care about.**

Saltzer, Reed, and Clark's 1984 end-to-end argument says that certain functions cannot be completely and correctly supplied at a low layer because only the application endpoints possess enough knowledge to establish the desired property. Their careful-file-transfer example considers failures at several points, including duplication and corruption, and motivates end-to-end verification rather than trusting each intermediate subsystem independently. citeturn20view0turn20view1 It is a design argument, not an experiment on modern data pipelines, but the transfer is structurally strong: “task completed” and “correct report for \(D\) is available” are properties at different layers.

Google SRE's production guidance makes the contemporary operational version explicit. White-box monitoring observes internals; black-box monitoring tests externally visible behavior. The SRE text gives the concrete example of an HTTP success code carrying wrong content and says only end-to-end tests can detect that condition. citeturn14view0 This directly contradicts architectures that treat HTTP acceptance, scheduler success, queue acknowledgment, or notification receipt as sufficient evidence of the downstream business result.

Gray-failure research provides counterexamples to assuming internal health and application health coincide. Huang et al. describe production cloud incidents in which system-level failure detectors did not observe a problem while applications were affected, labeling this mismatch “differential observability.” citeturn22view1turn20view3 Their scope is Azure/cloud infrastructure rather than daily analytics, and the paper contains reported incidents rather than a controlled trial, so it should not be treated as evidence for a particular monitoring product or threshold. It does, however, undermine the general assumption that one layer's “healthy” status is proof of another layer's outcome.

Airflow gives a particularly concrete modern example. A DAG run's terminal `success` is defined from leaf-node states, and the documentation warns that certain trigger-rule structures can yield a successful DAG run despite an intermediate task failure. citeturn22view0 This does **not** imply Airflow is unreliable; it implies that `DagRun.success` has a documented technical meaning. A team creates a category error when it silently upgrades that technical state into “the business report is correct.”

Yuan et al. provide relevant empirical evidence on failure handling, although not specifically observability. They analyzed 198 randomly selected user-reported failures across Cassandra, HBase, HDFS, Hadoop MapReduce, and Redis. They report that more than 30% of catastrophic failures in their corpus would have been preventable had their static checker identified and developers fixed the corresponding error-handling problems. citeturn22view2 The result supports the value of simple, explicit checks and testing over assumptions of correctness, but it **does not establish** that reconciliation controls reduce business-pipeline incidents by 30%, and that effect size must not be transferred.

Tracing solves a different problem. Google's Dapper technical report describes a production distributed-tracing infrastructure designed to help developers and operators understand behavior and performance across large distributed services. citeturn22view3 Correlating a `case_id` or `batch_id` across acquisition, warehouse transformation, publication, and alerting is therefore useful for diagnosis and handover. But this dossier treats tracing as **supporting evidence, not the success predicate**: a complete trace through an incorrectly parameterized pipeline can describe the wrong computation perfectly.

The same separation applies to retries and idempotency. They are important lifecycle mechanisms: retries can mask transient failure, duplicate delivery can result from replay, and recovery may require backfilling historical periods. Airflow supports clearing/rerunning tasks and historical backfill, for example. citeturn18view0 But the question here is downstream of those mechanisms: after retry or backfill, **how do we know the intended state was restored?** The answer remains: rerun the acceptance/reconciliation predicate for the original business case.

A comparison of viable designs follows.

| Approach | What it can establish | What it commonly misses | Upkeep | Best fit |
|---|---|---|---|---|
| **Manual acceptance checklist** | A human can inspect exactly the business artifact and reconciliation evidence that matter | Human omission, inconsistent interpretation, holidays/absence, poor scaling | Low technical upkeep; recurring human labor | Few important processes, unclear requirements, early validation |
| **Scheduler/native job monitoring** | Job/task execution state, retries, duration, logs | Missing workflow if expectation is defined only inside it; wrong/stale business result | Usually lowest incremental technical cost | Necessary layer, but insufficient alone |
| **Native freshness + data tests** | Staleness and explicitly encoded data invariants | Properties that have not been encoded; final BI/report access; missing expectation outside the tool | Moderate; tests evolve with models | Strong component of a data stack |
| **Independent expected-work + deterministic reconciliation** | Missing work, deadline breach, period mismatch, known completeness and business invariants | Unknown semantic errors; bad upstream control totals | Moderate; requires contract and case-state maintenance | **Recommended default for the scoped process** |
| **End-user black-box probe** | Whether a representative access path currently sees the expected result | User-specific states not represented by probe; side effects; permission complexity | Moderate to high | Critical reports where publication/access itself fails independently |
| **Distributed tracing/lineage** | Where known work travelled and how stages related | Work that never existed; semantic correctness by itself | Potentially significant | Diagnosis after detection, complex systems |
| **Generic anomaly detection** | Unexpected quantitative patterns not covered by known rules | Rare-event base-rate problems, changing normality, ambiguous actionability | Model/tuning/drift burden | Supplemental discovery when deterministic contract is insufficient |
| **Bounded AI triage** | Can potentially summarize deterministic evidence, retrieve runbooks and suggest hypotheses | Hallucination, overconfidence, no inherent ground truth | Prompt/model/evaluation/security upkeep | Supplemental triage, not acceptance or autonomous closure |

The deterministic expected-work pattern is particularly valuable for **silent failures**. An event-driven monitor reacts to events. A missing event is detectable only if another component knows an event was expected. I would therefore pre-materialize the expected-work ledger sufficiently far ahead—rather than generate today's expectation from the exact same scheduler that runs today's workflow—to reduce common-mode failure. That detail is a **reasoned architectural recommendation**, not a result directly measured in the cited literature.

An expected entry for a business date should not mean “the cron should have fired.” It should mean something closer to:

```text
Process: finance_daily
Business period: 2026-10-09
Expected result: executive_finance_report contains data through 2026-10-09
Deadline: 2026-10-10 08:00 Europe/Prague
Owner: Finance Analytics
Acceptance version: finance_daily/v7
```

Then an independent checker asks, after the deadline:

```text
For every expected case whose deadline has passed:
    Is there verified completion evidence?
    If not, is there already an open exception case for this exact business case?
```

This formulation also makes **calendar exceptions first-class**. Planned holidays, source closures, quarter-end schedules, and waived batches should alter the expected-work ledger before the deadline rather than be handled by routinely ignoring alerts. Otherwise every legitimate exception teaches responders that alerts are optional.

Alert quality deserves equal attention. Google's SRE guidance says pages should represent urgent, actionable conditions and describes how frequent pages can cause responders to skim, second-guess, or ignore alerts, allowing real problems to be masked by noise. citeturn14view0 The same chapter reports a Bigtable episode in which high alert volume consumed engineering time and obscured genuinely user-impacting problems; the team reduced alerting while addressing underlying problems. citeturn14view0 This is **provider-reported practice and retrospective case narrative**, not a controlled effect-size study and not grounds to prescribe Google's pager thresholds to a three-person analytics team.

A more transferable framework comes from Google's SRE Workbook: assess an alert detector by **precision, recall, detection time, and reset time**. citeturn15view3 For the bounded batch problem those definitions translate naturally:

\[
\text{precision}
=
\frac{\text{alerts corresponding to real failed business cases}}
{\text{all alerts}}
\]

\[
\text{recall}
=
\frac{\text{real failed business cases detected}}
{\text{all real failed business cases}}
\]

\[
\text{detection delay}
=
\text{first detection timestamp}
-
\text{time the failure became actionable}
\]

A low-noise rule can be useless if recall is poor; a detector that catches everything can be unusable if its false-alert rate overwhelms the owner. Those dimensions should be measured separately rather than summarized as “good observability.”

This matters particularly for anomaly detection because rare incidents create difficult base rates. As a **synthetic calculation, not an empirical estimate**, suppose true failures occur on \(0.1\%\) of task-days, a detector has \(90\%\) recall, and it falsely alerts on \(1\%\) of healthy task-days:

\[
P(F)=0.001,\quad
P(A|F)=0.90,\quad
P(A|\neg F)=0.01
\]

\[
P(F|A)
=
\frac{0.90\times0.001}
{0.90\times0.001 + 0.01\times0.999}
\approx 0.0826
\]

Only about **8.3% of alerts would correspond to a true failure under those synthetic assumptions**. The calculation does not show that real anomaly detectors have those rates; it shows why base rate, precision and ground truth must be made explicit before assuming that “AI anomaly detection” reduces noise.

For this reason, a deterministic question such as “expected case 2026-10-09 has no verified outcome five minutes after its declared deadline” is usually a cleaner starting detector than “today's row count looks unusual.” Anomaly detection is more defensible as a supplemental detector for business properties that cannot yet be specified deterministically.

The same caution applies to synthetic black-box traffic. It can be valuable because it checks behavior from outside, but Google's SRE Workbook warns that synthetic traffic exercises only part of a complex system's control surface and may succeed while actual users are experiencing errors. citeturn15view4 For a report, checking one API query or screenshot-equivalent access path therefore establishes only that *that representative path* succeeded.

Alert **grouping and deduplication should operate on notifications, not destroy business-case identity**. Prometheus Alertmanager explicitly supports grouping many similar alerts into one notification while retaining visibility into exactly which instances were affected. citeturn18view4 That is the right conceptual model for daily batches: one notification may say “three daily deliveries are incomplete,” but there should still be three durable case IDs. By contrast, Prometheus also documents an optional per-alert-name active-alert limit that can drop new alerts once the limit is reached. citeturn19view3 That feature may protect an alerting system under overload, but it illustrates why alert-manager state must not be the authoritative ledger of affected business cases.

Debouncing is similarly useful but semantically limited. Prometheus's `for` clause can require a condition to persist before firing and `keep_firing_for` can reduce flapping or false apparent resolutions caused by data loss. citeturn19view4 Neither is evidence that a business report has actually been restored. **Alert-condition clear**, **notification sent**, **human acknowledged**, and **business case reconciled** should remain different states.

A practical lifecycle model is therefore:

```text
EXPECTED
    |
    v
WAITING
    |
    +---- deadline missed --------------------> OPEN_EXCEPTION
    |
    v
INPUT_PRESERVED
    |
    +---- manifest mismatch ------------------> OPEN_EXCEPTION
    |
    v
INGESTED
    |
    +---- duplicate/partial/permission -------> OPEN_EXCEPTION
    |
    v
TRANSFORMED
    |
    +---- invariant failure ------------------> OPEN_EXCEPTION
    |
    v
PUBLISHED
    |
    +---- stale/wrong period/access failure --> OPEN_EXCEPTION
    |
    v
VERIFIED
    |
    +---- verified_at <= deadline -----------> VERIFIED_ON_TIME
    |
    +---- verified_at > deadline ------------> VERIFIED_LATE
```

A retry or repair can move an `OPEN_EXCEPTION` back into verification, but only verification closes it. Handover is therefore straightforward: the next responder needs the case identity, acceptance criteria, current evidence, attempted interventions, and next action, not merely a stream of historical notifications. Google SRE incident-response guidance likewise emphasizes defined roles and maintaining a working record during mitigation. citeturn15view6

## Worked failure scenario and alert-to-action design

The following scenario is **fully synthetic and was not executed against any production system**.

Assume process `sales_daily`, report deadline `08:00 Europe/Prague`, and business date `2026-10-08`.

The upstream system publishes a control manifest:

```text
batch_id: sales_daily_2026-10-08
business_date: 2026-10-08
expected_file_count: 2
expected_record_count: 100000
expected_net_sales_eur: 1250000.00
```

Only one of the two files actually reaches the ingestion location. It contains:

```text
actual_file_count: 1
actual_record_count: 80000
actual_net_sales_eur: 1000000.00
```

The loader is written to ingest all files it finds rather than verify expected files. It successfully loads 80,000 rows. SQL transformations succeed. The BI refresh succeeds. Airflow's final leaf task succeeds. The report contains rows for `2026-10-08`.

Several common checks therefore look reassuring:

| Signal | Synthetic observation | Does it prove success? |
|---|---:|---|
| Scheduler | Green | No |
| HTTP/report refresh request | Accepted | No |
| Warehouse row count | 80,000, therefore > 0 | No |
| Maximum business date | `2026-10-08` | No |
| Report opens | Yes | No |
| Manifest file count | expected 2, observed 1 | **Detects failure** |
| Source/target records | expected 100,000, observed 80,000 | **Detects failure** |
| Controlled sales total | expected €1,250,000, observed €1,000,000 | **Detects failure** |
| Business acceptance predicate | false | **Authoritative scoped outcome** |

This is a concrete form of the general problem identified in both end-to-end design and Google SRE monitoring: success at an intermediate layer does not establish the property required at the application/business endpoint. citeturn20view1turn14view0

At `08:00`, the independent deadline checker finds that the expected case has not reached `VERIFIED`. Suppose a synthetic five-minute grace period has been configured, so at `08:05` it opens:

```text
case_id = sales_daily|2026-10-08|all
class   = DATA_DELIVERY_INCOMPLETE
state   = OPEN
```

The proposed five-minute grace is **not evidence-based**; a real value must be negotiated from source behavior and business response needs.

Now add the required second failure: **the alert transport fails**. Perhaps the notification API returns an error or the receiver is unavailable. The architecture must not turn this into silence. The case already exists independently in the case store and remains `OPEN`; notification delivery is recorded as a separate attempted side effect:

```text
notification_attempt:
    case_id: sales_daily|2026-10-08|all
    channel: primary_on_call
    state: FAILED
    attempted_at: 2026-10-10T06:05:12Z
```

A separate notification-path sentinel can detect that the primary route is unhealthy and use an alternate route. This **sentinel/alternate-channel design is a synthetic architecture pattern**, not a measured result or documented guarantee of the cited providers. Critically, even successful transport would mean only “the notification path accepted/delivered what it was designed to deliver,” not “the report was repaired.”

On the next business date, suppose `sales_daily|2026-10-09|all` develops the same incompleteness before the first case is repaired. A naïve dedupe strategy such as:

```text
dedupe_key = DATA_DELIVERY_INCOMPLETE
```

could make two business failures appear to be one persistent alert. Instead retain:

```text
sales_daily|2026-10-08|all   OPEN
sales_daily|2026-10-09|all   OPEN
```

and group the **notification**:

```text
Sales daily: 2 incomplete deliveries
- sales_daily|2026-10-08|all
- sales_daily|2026-10-09|all
```

This follows the same separation embodied in Alertmanager grouping: one compact notification can still retain the identities of all affected alert instances. citeturn18view4

Recovery for `2026-10-08` should be evidence-driven:

1. Obtain and preserve the missing source file.
2. Verify its batch/business-date identity and checksum.
3. Re-ingest or backfill the affected period using the system's safe recovery mechanism.
4. Re-run transformations for the affected period.
5. Re-run deterministic invariants.
6. Reconcile the complete source manifest to warehouse/control totals.
7. Verify the downstream report's `data_through`/business period and intended access path.
8. Persist the verification evidence.
9. Mark the business case `VERIFIED_LATE` if verification happened after the original deadline.
10. Close the incident/case; do **not** change the historical SLA observation to “on time.”

Airflow's documented backfill and task-reexecution capabilities are examples of implementation mechanisms for steps three and four, not proof of steps five through seven. citeturn18view0

The resulting **alert-to-action inventory** should distinguish urgent notification from durable case management:

| Condition | Source of expectation/check | Suggested routing | Owner/action | Grouping rule | Resolved-case criterion |
|---|---|---|---|---|---|
| Expected export absent after deadline | Independent expected-work ledger vs preserved inputs | Urgent only if action is needed before business impact; otherwise ticket | Data responder confirms upstream status/recovery path | Group notification by process/owner; keep each business-date case | Export preserved **and** downstream acceptance rerun successfully |
| Source incomplete or manifest mismatch | Source control manifest vs observed files/counts/totals | Usually actionable case | Investigate missing/extra input before trusting downstream output | Never merge different batch IDs into one case | Manifest reconciliation passes after recovery |
| Transformation invariant fails | Deterministic SQL/data tests | Severity depends on report impact | Inspect failing records and transformation/version | Group messages by incident if common cause; preserve affected cases | Same invariant suite passes on repaired business period |
| Report stale/wrong period | Report metadata or representative black-box query vs expected period | High if consumers will use wrong data | Report/data owner traces publication stage | One case per report-period pair | Report exposes expected period and upstream reconciliation passes |
| Duplicate/replayed batch | Batch identity/uniqueness checks | Usually ticket unless it alters published numbers | Quarantine/reconcile duplicate | Per affected business period | Duplicate removed/neutralized and totals reconciled |
| Intended user/service cannot access report | Representative access-path probe | High when delivery requires access | Restore permissions/publication | Do not suppress merely because backend data is healthy | Expected principal/path succeeds and data-period check passes |
| Alert/notification route unhealthy | Independent channel-health sentinel | Alternate route/meta-alert | Observability owner repairs notification path | Group transport failures, not underlying business cases | Primary path tested healthy; existing business cases remain independently resolved/open |
| Condition disappeared without verification | Alert engine only | **No automatic business closure** | Re-run acceptance predicate | N/A | Explicit reconciliation and outcome check pass |

The draft editorial claim **“The automation sends so many notifications that nobody reads them. Which should remain?”** should be **qualified, not used literally as a universal fact**. Google SRE reports that high alert frequency can cause responders to skim or ignore alerts and describes a concrete internal over-alerting episode. citeturn14view0 But this is operational experience, not evidence that every team reaches a particular notification threshold or that “nobody” reads them.

A defensible rewrite is:

> **When alerts are frequent and many do not require action, important failures can be masked by noise. Keep human-interrupting alerts for conditions with a specific owner, a meaningful response, and a time-sensitive consequence; route lower-urgency but actionable work to a ticket/case queue, and remove signals that have no decision or action attached.**

That advice is closely aligned with Google's stated criteria that pages should be actionable and urgent and with its recommendation to favor symptom-oriented alerts. citeturn14view0 It should still be framed as practitioner guidance rather than a proven universal optimum.

One useful distinction is a three-layer state model:

```text
DETECTION STATE
    healthy / failing / unknown

CASE STATE
    open / assigned / mitigating / verification_pending / resolved

NOTIFICATION STATE
    pending / sent / failed / acknowledged-if-supported
```

A team should never infer:

```text
notification sent  => case resolved
alert condition gone => case resolved
job rerun succeeded => case resolved
```

Instead:

```text
case resolved => acceptance predicate rerun and passed,
                 with durable evidence
```

Prometheus's documentation is informative here because its own alert state semantics concern whether the alert expression remains active; it offers `keep_firing_for` specifically to mitigate flapping or apparent clearing due to data loss. citeturn19view4 Business resolution is a higher-level concept that the process owner must define.

## Reproducible evaluation and portable assets

A reasonable evaluation should not begin with historical incidents selected after the fact. Start with **known ground truth and deliberate failure injection** in a non-production fixture. This proposed evaluation has **not been performed**; no measured results are claimed.

The test universe should contain expected task instances such as:

```text
process_id × business_date × source_partition
```

and each injected failure should have a recorded truth label before monitoring is run.

A compact evaluation matrix is:

| Failure injection | Ground truth | Naïve job monitor expected result | Outcome-oriented monitor expected result |
|---|---|---|---|
| Scheduler never creates run | Failure | Potentially no job event | Deadline/absence check detects expected case |
| One of two source files missing | Failure | Job may succeed on present file | Manifest/file-count reconciliation detects |
| Yesterday's data delivered under today's processing timestamp | Failure | Fresh processing timestamp may look healthy | Business-period check detects |
| Duplicate replay of same batch | Failure | Both loads may technically succeed | Batch uniqueness/control-total check detects |
| Transform silently drops one customer segment | Failure | SQL execution succeeds | Segment/business invariant detects **if encoded** |
| Report refreshes but still points to \(D-1\) | Failure | Refresh job succeeds | Report-period black-box/metadata check detects |
| Report endpoint answers but intended principal lost access | Failure | Backend may look healthy | Representative access-path check detects |
| Notification transport fails | Detection may still be correct | Human receives nothing | Case remains open; channel sentinel detects transport problem |
| Two consecutive business dates fail same rule | Two failures | Naïve dedupe may represent one alert | Two case IDs retained, optionally one grouped notification |
| Source manifest itself contains wrong totals and all downstream systems match it | Failure | Likely healthy | **Expected blind spot unless independent invariant exists** |

That last case is essential. A reconciliation system is not omniscient. If its supposedly independent ground truth is wrong, agreement can certify the wrong answer. This is why the strongest business controls sometimes require a second independent source, ledger, aggregate, or domain constraint rather than simply comparing pipeline stages.

Recommended evaluation metrics are:

\[
\text{Failure Recall}
=
\frac{TP}{TP+FN}
\]

\[
\text{Alert Precision}
=
\frac{TP}{TP+FP}
\]

\[
\text{Median Detection Delay}
=
\operatorname{median}(t_{\text{detected}} - t_{\text{actionable}})
\]

\[
\text{Case Preservation Rate}
=
\frac{\text{distinct true failed cases represented}}
{\text{distinct true failed cases}}
\]

\[
\text{False Resolution Rate}
=
\frac{\text{cases marked resolved while acceptance still fails}}
{\text{all cases marked resolved}}
\]

\[
\text{Notification Delivery Rate}
=
\frac{\text{notification attempts confirmed by the defined transport criterion}}
{\text{notification attempts}}
\]

The final measure must not be substituted for failure recall or resolution. The alert transport can have 100% measured delivery while the business process has 0% recovery.

Google's SRE Workbook's precision/recall/detection/reset framework supports measuring detectors along multiple dimensions rather than simply counting alerts. citeturn15view3 Any anomaly or AI-based detector should be evaluated against the same hidden ground-truth fixture rather than judged from persuasive explanations.

For AI triage specifically, evaluate at least:

```text
Evidence faithfulness:
Did every factual statement in the triage summary come from supplied evidence?

Case identity:
Did it preserve all distinct affected batch IDs?

Action quality:
Did it recommend an action permitted by the runbook?

Abstention:
Did it say "insufficient evidence" when the fixture is ambiguous?

Resolution discipline:
Did it refrain from claiming resolution before deterministic acceptance passed?
```

There is no basis in the inspected evidence for delegating the acceptance predicate or autonomous repair to an LLM. A bounded assistant can instead consume deterministic case data and make the responder's context easier to digest.

**Portable asset — synthetic expected-work fixture.** Not production data; not executed.

```csv
process_id,case_id,business_date,source_partition,deadline_at,deadline_timezone,business_calendar,expected_file_count,expected_record_count,expected_net_sales_eur,outcome_owner,technical_owner,acceptance_version
sales_daily,sales_daily|2026-10-08|all,2026-10-08,all,2026-10-09T08:00:00,Europe/Prague,CZ_CORP_CALENDAR,2,100000,1250000.00,Sales Operations,Data Team,sales_daily/v1
sales_daily,sales_daily|2026-10-09|all,2026-10-09,all,2026-10-10T08:00:00,Europe/Prague,CZ_CORP_CALENDAR,2,102000,1285000.00,Sales Operations,Data Team,sales_daily/v1
```

**Portable asset — synthetic observations fixture.** Not executed.

```csv
case_id,observed_at,actual_file_count,actual_record_count,actual_net_sales_eur,warehouse_business_date,report_business_date,report_reachable
sales_daily|2026-10-08|all,2026-10-09T06:01:00Z,1,80000,1000000.00,2026-10-08,2026-10-08,true
sales_daily|2026-10-09|all,2026-10-10T06:01:00Z,1,81000,1015000.00,2026-10-09,2026-10-09,true
```

**Portable asset — deterministic reconciliation query.** ANSI-like SQL; minor timestamp/type syntax may require adaptation to the selected warehouse. Not executed.

```sql
/*
Purpose:
  Compare each expected business case with its observed downstream state.
  The query deliberately does NOT use scheduler/job success as a completion criterion.

Assumptions:
  - expected_work has one row per expected business case.
  - observations contains the latest verified observation for each case.
  - Exact equality is appropriate for this synthetic fixture.
    Real processes must define justified tolerances and mappings.
*/

WITH reconciliation AS (
    SELECT
        e.process_id,
        e.case_id,
        e.business_date,
        e.deadline_at,
        e.deadline_timezone,
        e.expected_file_count,
        e.expected_record_count,
        e.expected_net_sales_eur,

        o.observed_at,
        o.actual_file_count,
        o.actual_record_count,
        o.actual_net_sales_eur,
        o.warehouse_business_date,
        o.report_business_date,
        o.report_reachable,

        CASE
            WHEN o.case_id IS NULL THEN FALSE
            ELSE TRUE
        END AS observation_exists,

        CASE
            WHEN o.actual_file_count = e.expected_file_count THEN TRUE
            ELSE FALSE
        END AS file_count_ok,

        CASE
            WHEN o.actual_record_count = e.expected_record_count THEN TRUE
            ELSE FALSE
        END AS record_count_ok,

        CASE
            WHEN o.actual_net_sales_eur = e.expected_net_sales_eur THEN TRUE
            ELSE FALSE
        END AS control_total_ok,

        CASE
            WHEN o.warehouse_business_date = e.business_date THEN TRUE
            ELSE FALSE
        END AS warehouse_period_ok,

        CASE
            WHEN o.report_business_date = e.business_date THEN TRUE
            ELSE FALSE
        END AS report_period_ok,

        CASE
            WHEN o.report_reachable = TRUE THEN TRUE
            ELSE FALSE
        END AS report_access_ok

    FROM expected_work e
    LEFT JOIN observations o
        ON o.case_id = e.case_id
)

SELECT
    *,
    CASE
        WHEN observation_exists
         AND file_count_ok
         AND record_count_ok
         AND control_total_ok
         AND warehouse_period_ok
         AND report_period_ok
         AND report_access_ok
        THEN 'VERIFIED'
        ELSE 'OPEN_EXCEPTION'
    END AS outcome_state
FROM reconciliation
ORDER BY business_date, case_id;
```

**Portable asset — missing-run/deadline check.** `:now_timestamp` is supplied by an independent checker; not executed.

```sql
/*
Detect expected work that has passed its deadline without verified completion.

Important:
  Run this checker independently from the workflow being checked,
  or common-mode scheduler failure can defeat both workflow and monitor.
*/

SELECT
    e.process_id,
    e.case_id,
    e.business_date,
    e.deadline_at,
    e.outcome_owner,
    e.technical_owner
FROM expected_work e
LEFT JOIN verified_outcomes v
    ON v.case_id = e.case_id
   AND v.verification_status IN ('VERIFIED_ON_TIME', 'VERIFIED_LATE')
WHERE e.deadline_at < :now_timestamp
  AND v.case_id IS NULL;
```

**Portable asset — durable case schema.** Illustrative SQL; not executed.

```sql
CREATE TABLE automation_case (
    case_id                VARCHAR(255) PRIMARY KEY,
    process_id             VARCHAR(255) NOT NULL,
    business_date          DATE NOT NULL,
    source_partition       VARCHAR(255) NOT NULL,
    case_class             VARCHAR(100) NOT NULL,
    case_state             VARCHAR(40) NOT NULL,
    opened_at              TIMESTAMP NOT NULL,
    assigned_owner         VARCHAR(255),
    last_checked_at        TIMESTAMP,
    verified_at            TIMESTAMP,
    deadline_at            TIMESTAMP NOT NULL,
    resolution_summary     VARCHAR(2000),
    acceptance_version     VARCHAR(100) NOT NULL
);

CREATE TABLE automation_case_evidence (
    case_id                VARCHAR(255) NOT NULL,
    evidence_type          VARCHAR(100) NOT NULL,
    observed_at            TIMESTAMP NOT NULL,
    evidence_reference     VARCHAR(2000) NOT NULL,
    check_name             VARCHAR(255) NOT NULL,
    check_status           VARCHAR(40) NOT NULL,
    PRIMARY KEY (case_id, evidence_type, observed_at),
    FOREIGN KEY (case_id) REFERENCES automation_case(case_id)
);

CREATE TABLE notification_attempt (
    case_id                VARCHAR(255) NOT NULL,
    channel                VARCHAR(100) NOT NULL,
    attempted_at           TIMESTAMP NOT NULL,
    transport_state        VARCHAR(40) NOT NULL,
    transport_reference    VARCHAR(2000),
    PRIMARY KEY (case_id, channel, attempted_at),
    FOREIGN KEY (case_id) REFERENCES automation_case(case_id)
);
```

**Portable asset — bounded AI triage contract.** This is a proposed prompt specification, not a tested intervention.

```text
SYSTEM ROLE
You are an incident-triage assistant. You do not determine business success,
change production systems, close cases, send notifications, or execute recovery.

AUTHORITATIVE INPUTS
1. Expected-work record.
2. Deterministic check results.
3. Case history.
4. Approved runbook excerpts.
5. Observed lineage/log references.

REQUIRED OUTPUT
Return JSON with:
{
  "case_ids": [],
  "summary": "",
  "failed_checks": [],
  "confirmed_facts": [],
  "hypotheses": [
    {
      "hypothesis": "",
      "supporting_evidence": [],
      "contradicting_evidence": [],
      "confidence": "low|medium|high"
    }
  ],
  "recommended_human_actions": [],
  "missing_information": [],
  "resolution_status": "NOT_AUTHORIZED_TO_DETERMINE"
}

RULES
- Never infer that a business result is correct from job success,
  HTTP acceptance, notification delivery, or absence of an error.
- Never merge distinct case_ids.
- Every confirmed fact must cite an input evidence item.
- A hypothesis must be labeled as a hypothesis.
- If supplied evidence is insufficient, state that explicitly.
- Never claim the incident is resolved.
- Never recommend an action not present in the supplied approved runbook
  unless clearly labeled "requires human design/review".
```

The evaluation should run **manual baseline, deterministic controls, and any proposed anomaly/AI layer against the same fixture and failure injections**. Otherwise comparisons become meaningless because each approach gets a different task.

A particularly useful experiment would compare:

```text
A. Manual checklist
B. Scheduler-native monitoring only
C. Independent expected-work + deterministic reconciliation
D. C + black-box report check
E. D + AI triage summary
```

Measure detection recall, precision, detection delay, distinct-case preservation, false resolutions, operator time per case, and unsupported AI assertions. Operator time would require an actual human study; because none was performed here, there is no claim that option E reduces effort.

## Claim-to-source register

The table separates research, provider documentation, reported practice and editorial interpretation. **“Supported” means supported within the exact source scope, not universally proven.**

| Claim relevant to the decision | Primary/original source and exact section | Source date | Evidence type and exact scope | Status | Editorial advice | Direct URL / DOI |
|---|---|---:|---|---|---|---|
| Correctness properties that require application knowledge should be checked end to end rather than assumed from lower layers. | Saltzer, Reed & Clark, *End-to-End Arguments in System Design*, abstract and “End-to-end caretaking.” citeturn20view0turn20view1 | 1984 | Foundational system-design argument; examples include file transfer, acknowledgment and duplicate suppression. Not a data-pipeline experiment. | **Supported as design principle** | **Keep, qualify** as architectural reasoning rather than causal evidence. | https://doi.org/10.1145/357401.357402 |
| A successful workflow-run status need not mean the desired business result happened. | Apache Airflow 3.3.2, “Dag Run Status.” Success is derived from leaf task states; docs warn an `all_done` leaf can make a run succeed despite an intermediate failure. citeturn22view0 | v3.3.2, current at inspection | Provider documentation; exact Airflow semantics only. | **Supported for Airflow semantics** | **Keep.** Do not generalize identical semantics to every orchestrator. | https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dag-run.html |
| Internal health signals can disagree with application-visible failure. | Huang et al., *Gray Failure*, abstract and §2.2 “Under the radar of failure detectors.” citeturn22view1turn20view3 | May 2017 | Primary company/research paper reporting experience with Azure/cloud-scale production incidents; not controlled. | **Supported in source scope** | **Keep, qualify** transfer to business automation as structural analogy. | https://doi.org/10.1145/3102980.3103005 |
| HTTP success or another internal success signal can coexist with wrong content; end-to-end checks can detect conditions lower-level status misses. | Google SRE, “Black-Box Versus White-Box” and “Errors.” citeturn14view0 | SRE book web edition, copyright 2017 | Attributable Google operational guidance; practitioner evidence, not controlled study. | **Supported as guidance** | **Keep.** Strong direct analogy to accepted API/job vs wrong report. | https://sre.google/sre-book/monitoring-distributed-systems/ |
| Simple explicit testing can prevent important failure classes. | Yuan et al., OSDI, study of 198 randomly selected user-reported failures; report that >30% of catastrophic failures in their corpus would have been preventable if checker findings were fixed. citeturn22view2 | 2014 | Empirical retrospective failure study across Cassandra, HBase, HDFS, Hadoop MapReduce, Redis; checker validation. | **Supported in corpus; transfer conditional** | **Qualify heavily.** Never reuse the >30% as an expected pipeline-observability effect. | https://www.usenix.org/conference/osdi14/technical-sessions/presentation/yuan ; bibliographic DOI: https://doi.org/10.5555/2685048.2685068 |
| Tracing is valuable for correlating activity across distributed components, but is not itself proof of business correctness. | Sigelman et al., *Dapper*, abstract. citeturn22view3 | 2010 | Primary Google technical report describing production distributed tracing and its operational uses. The “not proof” part is this dossier's inference. | **Supported for tracing utility; completion limitation is interpretation** | **Keep with distinction.** Use lineage/tracing for diagnosis, not as acceptance predicate. | https://research.google/pubs/dapper-a-large-scale-distributed-systems-tracing-infrastructure/ |
| Business/event time must not be silently conflated with processing/observation time. | Akidau et al., *The Dataflow Model*, abstract; event-time ordering, out-of-order inputs and correctness/latency trade-offs. citeturn22view4 | 2015 | Research/system model for massive-scale unbounded processing; broader than daily batch. | **Conditionally supported** | **Keep, qualify.** Mapping `business_date` vs `observed_at` is dossier interpretation. | https://doi.org/10.14778/2824032.2824076 |
| Silence/missing execution needs assumptions external to the missing component; a deadline is therefore part of the detection model. | Chandra & Toueg, *Unreliable Failure Detectors for Reliable Distributed Systems*, abstract. citeturn13search39 | 1996 | Formal distributed-systems theory for asynchronous systems with crash failures and consensus, not business workflow monitoring. | **Conceptually supportive, highly conditional** | **Qualify.** Use only to motivate explicit timing assumptions, not to prescribe monitoring mechanics. | https://doi.org/10.1145/226643.226647 |
| High-volume, low-actionability paging can mask important problems and consume responder attention. | Google SRE “Why Monitor?”, alerting philosophy, and Bigtable over-alerting case. citeturn14view0 | SRE book web edition, copyright 2017 | Provider-reported operational experience and guidance from Google; no controlled general effect size. | **Supported as reported practice; universal causal claim unresolved** | **Rewrite** “nobody reads alerts” to conditional language about noise and actionability. | https://sre.google/sre-book/monitoring-distributed-systems/ |
| Alert quality should be evaluated across precision, recall, detection time and reset time. | Google SRE Workbook, “Alerting Considerations.” citeturn15view3 | 2018 book / current web edition | Provider/SRE design framework, illustrated for service SLO alerting. | **Supported as framework** | **Keep**, adapt denominators to expected business cases. | https://sre.google/workbook/alerting-on-slos/ |
| Synthetic/black-box probes have incomplete coverage and can succeed while real-user failures remain. | Google SRE Workbook, “Generating Artificial Traffic.” citeturn15view4 | 2018 book / current web edition | Practitioner design guidance for services; explicitly discusses partial coverage. | **Supported as limitation** | **Keep.** Prevent “end-to-end probe” from being described as absolute proof. | https://sre.google/workbook/alerting-on-slos/ |
| Similar alerts can be grouped into one notification while preserving affected-instance visibility. | Prometheus Alertmanager, “Grouping.” citeturn18view4 | Current documentation, inspected 2026-10-10 | Provider documentation of Alertmanager behavior. | **Supported for product behavior** | **Keep as implementation example.** General design: group notifications, not business cases. | https://prometheus.io/docs/alerting/latest/alertmanager/ |
| Aggressive alert limiting can hide new affected entities. | Prometheus Alertmanager, “Alert limits”: new alerts are dropped when per-alertname limit is reached; dropped count is exposed by a metric. citeturn19view3 | Current documentation, copyright 2026 | Provider documentation. | **Supported** | **Keep as counterevidence** to naïve noise suppression. | https://prometheus.io/docs/alerting/latest/alertmanager/ |
| Debounce/flapping controls change alert firing semantics, not business-resolution semantics. | Prometheus alerting rules, `for`, `keep_firing_for`, alert state and notification sections. citeturn19view4 | Current documentation, inspected 2026-10-10 | Provider documentation; business-resolution distinction is dossier interpretation. | **Supported/interpretive** | **Keep with qualification.** | https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/ |
| Native freshness checks can detect delayed sources/models but freshness alone is not completeness or correctness. | dbt `source`/`freshness` documentation. Current page says stale sources produce warning/error and nonzero exit; recommends scheduled retention of freshness results. citeturn18view7 | Last updated 2026-09-16 | Provider documentation. “Not completeness” is logical limitation, not a dbt deficiency claim. | **Supported/conditional** | **Keep.** Position freshness as one acceptance dimension. | https://docs.getdbt.com/reference/commands/source |
| Deterministic data/business assertions can be encoded as data tests. | dbt, “Add data tests to your DAG,” Overview. citeturn18view2 | Current docs inspected 2026-10-10 | Provider documentation: SQL assertions including built-ins and custom business logic. | **Supported** | **Keep**, but state tests only validate encoded assertions. | https://docs.getdbt.com/docs/build/data-tests |
| dbt's stored test failures should not automatically be used as durable historical case storage. | dbt, “Storing data test failures”: results for a test replace previous failures. citeturn18view3 | Current docs inspected 2026-10-10 | Provider behavior. | **Supported** | **Keep.** Use separate durable incident/case history when required. | https://docs.getdbt.com/docs/build/data-tests |
| A downstream report/application and owner can be represented separately from transformation nodes. | dbt exposures: downstream dashboard/application, dependencies and required owner. citeturn18view1 | Current docs inspected 2026-10-10 | Provider feature documentation. | **Supported** | **Keep as optional implementation example**, not a universal requirement. | https://docs.getdbt.com/docs/build/exposures |
| Calendar/timezone semantics can cause deadline errors; explicit timezone handling is required. | Airflow 3.3.2 “Time Zones,” including UTC storage, IANA zones, DST ambiguity and timezone-data updates. citeturn18view6turn19view2 | v3.3.2, current at inspection | Provider documentation. | **Supported** | **Keep.** Applies materially to both Czech and multi-timezone US processes. | https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/timezone.html |
| “Automation sends so many notifications that nobody reads them.” | Closest evidence: Google SRE reports frequent pages can cause responders to skim/ignore alerts and gives an over-alerting Bigtable narrative. citeturn14view0 | Web edition copyright 2017 | Practitioner report; no universal frequency threshold or small-team controlled effect. | **Literal claim unsupported/overstated** | **Rewrite:** “High-volume, low-actionability alerting can mask important failures; retain alerts that drive a specific timely action.” | https://sre.google/sre-book/monitoring-distributed-systems/ |
| AI incident triage improves resolution in this scoped process. | No suitable controlled primary evidence established in this research pass. | — | Evidence gap. Vendor claims were not treated as causal evidence. | **Unresolved** | **Do not claim.** Present AI triage only as a bounded option to evaluate. | — |

A useful hierarchy emerges from the register. Saltzer and Chandra/Toueg are foundational/theoretical; Gray Failure and Yuan et al. provide real-system observations with different scopes; Dapper and Google's SRE material document primary company practice; Airflow, dbt, and Prometheus document concrete product semantics. None alone proves that the proposed expected-work ledger reduces incidents in a small Czech or US data team. The recommendation comes from **triangulating compatible mechanisms while preserving those scope limits**.

## Contradictions, unresolved gaps, article angles, and freshness triggers

Several apparent contradictions should remain visible rather than being smoothed away.

**End-to-end checking is necessary for some properties, but an end-to-end probe is not omniscient.** Saltzer et al. motivate moving checks to endpoints where application knowledge exists. citeturn20view0turn20view1 Yet Google's later SRE guidance warns that synthetic requests cover only part of a complex service's possible states and can succeed while real users fail. citeturn15view4 The right conclusion is not “one end-to-end test proves success”; it is “verify the property at the highest practical semantic layer, and state exactly what that check covers.”

**More alerts increase possible detection coverage, but can degrade actionability.** Google reports operational harm from noisy paging and urges low-noise, actionable alerts. citeturn14view0 Prometheus supplies grouping, inhibition and limiting mechanisms to manage volume, but its documentation also makes clear that limits can literally drop new alerts. citeturn19view3 The reconciliation ledger therefore has to be upstream of notification suppression: suppress pages if appropriate, not affected-case existence.

**Grouping is not deduplication of reality.** Two batches with the same root cause can be one incident for coordination and two business cases for reconciliation. Prometheus's “one page while still seeing exactly which instances were affected” is a useful implementation analogy. citeturn18view4 A design that compresses both into one undifferentiated alert saves visual noise at the cost of losing reconciliation state.

**A deterministic monitor can be more interpretable than anomaly detection, but only for things the team can specify.** A missing expected batch, wrong business date, duplicate primary key, or control-total difference has relatively clear ground truth. Novel semantic corruption may not. The unresolved design question is therefore not “rules or AI?” but “which important failures have specifiable invariants, and what residual uncertainty remains after those checks?”

**The expected-work ledger becomes infrastructure and can itself fail.** If today's expectation is generated by the same broken scheduler as today's workflow, missing-run detection can disappear with the job. Pre-materializing work or using a separate mechanism reduces that common mode but introduces another artifact to maintain. This trade-off is architectural reasoning; I found no controlled comparison establishing an optimum independence boundary.

**Business acceptance criteria decay.** A once-correct reconciliation rule can become wrong when source schemas, filtering rules, accounting treatments, BI definitions or calendars change. The checks therefore need an `acceptance_version`, named owner and review trigger. This maintenance burden is the principal hidden cost of outcome observability. Google SRE explicitly warns more generally that monitoring systems themselves become fragile and costly when complexity accumulates. citeturn14view0

**A source manifest is evidence, not ultimate truth.** If the exporter computes the wrong count and the pipeline faithfully reproduces it, source-to-target reconciliation passes. High-value processes may need an independent business control—for example a ledger balance, expected set of entity IDs, prior-system total, or invariant that the producing system cannot falsify through the same defect. There is no universal control that proves arbitrary business semantics.

**A repaired incident and a met deadline are different facts.** `VERIFIED_LATE` is valuable because it permits the operational incident to close without erasing the service failure. This distinction is important for trend analysis: otherwise a system that is repeatedly repaired after the deadline may appear perfectly reliable if only current state is queried.

**Receipt is not resolution.** This research did not establish a robust primary-source basis for treating any generic notification-provider acknowledgment as human receipt, and no such claim is made. More importantly, even confirmed human acknowledgment is conceptually distinct from business repair. The durable case and acceptance evidence should therefore remain authoritative.

**Software-operations alert-fatigue evidence is weaker than the confidence with which the topic is often discussed.** Google's material contains specific internal narratives and explicit guidance, but it is not a randomized study. I did not find, within this bounded research pass, a controlled study that provides a transferable effect size for “number of alerts versus probability a small data team misses a daily delivery.” Healthcare alarm-fatigue results were deliberately not imported because the user requested that other-domain effects not be transferred without justification.

**Anomaly-detection evidence is also insufficiently task-specific here.** Broad anomaly-detection literature exists, but I did not establish a directly relevant controlled comparison for deterministic expected-work reconciliation versus statistical anomaly detection in a small daily-data operation. Accordingly, no claim is made that anomaly detection improves or worsens operational outcomes in this exact setting.

**AI triage remains an evidence gap.** Vendor AIOps/incident products now advertise AI triage and root-cause capabilities, but marketing claims are not evidence of causal operational improvement. No such claims were used. For this dossier, the reasonable deployment boundary is an assistant that summarizes deterministic evidence and approved runbooks while humans retain action and resolution authority.

**No firsthand execution was performed.** No production system was inspected or changed, no messages were sent, no alerting provider was tested, no customer data was used, and no failure-injection experiment was run. The worked cases, thresholds, counts, deadlines, SQL fixtures and AI contract in this dossier are explicitly synthetic.

**Source-access gaps:** the core cited papers and provider pages above were publicly inspectable. The Saltzer, Gray Failure, and Yuan PDF material used here was visually inspected where relevant. Official Google publication pages were used for Dapper and Dataflow. During the final source pass, ACM pages associated with broad anomaly-detection/base-rate literature did not return usable content, so they were not made load-bearing sources for recommendations. A targeted search for primary PagerDuty notification-delivery semantics did not yield a suitable source, so the dossier deliberately avoids equating transport status with human receipt. These absences should not be silently replaced with vendor-blog claims.

Distinct candidate article angles supported by the dossier include:

| Candidate angle | Core proposition | Evidence strength |
|---|---|---|
| **Your automation isn't done when the job turns green** | Define completion at the business endpoint, not scheduler layer | Strong conceptual + documented product behavior |
| **The missing-run paradox** | A workflow cannot reliably report its own nonexistence; expected work must be defined outside the event stream | Strong architecture logic; formal theory is only indirect support |
| **Alerts are messages; incidents are cases** | Preserve case identity and resolution evidence independently of notification routing | Strong design synthesis; provider features support mechanics |
| **How to know yesterday's report is actually ready** | Business date + completeness + invariants + reconciliation + final artifact is more meaningful than row-count presence | Strong practical synthesis; effect on business outcomes unmeasured |
| **Which automation alerts should survive?** | Keep human-interrupting signals that are actionable, time-sensitive and owned; group delivery without deleting distinct cases | Strong practitioner guidance, weaker causal research |
| **Fresh is not necessarily complete** | Freshness, completeness, correctness and business-period identity are distinct properties | Strong logical distinction + native-tool documentation |
| **AI can be the incident clerk, not the judge** | Let AI summarize evidence; keep deterministic acceptance and human recovery authority | Prudent design position; operational benefit currently unresolved |
| **Recovery needs a test, not a thumbs-up** | A rerun is an intervention; successful reconciliation is evidence of recovery | Strong conceptual synthesis |
| **Two failures, one notification** | Reduce notification noise by grouping without collapsing affected business cases | Strong implementation support from alert-manager semantics |

Useful reusable assets from this dossier are already provided inline: the expected-work fixture, observation fixture, deterministic reconciliation query, missing-run query, case/evidence schemas, alert-to-action inventory, failure-injection matrix and bounded AI-triage contract. **All are proposed/synthetic and unexecuted.**

The evidence and implementation guidance should be rechecked when any of the following occurs:

- **Process contract changes:** new deadline, source, report, business calendar, control total, tolerance, or owner. This is the highest-priority freshness trigger because a technically functioning monitor can become semantically obsolete.
- **Airflow upgrades:** recheck DAG-run, data-interval, catchup/backfill and timezone semantics against the deployed version; this dossier inspected Airflow 3.3.2. citeturn22view0turn18view6
- **dbt upgrades:** recheck freshness, data-test, stored-failure and exposure behavior; the inspected source-command page was last updated 2026-09-16 and applies to dbt v2 behavior described there. citeturn18view7
- **Prometheus/Alertmanager changes:** recheck grouping, inhibition, limits and firing-state semantics, especially if case identity is mapped into alert labels. citeturn18view4turn19view3turn19view4
- **Timezone or business-calendar changes:** update IANA timezone data and process calendars before deadline logic is trusted. Airflow specifically notes the need to keep underlying timezone information current when daylight-saving regulations change. citeturn19view2
- **Alert volume changes materially:** remeasure precision, recall, case-preservation rate and responder action rather than assuming yesterday's routing policy remains appropriate. Google's guidance explicitly treats monitoring as something that evolves with architecture and operational behavior. citeturn14view0
- **New controlled research appears on LLM incident triage or software-operations alert fatigue:** these are the two most consequential evidence gaps in the current dossier.
- **The monitoring stack becomes harder to understand than the process it monitors:** that is a design trigger to simplify, not automatically to add another observability layer. Google SRE's long-running recommendation is to keep the critical monitoring/alerting path simple and comprehensible. citeturn14view0

The core decision rule survives all of those implementation changes: **first define the expected business case; then observe and reconcile the resulting business state; let jobs, traces and notifications serve as evidence and transport around that case rather than substitutes for it.** This conclusion is strongly supported as an architecture pattern by end-to-end system design, differential-observability research, SRE practice, and current orchestration/monitoring semantics, while its quantitative operational benefit for a small Czech or US data team remains an appropriate subject for the proposed controlled failure-injection evaluation rather than an established result. citeturn20view0turn22view1turn14view0turn22view0