# Data Quality Before Adding AI: Evidence Dossier for a Decision-Maker

**Research snapshot:** 2026-10-10, Europe/Prague  
**Decision supported:** Whether to fix data, constrain an AI analytics pilot, or postpone it  
**Working environment:** Small professional data team; ingestion pipelines → cloud analytical warehouse → dbt-style transformations/semantic definitions → AI analytical assistant  
**Concrete decision used throughout:** “Why did monthly revenue or active-customer count change?”  
**Evidence boundary:** Public primary sources, official/versioned documentation, published evaluation evidence, and synthetic fixtures only. No private company data or production systems were accessed.

## Decision answer and key uncertainties

### Bottom line

**Do not wait for all data to become “perfect” before adding AI. But do not let an AI assistant decide whether the data underlying a material metric is trustworthy.**

For monthly revenue or active-customer explanations, the minimum sensible architecture is:

> **authoritative business definition → deterministic data-quality gates → versioned lineage and permissions → deterministic query/calculation → AI interpretation → human judgment where needed**

The most dangerous defects are not necessarily spectacular pipeline failures. They are defects that leave a **plausible, executable, well-formatted number**: missing events partly offset by duplicates, join fan-out, stale-but-reasonable data, a silent metric-definition change, or an AI query that selects the wrong valid table. Those failures can survive syntax checking and produce fluent explanations.

This distinction matters because current evidence does **not** support treating model capability as equivalent to analytical-system reliability. Spider 2.0 contains 632 enterprise-like text-to-SQL workflow problems, often over databases with more than 1,000 columns; its published o1-preview-based agent solved 21.3% of tasks, versus much higher results on older benchmarks. [1](https://arxiv.org/abs/2411.07763) A much newer Data Agent Benchmark revision, posted October 7, 2026, reports 104 questions across 17 datasets and four database systems, with its best evaluated agent at 57% pass@1. [2](https://arxiv.org/abs/2603.20576) These are **capability benchmarks, not estimates of failure rates in your production workflow**. Moreover, the evaluation artifacts themselves can be wrong: an ACL study found question and gold-SQL noise in BIRD, and a 2026 audit reported annotation errors in 52.8% of the audited BIRD Mini-Dev examples and 62.8% of the Spider 2.0-Snow examples for which gold queries were publicly available. [3](https://aclanthology.org/2024.acl-short.34/)

So the practical decision is not:

> “Is our data good enough for AI?”

It is:

> **“For this decision, can we establish enough independent evidence that the metric, source, transformations and permissions are fit before AI interprets the result?”**

My recommendation is therefore **a staged AI pilot behind an evidence gate**, unless one of three blocking conditions exists:

| Decision state | What it means |
|---|---|
| **Proceed with a gated pilot** | The metric definition and grain are authoritative and versioned; the period is demonstrably complete enough; key reconciliation, duplication, schema and grain checks pass; the relevant lineage path is known; assistant permissions are tested; and the deterministic numerical result is reproducible. |
| **Constrain the pilot** | Data is usable for some questions but not others, or the period is still open. Let the assistant summarize validated evidence, surface test failures and request clarification, but prevent it from issuing an unqualified “why” explanation. |
| **Postpone this use case** | The team cannot establish the authoritative metric definition, cannot independently assess source completeness, has material unexplained reconciliation differences, lacks enough lineage to isolate the affected path, or cannot prove permission enforcement. |

This is a **decision-specific fitness standard**, not a recommendation to remediate the whole warehouse.

### What is actually uncertain

There is little strong empirical literature measuring an effect such as “5% missing warehouse events cause X% more hallucinations in an AI analyst.” The strongest evidence relevant here comes from three different categories that should not be conflated:

**Empirical research** shows that complex data-agent and text-to-SQL workflows remain fallible and that benchmark labels themselves can be noisy. [1](https://arxiv.org/abs/2411.07763)

**Provider and standard documentation** establishes what concrete controls can observe: freshness thresholds, semantic entities and grain, row/column permissions, message-redelivery semantics, lineage run IDs, code versions and field transformations. [4](https://docs.getdbt.com/docs/deploy/source-freshness)

**The defect-to-risk mapping in this dossier is an engineering interpretation** of those mechanisms. It is not presented as measured causality or as a universal effect size.

A further semantic warning is important: for this use case, “explain why revenue changed” should normally mean **account for the arithmetic movement**—for example price, volume, customer, product, FX or refund components. It should not silently become a causal claim such as “the campaign caused revenue growth.” Descriptive warehouse data alone generally does not establish such causality.

## Evidence synthesis around the decision

### The defects that matter

A useful mental model is to separate four questions:

1. **Did the world get captured correctly?**
2. **Did the data platform preserve and transform it correctly?**
3. **Are we asking the intended business question over the intended version of that data?**
4. **Did the AI system retrieve, calculate and describe that evidence correctly?**

A model can fail even with perfect data. Perfect model reasoning cannot recover events that never reached the observable system.

The table below is a decision analysis rather than a measured ranking of defect frequencies.

| Defect class | How it can corrupt monthly revenue / active customers | What can establish it deterministically | What AI can reasonably help with | What AI cannot establish from the corrupted dataset alone |
|---|---|---|---|---|
| **Upstream capture defect** | Transactions or activity never recorded; revenue/customers understated | Source-system control totals, source audit log, event-ID manifest, independent accounting/billing balance | Notice discrepancies when an independent control is supplied; investigate likely affected sources | Prove that an event that left no trace ever occurred |
| **Late or missing events** | A closed-looking month is incomplete; comparisons move as events arrive | Watermarks, expected partitions/files, event-time vs ingest-time distribution, source-vs-warehouse reconciliation | Identify unusual lag patterns and summarize affected periods | Know completeness from `MAX(loaded_at)` alone |
| **Duplicate / replayed ingestion** | Additive revenue inflated; counts may or may not inflate depending on `COUNT(*)` vs `COUNT(DISTINCT ...)` | Business/event-key uniqueness, idempotency tests, source-vs-target counts and amounts | Discover repeated keys and propose deduplication logic | Decide which repeated records are legitimate without business semantics |
| **Schema drift** | Values move column, become null, change type or are mapped to stale fields | Schema contract/diff, required-column and type checks | Suggest mapping changes and explain schema diffs | Determine intended business semantics solely from a renamed field |
| **Transformation / grain defect** | Join fan-out, incorrect many-to-many join, double aggregation or wrong temporal join yields plausible wrong totals | Primary-grain tests, relationship/cardinality checks, conservation/reconciliation checks, known-answer fixtures | Inspect SQL, identify suspicious joins, suggest tests | Prove that executable SQL has the intended business grain |
| **Temporal freshness** | Assistant answers with yesterday’s or last week’s state as though current | Explicit freshness SLA and snapshot/as-of timestamp | Explain a red freshness signal | Equate a fresh timestamp with complete data |
| **Semantic-definition change** | “Revenue” changes from bookings to recognized revenue, refund timing changes, “active” goes from 30 to 28 days, etc. | Versioned metric contract and effective dates | Compare definitions/changelogs and flag differences | Decide which definition management intended |
| **Retrieval / SQL / model error** | Wrong table, wrong metric version, wrong date condition, wrong arithmetic, unsupported narrative | Reference queries, result comparison, version/source IDs, deterministic computation and tests | Query planning, SQL generation, interpretation | Make fluent prose evidence of correctness |
| **Permission or source-fidelity failure** | Assistant sees rows/columns the user should not see, or explanation cites a different artifact/version than the calculation used | Execution under real assistant/user identity, row/column access tests, query/run logs, source/result hashes | Report the policy outcome and attach evidence | Replace warehouse authorization with a prompt instruction |

Two operational lessons follow.

First, **freshness is not completeness**. dbt's source-freshness mechanism evaluates source data against freshness criteria/SLA, and current dbt documentation explicitly notes that ordinary `dbt build` does not automatically include source-freshness checks; the team must configure them in the job. [5](https://docs.getdbt.com/docs/deploy/source-freshness) A table can therefore be recently updated while still missing a partition, tenant, event type or subset. “Fresh” answers one question—how recently something arrived—not whether everything expected arrived.

Second, **counts alone are weak reconciliation evidence**. One missing event plus one replayed event can leave row count unchanged. A better gate combines distinct business keys, amount/control totals, expected partitions and—where feasible—source-side manifests.

This is also why declaring data assumptions explicitly is more defensible than asking an LLM to intuit them. Deequ was developed to express assumptions such as missingness and categorical constraints declaratively and automatically validate them at scale; its authors report production use cases at Amazon. [6](https://www.amazon.science/publications/deequ-data-quality-validation-for-machine-learning-pipelines) That research concerned ML data-quality validation rather than an LLM analytical assistant, but the design principle transfers directly: **known invariants should be tests, not prompts**.

### Grain and semantics should come before the model

Before asking “why did revenue change?”, the team needs a contract resembling:

> **recognized_net_revenue_eur v3**  
> Grain: one row per recognized invoice line  
> Time dimension: service/recognition month  
> Formula: gross recognized amount − applicable refunds/credits  
> Currency: EUR after specified FX treatment  
> Inclusion/exclusion rules: explicit  
> Effective definition version: explicit

For “active customers” the corresponding unresolved choices are often even more consequential: natural person vs account vs organization; event qualifying as activity; rolling-window length; timezone; suspended users; identity merges; anonymous activity; and whether the measure is a point-in-time state or “had activity during period.”

Modern semantic-layer tooling can encode part of this structure. dbt's current Semantic Layer documentation makes entities, dimensions and metrics explicit, and its example specifically declares the fact-table grain as one row per order; semantic models map to dbt SQL models and define entities used for joining. [7](https://docs.getdbt.com/guides/sl-qs)

That is useful infrastructure, but it is not an oracle. A semantic layer can consistently propagate a **wrong** definition. What matters is ownership, versioning and effective dates as much as centralization.

### Lineage is a diagnostic index, not proof of truth

Lineage is high leverage because it turns:

> “Why is this number wrong?”

into something closer to:

> “Which source, run and transformation capable of influencing this field changed between the two snapshots?”

OpenLineage distinguishes runtime events from design-time job/dataset metadata and can represent input/output datasets, run identifiers, source-code location/version and SQL-related metadata. [8](https://openlineage.io/docs/spec/object-model/) Its column-lineage facet can distinguish direct transformations and aggregations from indirect dependencies such as joins, filters, grouping, windows and conditionals. [9](https://openlineage.io/docs/spec/facets/dataset-facets/column_lineage_facet/)

For a revenue discrepancy, that enables a useful narrowing strategy:

**revenue metric → fact revenue column → transforming model/run → join/filter dependencies → staging datasets → ingestion/source**

That is materially better than scanning every data-quality dashboard in the warehouse.

But lineage itself does **not** tell you whether the original source was complete or whether finance's definition is correct. It is a map of derivation, not a ground truth certificate. That limitation should be explicit.

### Replay is not an exotic edge case

Duplicate protection deserves to be a standard control in many ingestion architectures because common messaging systems explicitly expose at-least-once behavior. AWS documents that standard SQS delivery can result in the same message being received again and recommends idempotent processing. [10](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html) Google Pub/Sub likewise documents at-least-once delivery as the default, notes that redelivery can occur even after acknowledgment, and recommends idempotent subscribers; exactly-once delivery is an optional capability rather than the default behavior described there. [11](https://docs.cloud.google.com/pubsub/docs/subscription-overview)

This does **not** imply that every warehouse ingestion pipeline contains duplicates. It establishes that replay is an expected design condition in representative, widely used ingestion infrastructure, so additive metrics should not depend on wishful uniqueness.

### Permissions belong below the AI layer

For data access, “the system prompt says not to reveal PII” is the wrong control boundary.

A warehouse should evaluate the assistant under the actual principal whose entitlements are intended to apply. BigQuery, as one representative implementation, supports row-level policies that filter rows according to qualifying users and column-level policy tags checked when column data is queried. [12](https://docs.cloud.google.com/bigquery/docs/row-level-security-intro?utm_source=chatgpt.com)

There is a subtle transformation risk as well: BigQuery documents that when query results are copied into a new table, column policy tags are **not automatically propagated in most cases**. [13](https://docs.cloud.google.com/bigquery/docs/column-level-security?utm_source=chatgpt.com) That is provider-specific, but it illustrates the broader reason to test permissions at the final assistant-accessible object rather than infer them from the source table.

For Czech/EU teams, this intersects with existing GDPR obligations when personal data is involved. Article 5 requires personal data to be adequate/relevant/limited to what is necessary and accurate/kept up to date where necessary. [14](https://eur-lex.europa.eu/eli/reg/2016/679/oj) That makes indiscriminately feeding raw customer-level context to an assistant particularly hard to defend when an aggregate or pseudonymized representation is sufficient. This does not materially change the technical diagnosis between Prague and the United States; it changes the governance pressure around unnecessary personal-data exposure.

## Reliable system design and analyst baseline

### Separate five different capabilities

An “AI analyst” is usually several systems hidden behind one chat box. Treating them separately makes failures diagnosable.

| Layer | Appropriate responsibility | Evidence required before trusting it |
|---|---|---|
| **Retrieval / semantic resolution** | Find the authorized metric, metadata, schema and documentation | Exact object/version retrieved; ambiguity explicitly resolved |
| **SQL or code generation/execution** | Produce and run the query | Executed query preserved; database/runtime errors surfaced; output compared with reference cases |
| **Deterministic calculation** | Percent changes, decompositions, reconciliations | Calculation in SQL/code, not free-form mental arithmetic; units visible |
| **Model interpretation** | Turn validated figures and diagnostic evidence into a readable explanation | Every material factual statement traceable to the evidence envelope |
| **Human judgment** | Decide materiality, intended semantics, whether descriptive attribution is enough, whether causality is being implied, whether remediation is justified | Named owner/reviewer and review record for material decisions |

The LLM is most valuable toward the **right-hand side**: querying, navigation, hypothesis generation and explanation. The farther left the responsibility moves toward defining ground truth, the weaker the case for probabilistic control.

Current data-agent benchmarks support caution rather than fatalism. Spider 2.0 demonstrates that enterprise-like database environments involving extensive metadata and project documentation remain much harder than traditional text-to-SQL tasks. [1](https://arxiv.org/abs/2411.07763) DAB v3, revised only three days before this research snapshot, similarly reports substantial remaining failure on end-to-end data questions. [2](https://arxiv.org/abs/2603.20576) Neither result says a carefully constrained assistant cannot be valuable. They argue against making “the model looked impressive in a demo” a release criterion.

### The evidence envelope I would require

For every material explanation, the assistant should receive—or the system should retain—an envelope along these lines:

```json
{
  "decision": "explain_monthly_revenue_change",
  "metric": {
    "name": "recognized_net_revenue_eur",
    "definition_version": "rev_v3",
    "grain": "one row per recognized invoice line",
    "time_dimension": "recognition_month",
    "currency": "EUR"
  },
  "snapshot": {
    "as_of_utc": "2026-11-03T08:00:00Z",
    "period_status": "closed"
  },
  "source_controls": {
    "expected_event_count": 123456,
    "expected_amount_eur": 9876543.21,
    "reconciliation_status": "PASS"
  },
  "deterministic_tests": {
    "freshness": "PASS",
    "required_partitions": "PASS",
    "business_key_uniqueness": "PASS",
    "schema_contract": "PASS",
    "fact_grain": "PASS",
    "join_cardinality": "PASS",
    "source_to_fact_reconciliation": "PASS"
  },
  "lineage": {
    "pipeline_run_id": "RUN_ID_HERE",
    "transformation_code_version": "GIT_SHA_HERE",
    "source_dataset_versions": [
      "SOURCE_VERSION_HERE"
    ]
  },
  "permissions": {
    "execution_principal": "ai_analytics_assistant",
    "row_column_policy_test": "PASS"
  },
  "query": {
    "sql_or_query_id": "QUERY_ID_HERE",
    "sql_hash": "sha256:...",
    "result_hash": "sha256:..."
  },
  "decision_gate": "PASS"
}
```

The values above are illustrative placeholders, not observed production data.

OpenLineage's object model is well aligned with retaining run identity, source-code version and input/output dataset evidence. [8](https://openlineage.io/docs/spec/object-model/) The point is not that OpenLineage is mandatory; it is that **“which exact inputs and code produced this answer?” must have a machine-readable answer**.

### Minimum deterministic controls before interpretation

For the selected monthly-metric workflow, I would consider these the minimal sufficient set:

| Control | Why it is blocking |
|---|---|
| **Metric-definition/version check** | Without it, the question itself is under-specified. |
| **Grain and business-key test** | Prevents many double-count/fan-out failures that still produce valid SQL. |
| **Required schema/type contract** | Prevents silent nulls/remapping after upstream changes. |
| **Period completeness / expected-partition check** | A fresh table can still be incomplete. |
| **Source-to-staging and fact reconciliation** | Gives an independent check that amounts/counts survived the path. |
| **Duplicate/replay test** | Required where business/event uniqueness is expected. |
| **Join-cardinality check for metric-critical joins** | Protects additive measures from fan-out. |
| **Lineage/run/code-version capture** | Makes diagnosis and reproduction tractable. |
| **Real-principal permission test** | Prevents the assistant layer from bypassing intended row/column restrictions. |
| **Deterministic final calculation** | Leaves the LLM to interpret numbers instead of inventing them. |

This is **not** a mandate to run hundreds of tests on unrelated columns. A defect in an unused marketing-enrichment field is irrelevant to a finance explanation unless lineage shows that it influences the selected metric or explanatory dimension.

That gives a useful prioritization rule:

> **Fix defects on the decision's dependency path in proportion to their ability to change the decision, not in proportion to the number of red dots on the data-quality dashboard.**

### What should force clarification, abstention or human review

**Clarification** should be mandatory when there are multiple legitimate interpretations of the question: gross vs net revenue, booked vs recognized revenue, local vs reporting currency, calendar vs fiscal month, natural-person vs account-level active customer, or the intended comparison period.

**Abstention** should be mandatory when a blocking deterministic gate fails, the metric definition/version is unavailable, a period expected to be closed cannot be reconciled, the source snapshot cannot be identified, or the caller does not have the required permission.

**Human review** should be mandatory for a material result after a semantic-definition migration; for a novel or unreconciled defect; when a narrative crosses from arithmetic attribution into causal explanation; or when remediation requires changing production data rather than merely rerunning validated transformations.

A good assistant response to a failed gate is therefore not a clever guess. It is something like:

> “I cannot reliably explain the October movement yet. The warehouse fact is €70 above the source control total and the customer dimension violates its expected key grain. I can show those discrepancies and the affected lineage path, but I will not attribute the business movement until they are resolved.”

That is a feature, not a model failure.

### Evaluate against the analyst doing the same job

The correct baseline is not “AI versus no answer.” It is a competent analyst with the **same available data, definitions and permissions**.

I would run a local evaluation with a deliberately small but stratified case set, for example:

- clean, unambiguous metric questions;
- intentionally ambiguous terminology;
- missing/late events;
- replayed events;
- schema changes;
- bad joins/grain;
- metric-definition version changes;
- stale snapshots;
- inaccessible dimensions/rows;
- retrieval of an obsolete but valid-looking metric.

The number of cases is a local design decision; forty or fifty well-chosen cases is only an example, not an industry benchmark.

Score both analyst and AI-assisted workflow on:

| Dimension | Observable measure |
|---|---|
| **Factual/query correctness** | Does the validated result equal the reference result or an explicitly permitted tolerance? |
| **Source/version fidelity** | Did it use exactly the authorized metric, snapshot and transformation version? |
| **Ambiguity handling** | Did it clarify or abstain when multiple definitions were legitimate? |
| **Permission compliance** | Did it access or reveal anything outside the test principal's entitlement? |
| **Reproducibility** | Can the same snapshot/code/query reproduce the material figures? |
| **Failure recovery** | When a gate fails, does it localize the issue rather than fabricate an answer? |
| **Review effort** | Analyst minutes required per accepted answer |
| **Latency** | Time from question to accepted answer |
| **Cost** | Warehouse + retrieval/model + review cost per accepted answer |

Do **not** make generated-SQL string equality the main criterion. Multiple SQL formulations can be equivalent; the material criterion is the validated result and its provenance.

## Worked synthetic diagnosis

### Why one plausible number can conceal three defects

This fixture is deliberately small. It illustrates why an AI assistant should not interpret the final fact table before deterministic diagnostics have run.

Assume the authoritative monthly metric is:

\[
R_m = \sum_{i \in m} \text{net\_revenue\_eur}_i
\]

at **one row per invoice line**, assigned to the service/recognition month.

The synthetic authoritative values are:

| Period | Authoritative revenue | Change from September |
|---|---:|---:|
| September | €600 | — |
| October | €540 | \((540-600)/600=-10.00\%\) |

Now inject three defects:

1. the €100 October line `o3` never arrives;
2. the €80 line `o1` is replayed once;
3. customer `C2` exists twice in a dimension, faning out the €90 line during a join.

The raw October ingestion therefore totals:

\[
540 - 100 + 80 = €520
\]

but still contains **five physical rows**: four legitimate unique events plus one duplicate. A naive row-count check therefore passes.

The bad fact after the duplicate dimension join totals:

\[
520 + 90 = €610
\]

and now appears to show:

\[
(610-600)/600 = +1.67\%
\]

So an ungated assistant can quite plausibly say:

> “October revenue increased 1.7%.”

The correct source-level movement is **−10.0%**.

The erroneous transformation has reversed the direction of the business story.

### Copyable synthetic SQL fixture

The following is a self-contained synthetic fixture written using broadly portable SQL constructs. Minor type syntax may need adaptation by warehouse. **This exact SQL was not executed against any production warehouse during this research; the expected values below were arithmetically cross-checked from the fixture rows.**

```sql
/*
Synthetic diagnostic fixture.

Authoritative monthly revenue:
  September = EUR 600
  October   = EUR 540

Injected October defects:
  1. Missing source event o3 worth EUR 100
  2. Replayed ingestion event o1 worth EUR 80
  3. Duplicate customer dimension key C2, causing a EUR 90 join fan-out

Expected diagnostics:
  authoritative_oct_revenue_eur = 540
  raw_oct_revenue_eur           = 520
  bad_fact_oct_revenue_eur      = 610

  authoritative_oct_keys = 5
  raw_oct_rows            = 5   -- misleadingly "correct"
  raw_oct_distinct_keys   = 4
  raw_duplicate_keys      = 1
  missing_source_keys     = 1
  duplicate_dim_keys      = 1
*/

WITH source_authoritative AS (

    -- September authoritative invoice lines: total EUR 600
    SELECT '2026-09' AS revenue_month, 's1' AS line_id, 'C1' AS customer_id, 100.00 AS net_revenue_eur
    UNION ALL
    SELECT '2026-09', 's2', 'C2', 110.00
    UNION ALL
    SELECT '2026-09', 's3', 'C3', 120.00
    UNION ALL
    SELECT '2026-09', 's4', 'C4', 130.00
    UNION ALL
    SELECT '2026-09', 's5', 'C5', 140.00

    UNION ALL

    -- October authoritative invoice lines: total EUR 540
    SELECT '2026-10', 'o1', 'C1',  80.00
    UNION ALL
    SELECT '2026-10', 'o2', 'C2',  90.00
    UNION ALL
    SELECT '2026-10', 'o3', 'C3', 100.00
    UNION ALL
    SELECT '2026-10', 'o4', 'C4', 120.00
    UNION ALL
    SELECT '2026-10', 'o5', 'C5', 150.00
),

raw_ingested AS (

    -- o3 is missing.
    SELECT 'ingest_001' AS ingest_id,
           '2026-10' AS revenue_month,
           'o1' AS line_id,
           'C1' AS customer_id,
           80.00 AS net_revenue_eur,
           '2026-11-01T01:00:00Z' AS ingested_at

    UNION ALL
    SELECT 'ingest_002', '2026-10', 'o2', 'C2',  90.00, '2026-11-01T01:00:01Z'
    UNION ALL
    SELECT 'ingest_004', '2026-10', 'o4', 'C4', 120.00, '2026-11-01T01:00:03Z'
    UNION ALL
    SELECT 'ingest_005', '2026-10', 'o5', 'C5', 150.00, '2026-11-01T01:00:04Z'

    -- Replay of o1.
    UNION ALL
    SELECT 'ingest_006', '2026-10', 'o1', 'C1',  80.00, '2026-11-01T01:05:00Z'
),

customer_dim AS (

    SELECT 'C1' AS customer_id, 'segment_a' AS segment
    UNION ALL
    SELECT 'C2', 'segment_b'

    -- Erroneous second current row for C2.
    UNION ALL
    SELECT 'C2', 'segment_b_duplicate'

    UNION ALL
    SELECT 'C3', 'segment_c'
    UNION ALL
    SELECT 'C4', 'segment_d'
    UNION ALL
    SELECT 'C5', 'segment_e'
),

bad_fact AS (
    /*
    This intentionally demonstrates the bad transformation:
    neither ingestion replay nor dimension-key duplication is removed.
    */
    SELECT
        r.revenue_month,
        r.line_id,
        r.customer_id,
        r.net_revenue_eur,
        d.segment
    FROM raw_ingested r
    LEFT JOIN customer_dim d
      ON r.customer_id = d.customer_id
),

authoritative_oct AS (
    SELECT
        COUNT(*) AS authoritative_oct_keys,
        SUM(net_revenue_eur) AS authoritative_oct_revenue_eur
    FROM source_authoritative
    WHERE revenue_month = '2026-10'
),

raw_oct AS (
    SELECT
        COUNT(*) AS raw_oct_rows,
        COUNT(DISTINCT line_id) AS raw_oct_distinct_keys,
        SUM(net_revenue_eur) AS raw_oct_revenue_eur
    FROM raw_ingested
    WHERE revenue_month = '2026-10'
),

bad_fact_oct AS (
    SELECT
        COUNT(*) AS bad_fact_oct_rows,
        SUM(net_revenue_eur) AS bad_fact_oct_revenue_eur
    FROM bad_fact
    WHERE revenue_month = '2026-10'
),

raw_duplicate_key_count AS (
    SELECT COUNT(*) AS raw_duplicate_keys
    FROM (
        SELECT line_id
        FROM raw_ingested
        GROUP BY line_id
        HAVING COUNT(*) > 1
    ) x
),

missing_source_key_count AS (
    SELECT COUNT(*) AS missing_source_keys
    FROM source_authoritative s
    WHERE s.revenue_month = '2026-10'
      AND NOT EXISTS (
          SELECT 1
          FROM raw_ingested r
          WHERE r.line_id = s.line_id
      )
),

duplicate_dim_key_count AS (
    SELECT COUNT(*) AS duplicate_dim_keys
    FROM (
        SELECT customer_id
        FROM customer_dim
        GROUP BY customer_id
        HAVING COUNT(*) > 1
    ) x
)

SELECT
    a.authoritative_oct_keys,
    a.authoritative_oct_revenue_eur,

    r.raw_oct_rows,
    r.raw_oct_distinct_keys,
    r.raw_oct_revenue_eur,

    b.bad_fact_oct_rows,
    b.bad_fact_oct_revenue_eur,

    d.raw_duplicate_keys,
    m.missing_source_keys,
    c.duplicate_dim_keys,

    /* Reconciliation deltas */
    r.raw_oct_revenue_eur
        - a.authoritative_oct_revenue_eur
        AS raw_vs_authoritative_delta_eur,

    b.bad_fact_oct_revenue_eur
        - a.authoritative_oct_revenue_eur
        AS fact_vs_authoritative_delta_eur

FROM authoritative_oct a
CROSS JOIN raw_oct r
CROSS JOIN bad_fact_oct b
CROSS JOIN raw_duplicate_key_count d
CROSS JOIN missing_source_key_count m
CROSS JOIN duplicate_dim_key_count c;
```

Expected outputs from the fixture are:

| Check | Expected result | Diagnostic meaning |
|---|---:|---|
| Authoritative Oct keys | 5 | Expected business events |
| Raw physical rows | 5 | **Misleading pass if used alone** |
| Raw distinct business keys | 4 | One unique event missing |
| Duplicate raw keys | 1 | Replay present |
| Missing source keys | 1 | `o3` absent |
| Duplicate dimension keys | 1 | Join cardinality broken |
| Authoritative Oct revenue | €540 | Control |
| Raw Oct revenue | €520 | −€20 / −3.70% versus source |
| Bad fact Oct revenue | €610 | +€70 / +12.96% versus source |
| Bad fact versus September | +1.67% | Wrong direction |
| True October versus September | −10.00% | Correct movement |

This example exposes an important failure mode: **different data defects can partly cancel each other**, so “the total is within 1%” is not sufficient evidence if offsetting errors are plausible.

The same defects affect active-customer counts differently. In this fixture, `COUNT(DISTINCT customer_id)` would not be inflated by the dimension fan-out or replay, but the missing customer/event could still undercount activity. A `COUNT(*)`-based activity measure could be affected by all three. This is why controls must derive from the actual metric grain and formula rather than from a generic “data quality score.”

### What evidence justifies repair versus abstention

For the synthetic case, the correct response is to **abstain first**.

A justified repair sequence would be:

1. establish whether `o3` exists in the authoritative source or re-extract the affected source partition;
2. identify replay semantics for `o1` and apply the defined event/business-key deduplication rule;
3. restore the expected uniqueness/effective-dating rule for `customer_dim`;
4. rebuild the affected lineage path from a known transformation version;
5. rerun source → staging → fact reconciliation;
6. preserve the run/source/code identifiers;
7. only then allow the assistant to decompose the actual business movement.

An LLM can help investigate steps 1–4, but it should not certify them.

If no independent source-side evidence exists—no event manifest, accounting balance, source aggregate or controlled re-extraction—then an assistant should not announce that the missing event problem is solved merely because warehouse totals now “look normal.”

## Decision economics, ownership and release criteria

### Compare the viable strategies

| Strategy | Advantage | Main weakness | My view |
|---|---|---|---|
| **Make all data perfect before AI** | Lowest obvious data-quality anxiety | Open-ended scope; fixes unrelated data; delays learning about the actual AI workflow | Usually the wrong target |
| **Keep analyst-only workflow** | Known operating model; easy accountability | Manual investigation cost and slower scaling remain | Essential baseline, not necessarily end state |
| **Gated AI + analyst review** | Concentrates controls on material dependency paths while learning where AI actually saves work | Requires evaluation harness, ownership and ongoing control maintenance | **Recommended** |
| **Unconstrained AI over warehouse** | Fastest demo and broadest apparent capability | Combines data defects, model errors and permission risk in one opaque surface | Poor choice for material metrics |

The recommendation changes if the team cannot state the authoritative metric in the first place. Then the first project is semantic governance, not an AI assistant.

### Transparent synthetic economics

The following is deliberately a **hypothetical decision model**, not measured ROI.

Assumptions:

| Input | Synthetic base assumption |
|---|---:|
| Investigations per month \(N\) | 80 |
| Analyst-only time per investigation | 45 min |
| AI-assisted time including human review | 20 min |
| Loaded analyst cost | €90/hour |
| Control maintenance | 6 engineering hours/month |
| Loaded engineering cost | €100/hour |
| AI + retrieval + query infrastructure | €400/month |
| False alarms | 5/month |
| Review cost per false alarm | 0.5 analyst hour |
| One-time control/evaluation build | 60 engineering hours = €6,000 |

Time-value formula:

\[
\text{gross monthly labor value}
=
N
\times
\frac{t_{baseline}-t_{AI}}{60}
\times
C_{analyst}
\]

Base case:

\[
80
\times
\frac{45-20}{60}
\times €90
=
€3,000 / month
\]

Recurring control cost:

\[
6h \times €100 = €600/month
\]

False-alarm review cost:

\[
5 \times 0.5h \times €90
=
€225/month
\]

Total recurring incremental costs:

\[
€600 + €400 + €225 = €1,225/month
\]

Synthetic operating benefit:

\[
€3,000 - €1,225
=
€1,775/month
\]

Synthetic simple payback:

\[
€6,000 / €1,775
\approx 3.38\ months
\]

Again: **none of these inputs is claimed as a market rate, customer result or expected outcome.**

Sensitivity to investigation volume, holding the other assumptions fixed:

| Investigations/month | Hours saved/month | Gross labor value | Net after €1,225 recurring cost | Simple payback on €6,000 |
|---:|---:|---:|---:|---:|
| 20 | 8.33 h | €750 | **−€475** | None |
| 50 | 20.83 h | €1,875 | €650 | 9.23 months |
| 80 | 33.33 h | €3,000 | €1,775 | 3.38 months |
| 150 | 62.50 h | €5,625 | €4,400 | 1.36 months |

Review effort is the critical economic variable. If AI-assisted work takes **35 rather than 20 minutes** in the 80-investigation case:

\[
80\times\frac{45-35}{60}\times€90
=
€1,200
\]

which is less than the assumed €1,225 recurring cost.

The synthetic operating benefit becomes:

\[
€1,200 - €1,225 = -€25/month
\]

That is why the pilot should measure **accepted-answer review time**, not token usage or the number of questions asked.

### False alarms are a real control cost

A very sensitive data-quality gate can be economically worse than a narrow one if it continuously stops valid analysis.

For each control, track at least:

\[
\text{monthly false-alarm cost}
=
F
\times
t_{review}
\times
C_{reviewer}
\]

and distinguish:

- **false positives:** control blocks good data;
- **true positives with negligible materiality:** technically real defect, irrelevant decision impact;
- **true material defects:** the cases that justify the control.

This strongly favors testing the lineage path and metric invariants rather than trying to detect every oddity in every warehouse table.

### Ownership

The assistant should not become the accidental owner of unresolved business semantics.

| Responsibility | Suggested owner |
|---|---|
| Revenue/active-customer definition and materiality | Finance/business metric owner |
| Source controls and ingestion health | Data platform / source owner |
| Grain, transformations, dbt tests, reconciliation | Analytics engineering |
| Lineage completeness | Data platform / analytics engineering |
| Row/column permissions | Security/data governance |
| Retrieval, SQL-generation and model evaluation | AI/data product team |
| Gold/reference cases and accepted-answer review | Analysts + metric owner |
| Production-data repair | Existing operational owner, **not autonomous AI** |

### Proposed go/no-go criteria

These are **design choices for this workflow**, not universal industry thresholds.

**Go to a human-reviewed pilot** only when:

- an owner has approved the metric definition, grain and version;
- every material input on the selected lineage path is identifiable;
- closed periods have an explicit completeness/reconciliation control, not just a freshness timestamp;
- no unexplained duplicate business keys exist where uniqueness is required;
- metric-critical dimension relationships meet expected cardinality;
- the exact query/result can be reproduced from a named snapshot and code version;
- the assistant is tested using its actual warehouse identity;
- the evaluation set contains clean and deliberately defective cases;
- the AI-assisted workflow is compared with the analyst baseline on the same cases.

A possible release standard for a **small critical pilot** could be zero observed permission violations, zero source/version substitutions, and zero material numerical errors that passed the deterministic gate in the release set. Those are deliberately conservative design thresholds, not statistical guarantees about future production behavior.

A business materiality threshold should be supplied by the metric owner. For illustration only, a finance team might block an explanation when an unexplained reconciliation delta exceeds:

\[
\max(€25,000,\;0.5\%\times\text{monthly revenue})
\]

There is no evidence in this dossier that €25,000 or 0.5% is a generally correct threshold; they are examples of how to make materiality explicit.

### Proposed stop criteria

Pause the pilot immediately for investigation if:

- unauthorized data is accessed or disclosed;
- a materially wrong numerical conclusion passes all purported gates;
- an answer cites or describes a different metric/source version from the one actually executed;
- a system that is supposed to abstain when a blocking test is red instead gives a confident business explanation;
- a production correction is attempted automatically without the established operational approval path.

A softer review trigger is appropriate when analyst review effort ceases to improve materially over the non-AI baseline. The economics above show why an assistant requiring near-full manual reconstruction can be technically impressive but economically pointless.

## Contradictions, failure conditions and regional relevance

### “AI can detect bad data” and “AI repeats bad data” are both true

There is no real contradiction once the source of evidence is made explicit.

An AI system can detect a duplicate key **if the key definition and data are observable**. It can notice a schema diff **if previous/current schema metadata are supplied**. It can point out that a reconciliation test is red **if that test exists**.

It cannot infer with certainty that 10,000 upstream events were never emitted when nothing in its evidence establishes the expected 10,000.

This distinction is exactly why declarative validation systems such as Deequ are useful: assumptions become observable constraints rather than model intuition. [6](https://www.amazon.science/publications/deequ-data-quality-validation-for-machine-learning-pipelines)

### Fresh data can be wrong; complete data can be old

dbt's freshness mechanism is explicitly centered on user-defined freshness/SLA information. [5](https://docs.getdbt.com/docs/deploy/source-freshness) It should therefore be treated as one signal, not an overall certificate.

A dataset can be:

- recent but missing one region;
- old but fully reconciled for a historical closed period;
- fresh and complete but semantically wrong;
- fresh, complete and semantically valid but duplicated after a downstream join.

The gate should retain those dimensions separately.

### Exactly-once ingestion does not eliminate downstream duplication

Some infrastructure supports stronger delivery modes than the default at-least-once semantics documented for representative systems such as Pub/Sub. [11](https://docs.cloud.google.com/pubsub/docs/subscription-overview) Even with perfect ingestion deduplication, a transformation can still create double counting through a many-to-many join, slowly changing dimension error or aggregation at the wrong grain.

Therefore “we use exactly-once” is not evidence that the revenue fact has a valid grain.

### A semantic layer reduces inconsistency but can institutionalize a mistake

A central metric definition is valuable because consumers can converge on the same entities, dimensions and aggregations. dbt's current Semantic Layer provides explicit structures for exactly that. [7](https://docs.getdbt.com/guides/sl-qs)

But centralization raises the blast radius of a mistaken definition. This argues for metric versioning, effective dates, owners and regression fixtures—not against semantic layers.

### Lineage can be incomplete

OpenLineage can encode detailed run, source-code and field-level relationships. [8](https://openlineage.io/docs/spec/object-model/) Actual lineage coverage still depends on instrumentation. Dynamic SQL, user-defined functions, externally generated jobs or transformations occurring outside instrumented systems can create gaps.

A missing lineage edge should therefore be “unknown dependency,” not “no dependency.”

### Benchmarks are useful, but not as production acceptance tests

The newest DAB version changed as recently as October 7, 2026 and now reports 104 queries, 17 datasets and a 57% best pass@1 result. [2](https://arxiv.org/abs/2603.20576) Spider 2.0 is a different workload and reports 21.3% for its cited code-agent configuration. [1](https://arxiv.org/abs/2411.07763)

These do not conflict: benchmark task sets, versions, models, harnesses and scoring differ.

More importantly, benchmark-label quality itself is a live issue. Wretblad et al. found substantial question/gold-query noise in BIRD, while the 2026 benchmark audit reports that correcting benchmark annotations can materially change model rankings. [3](https://aclanthology.org/2024.acl-short.34/)

So leaderboard position is evidence of capability under a benchmark definition, not evidence that “our October revenue explanation will be correct.”

### Provider documentation is not empirical performance evidence

AWS, Google Cloud, dbt and OpenLineage documentation are strong sources for how their mechanisms are specified to behave. They do not measure how often your pipeline fails or demonstrate that adopting the documented feature causes better business outcomes.

Similarly, the Amazon Deequ publication supports declarative data assumptions and automatic checking; it does not measure ROI for an LLM analytics assistant. [6](https://www.amazon.science/publications/deequ-data-quality-validation-for-machine-learning-pipelines)

### Czech/EU versus US relevance

The **technical readiness standard is essentially the same** in both contexts: authoritative metric, grain, reconciliation, lineage, permission enforcement and reproducibility.

The most material difference for this scoped use case is privacy governance when customer-level personal data enters the workflow. In the EU, including Czechia, GDPR Article 5 expressly requires data minimization and accuracy for personal data. [14](https://eur-lex.europa.eu/eli/reg/2016/679/oj) That strengthens the case for giving an assistant aggregate or pseudonymized evidence instead of unrestricted customer data where identifiable detail is unnecessary.

The EU AI Act contains data/data-governance requirements for **high-risk AI systems** in Article 10. [15](https://eur-lex.europa.eu/eli/reg/2024/1689/oj) It should not be casually treated as a blanket requirement for every internal revenue analytics assistant; classification and applicability depend on the system and use. This dossier is not legal advice.

In the US, NIST's Generative AI Profile is a voluntary cross-sector risk-management reference rather than a deployment-specific data-quality mandate. NIST describes it as a companion to the AI RMF intended to help organizations incorporate trustworthiness considerations into AI design, development, use and evaluation. [16](https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence)

So I would **not** build different data-quality architectures for Prague and New York merely because of geography. I would apply the same technical evidence gate and layer relevant privacy, contractual and regulatory controls on top.

### Where the recommendation can fail

The proposed approach is weaker when:

- there is no credible independent source/control against which completeness can be checked;
- the primary problem is inherently unstructured external context rather than warehouse data;
- the metric itself is exploratory and intentionally undefined;
- the decision is low consequence enough that a strict evidence gate costs more than occasional error;
- source systems have inconsistent business semantics that cannot be resolved by the data team;
- an assistant is being used purely for brainstorming rather than as a factual analytical interface;
- the analysis needs causal inference rather than arithmetic decomposition;
- permission systems cannot represent the business policy being requested.

The reverse situation also exists: in a mature, tightly contracted, low-latency data product with strong source controls and simple metrics, some proposed gates will be redundant. “Minimum sufficient” should stay minimum.

### The largest unresolved evidence gaps

I found **no high-quality public study directly comparing a gated AI analytical assistant with a skilled analyst on the same monthly-revenue-change workflow while injecting source missingness, replay, join fan-out, semantic changes, lineage gaps and permission restrictions**. That would be the most decision-relevant experiment.

There is also no defensible universal materiality threshold, acceptable error rate, control count or ROI figure for this workflow. Those need to come from the organization's own decision cost and pilot observations.

Finally, text-to-SQL and data-agent research is moving very quickly. The DAB paper itself was revised on October 7, 2026, only three days before this snapshot. [2](https://arxiv.org/abs/2603.20576) Any article quoting current frontier scores should therefore preserve benchmark version, evaluation date and harness rather than present a timeless “AI accuracy” number.

No load-bearing source was inaccessible during this research. Where a source also offered a PDF, I relied on its public HTML publication page or HTML paper rendering for the claims used here rather than inferring PDF-only content.

## Claim-to-source dossier, publication angles and freshness triggers

### Claim-to-source table

The **inspected date for every source below is 2026-10-10**.

| Claim supported | Original source and source date | Direct URL / DOI | Exact evidentiary scope | Evidence type |
|---|---|---|---|---|
| dbt source freshness is an explicit SLA-oriented check; `dbt build` does not automatically include it, and job configuration affects whether failure blocks later steps. | dbt Labs, *Source freshness*, updated 2026-10-06. [5](https://docs.getdbt.com/docs/deploy/source-freshness) | https://docs.getdbt.com/docs/deploy/source-freshness | Current documented dbt behavior only; does not establish completeness or business correctness. | Versioned provider documentation |
| Semantic models represent entities, dimensions and metrics, and dbt's example explicitly declares fact grain as one row per order. | dbt Labs, *Quickstart for the dbt Semantic Layer*, current 2026 documentation. [7](https://docs.getdbt.com/guides/sl-qs) | https://docs.getdbt.com/guides/sl-qs | Describes dbt v1.12+ semantic-model structure; not empirical evidence that semantic definitions are correct. | Provider documentation |
| Runtime/design lineage can represent jobs, datasets, runs, SQL/source-code metadata and dataset versions. | OpenLineage Object Model, living specification; page publication date not stated. [8](https://openlineage.io/docs/spec/object-model/) | https://openlineage.io/docs/spec/object-model/ | Specification of available lineage metadata; does not guarantee instrumentation completeness. | Open standard/specification |
| Field lineage can distinguish aggregation from indirect dependencies such as joins, filters, groups, windows and conditionals. | OpenLineage Column Level Lineage Dataset Facet, living specification; page publication date not stated. [9](https://openlineage.io/docs/spec/facets/dataset-facets/column_lineage_facet/) | https://openlineage.io/docs/spec/facets/dataset-facets/column_lineage_facet/ | Specification of lineage representations, useful for narrowing transformation diagnosis. | Open standard/specification |
| Explicit data assumptions can be declaratively validated at scale. | Schelter et al., *DEEQU – Data quality validation for machine learning pipelines*, NeurIPS 2018. [6](https://www.amazon.science/publications/deequ-data-quality-validation-for-machine-learning-pipelines) | https://www.amazon.science/publications/deequ-data-quality-validation-for-machine-learning-pipelines | ML-pipeline data validation and reported Amazon production requirements; not an LLM-assistant experiment. | Primary research/company publication |
| Standard SQS queues may deliver the same message more than once; consumers should be idempotent. | AWS, *Amazon SQS at-least-once delivery*, living documentation; page publication date not stated. [10](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html) | https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html | SQS Standard delivery semantics only. | Provider documentation |
| Pub/Sub defaults to at-least-once/no-ordering semantics and may redeliver messages; exactly-once is separately supported. | Google Cloud, *Subscription overview*, updated 2026-10-09. [11](https://docs.cloud.google.com/pubsub/docs/subscription-overview) | https://docs.cloud.google.com/pubsub/docs/subscription-overview | Google Pub/Sub subscription behavior; representative ingestion evidence, not a claim about every pipeline. | Provider documentation |
| BigQuery row-level security filters rows based on user conditions. | Google Cloud, *Introduction to BigQuery row-level security*, living documentation; page publication date not surfaced. [12](https://docs.cloud.google.com/bigquery/docs/row-level-security-intro?utm_source=chatgpt.com) | https://docs.cloud.google.com/bigquery/docs/row-level-security-intro | BigQuery-specific authorization mechanism. | Provider documentation |
| BigQuery checks column policy authorization at query time and combines it with other permissions. | Google Cloud, *Introduction to column-level access control*, living documentation. [17](https://docs.cloud.google.com/bigquery/docs/column-level-security-intro?utm_source=chatgpt.com) | https://docs.cloud.google.com/bigquery/docs/column-level-security-intro | BigQuery-specific column security. | Provider documentation |
| Query-result copies do not automatically inherit BigQuery column policy tags in most cases. | Google Cloud, *Restrict access with column-level access control*, living documentation. [13](https://docs.cloud.google.com/bigquery/docs/column-level-security?utm_source=chatgpt.com) | https://docs.cloud.google.com/bigquery/docs/column-level-security | Provider-specific example of permission propagation risk through transformations/copies. | Provider documentation |
| Spider 2.0 contains 632 enterprise-like workflow tasks, databases often over 1,000 columns, and its cited agent configuration solved 21.3%. | Lei et al., v1 2024-11-12; v2 revised 2025-03-17; ICLR 2025 Oral. [1](https://arxiv.org/abs/2411.07763) | https://arxiv.org/abs/2411.07763 ; DOI: https://doi.org/10.48550/arXiv.2411.07763 | Enterprise-oriented text-to-SQL capability benchmark; not a production error-rate estimate. | Primary benchmark research |
| DAB v3 has 104 queries, 17 datasets and four DBMS; best evaluated agent reported 57% pass@1. | Ma et al., v1 2026-03-21; v3 revised 2026-10-07. [2](https://arxiv.org/abs/2603.20576) | https://arxiv.org/abs/2603.20576 ; DOI: https://doi.org/10.48550/arXiv.2603.20576 | End-to-end data-agent benchmark grounded in studied enterprise workloads; still a benchmark, not the target production workflow. | Primary benchmark/preprint |
| Noise/errors in questions and gold SQL can undermine text-to-SQL benchmark reliability. | Wretblad et al., ACL 2024, August 2024. [3](https://aclanthology.org/2024.acl-short.34/) | https://aclanthology.org/2024.acl-short.34/ ; DOI: https://doi.org/10.18653/v1/2024.acl-short.34 | Study of BIRD-Bench noise and corrected-query evaluations. | Peer-reviewed empirical research |
| A 2026 audit found annotation errors in 52.8% of audited BIRD Mini-Dev examples and 62.8% of 121 Spider 2.0-Snow problems with released gold queries, and reported ranking shifts after corrections. | *Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards*, 2026, v3 inspected. [18](https://arxiv.org/html/2601.08778v3) | https://arxiv.org/html/2601.08778v3 | Manual/aided benchmark annotation audit. These rates describe the audited benchmark examples, **not production AI failure rates**. | Primary preprint/audit |
| NIST's GenAI Profile is a voluntary cross-sector companion to AI RMF for incorporating trustworthiness into AI lifecycle activities. | NIST AI 600-1, published 2024-07-26; NIST page updated 2026-04-08. [16](https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence) | https://doi.org/10.6028/NIST.AI.600-1 | Governance/risk-management framework, not empirical effectiveness evidence. | Official US government publication |
| GDPR requires data minimization and accuracy for personal data. | Regulation (EU) 2016/679, adopted 2016-04-27; OJ publication 2016-05-04, Article 5. [14](https://eur-lex.europa.eu/eli/reg/2016/679/oj) | https://eur-lex.europa.eu/eli/reg/2016/679/oj | Binding EU legal text applicable within its scope; not a generic rule for non-personal business metrics. | Primary legislation |
| EU AI Act Article 10 contains data/data-governance requirements tied to high-risk AI systems. | Regulation (EU) 2024/1689, 2024, Article 10. [15](https://eur-lex.europa.eu/eli/reg/2024/1689/oj) | https://eur-lex.europa.eu/eli/reg/2024/1689/oj | Conditional regulatory relevance; should not be generalized to every analytics assistant. | Primary legislation |

### Evidence-strength interpretation

The strongest basis for the core recommendation is not one study saying “do these ten checks.” It is the convergence of several independent facts:

- enterprise data-agent tasks remain difficult even for strong models; [1](https://arxiv.org/abs/2411.07763)
- benchmark scores themselves can overstate or misrank capability when evaluation data is wrong; [3](https://aclanthology.org/2024.acl-short.34/)
- deterministic data assumptions can be directly encoded and checked; [6](https://www.amazon.science/publications/deequ-data-quality-validation-for-machine-learning-pipelines)
- freshness, grain, permissions and lineage are separately observable system properties rather than things the model must infer; [4](https://docs.getdbt.com/docs/deploy/source-freshness)
- common ingestion mechanisms can legitimately expose the pipeline to replay, making uniqueness/idempotency a system responsibility rather than an AI reasoning problem. [10](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html)

That supports the architectural conclusion: **use probabilistic AI on top of validated evidence rather than as the mechanism that validates the evidence.**

### Candidate distinct article angles

**“You do not need perfect data for AI. You need proof that the data is fit for one decision.”**  
The decision-maker angle. It rejects both “clean everything first” and “LLMs will figure it out,” replacing them with decision-specific fitness and materiality.

**“The most dangerous data-quality problem is a plausible number.”**  
Lead with the synthetic €600 → €610 example: missing data, replay and join fan-out transform a real −10% movement into an apparent +1.7%. Strong practical narrative.

**“Your AI analyst needs a pre-flight checklist, not a bigger prompt.”**  
Focus on deterministic gating: freshness, completeness, uniqueness, grain, reconciliation, lineage and permissions before model interpretation.

**“Lineage is how you turn ‘the AI is wrong’ into a bounded engineering problem.”**  
Explain how field/run/version lineage narrows the search path while emphasizing that lineage is not proof of source truth.

**“Why a text-to-SQL leaderboard cannot tell you whether your revenue answer is trustworthy.”**  
Contrast Spider 2.0/DAB capability evidence with annotation-noise findings and target-workflow evaluation.

**“AI should explain the result, not certify the ledger.”**  
Sharper executive framing for finance/revenue analytics: deterministic systems establish the number; AI accelerates investigation and communication.

### Reusable assets from this dossier

The most useful publication or implementation assets are already portable within this dossier:

- the **defect → failure → deterministic control → AI limitation matrix**;
- the **synthetic SQL fixture** showing missing event + replay + join fan-out;
- the **evidence-envelope JSON** for reproducibility and source/version fidelity;
- the **analyst-vs-AI evaluation dimensions**;
- the **transparent ROI formula and sensitivity model**;
- the **go/constrain/postpone decision model**.

A particularly useful visual adaptation would be a pipeline with two parallel failure surfaces:

> **Source → Ingestion → Warehouse → Transformation → Semantic Metric → Deterministic Result**  
> *data/system reliability surface*

followed by:

> **Retrieval → SQL generation → Result selection → Explanation**  
> *AI capability/reliability surface*

with **lineage and permissions running across both**.

### Freshness triggers for future reuse

This dossier should be re-checked when any of the following occurs:

**Data-agent evaluation changes.** DAB was revised on October 7, 2026, only three days before this snapshot, so quoted frontier results are especially freshness-sensitive. [2](https://arxiv.org/abs/2603.20576)

**dbt behavior or Semantic Layer version changes.** The current freshness page was updated October 6, 2026, and dbt's semantic model behavior is version-specific. [5](https://docs.getdbt.com/docs/deploy/source-freshness)

**Lineage-spec changes or instrumentation changes.** A new facet specification is less important than whether your actual orchestration/query stack emits the relevant edges and run/version metadata. [8](https://openlineage.io/docs/spec/object-model/)

**Cloud access-control behavior changes.** Particularly re-check propagation semantics for derived tables, views, masking and row/column policies. [12](https://docs.cloud.google.com/bigquery/docs/row-level-security-intro?utm_source=chatgpt.com)

**Ingestion delivery semantics change.** Moving between at-least-once and exactly-once configurations can change which duplicate controls are necessary, although it does not remove downstream join/grain risk. [10](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html)

**The business metric changes.** A new revenue-recognition policy, identity model, active-customer window, currency treatment or refund rule should invalidate the previous evidence envelope until regression tests and historical comparability are reviewed.

**The model, retrieval stack or agent harness changes.** The AI portion should be re-evaluated even when the data layer is unchanged; model capability is a separate failure surface.

**EU/US governance changes.** For Czech/EU deployments, refresh GDPR/AI Act applicability analysis when processing scope or use-case classification changes. For US-oriented governance, NIST itself notes that its GenAI Profile sits within an evolving AI risk-management framework. [14](https://eur-lex.europa.eu/eli/reg/2016/679/oj)

The durable decision principle is less likely to expire:

> **Before AI explains a material metric, require independent evidence that the metric means what you think it means, the expected data arrived, the transformations preserved the intended grain, the query used the authorized version, and the caller was allowed to see the result. Let AI investigate and interpret that evidence; do not make it the evidence.**

## Portable citation index

1. [[2411.07763] Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows](https://arxiv.org/abs/2411.07763)
2. [[2603.20576] Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents](https://arxiv.org/abs/2603.20576)
3. [Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark - ACL Anthology](https://aclanthology.org/2024.acl-short.34/)
4. [Source freshness | dbt Developer Hub](https://docs.getdbt.com/docs/deploy/source-freshness)
5. [Source freshness | dbt Developer Hub](https://docs.getdbt.com/docs/deploy/source-freshness)
6. [DEEQU - Data quality validation for machine learning pipelines - Amazon Science](https://www.amazon.science/publications/deequ-data-quality-validation-for-machine-learning-pipelines)
7. [Quickstart for the dbt Semantic Layer | dbt Developer Hub](https://docs.getdbt.com/guides/sl-qs)
8. [Object Model | OpenLineage](https://openlineage.io/docs/spec/object-model/)
9. [Column Level Lineage Dataset Facet | OpenLineage](https://openlineage.io/docs/spec/facets/dataset-facets/column_lineage_facet/)
10. [Amazon SQS at-least-once delivery - Amazon Simple Queue Service](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html)
11. [Subscription overview  |  Pub/Sub  |  Google Cloud Documentation](https://docs.cloud.google.com/pubsub/docs/subscription-overview)
12. [Introduction to BigQuery row-level security](https://docs.cloud.google.com/bigquery/docs/row-level-security-intro?utm_source=chatgpt.com)
13. [Restrict access with column-level access control | BigQuery](https://docs.cloud.google.com/bigquery/docs/column-level-security?utm_source=chatgpt.com)
14. [Regulation - 2016/679 - EN - gdpr - EUR-Lex](https://eur-lex.europa.eu/eli/reg/2016/679/oj)
15. [Regulation - EU - 2024/1689 - EN - EUR-Lex](https://eur-lex.europa.eu/eli/reg/2024/1689/oj)
16. [Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile | NIST](https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence)
17. [Introduction to column-level access control | BigQuery](https://docs.cloud.google.com/bigquery/docs/column-level-security-intro?utm_source=chatgpt.com)
18. [Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards](https://arxiv.org/html/2601.08778v3)


## Import verification and asset availability

Captured from genuine ChatGPT Deep Research with snapshot 2026-10-10; completed in 8 minutes. Retrieved 2026-10-10. Original prompt and canonical conversation are retained on this run. Complete synthetic code/examples are inline in this report; no external sandbox asset download is required or claimed. Embedded examples and report statements about execution have not been independently run or verified during import. Delivery is not independent validation of source claims, dates or applicability. Evidence remains Captured.