# AI Analytics: When Can You Trust an AI to Work With Company Data?

**Evidence dossier — research snapshot: 3 October 2026**

The scope, exclusions, analytical chain, evidence priorities, and requested deliverables in this dossier follow the supplied research brief. fileciteturn0file0

**Central finding.** Trustworthiness in AI analytics is **not a property of an LLM, a text-to-SQL score, or a semantic layer in isolation**. It is a property of a particular analytical task performed against particular data, definitions, permissions, and time state, with an evaluation procedure capable of detecting the failures that matter. NIST's definition of validation is useful here: objective evidence that requirements are fulfilled for a specified intended use. citeturn8search13turn8search17

The strongest current case for trust is therefore narrow: **AI as a semantic interface and orchestrator over governed, deterministic analytical machinery**. An AI can be comparatively reliable when it selects an already-defined metric, applies bounded filters or groupings, executes a governed query or deterministic calculation, and exposes enough provenance to reproduce the answer. Reliability becomes much weaker when the AI must invent business meaning, judge questionable data, infer causality, forecast unfamiliar regimes, choose interventions, or act without an independently enforceable control layer. Evidence from enterprise text-to-SQL, interactive SQL, end-to-end data-analysis, causal-reasoning, and forecasting benchmarks consistently shows that these are materially different capability classes. citeturn0academia12turn0academia13turn12search1turn4search3turn5search2

A second finding is equally important: **SQL generation is a middle step, not the unit of trust.** SQL can parse, execute, and even reproduce a reference result while still answering the wrong business question because it used the wrong definition, grain, source, time boundary, population, or implicit assumption. Research on semantic evaluation already showed why exact SQL matching is an inadequate correctness measure; enterprise benchmarks make the larger problem clearer because schemas, metadata, business rules, and interaction matter. citeturn9search1turn0academia12

A practical evidence hierarchy emerges from the literature:

| Evidence level | What has actually been demonstrated | What has **not** been demonstrated |
|---|---|---|
| **Syntactic** | SQL/code parses | Correct data or business interpretation |
| **Executable** | Query runs | Correct result |
| **Result-correct** | Output matches an oracle on tested data | Correct business definition outside that test |
| **Metric-correct** | Correct governed metric, grain, filters and dimensions | Correct interpretation or causality |
| **Analytically valid** | Method matches question and assumptions | Decision quality |
| **Decision-valid** | Evidence supports stated conclusion under known uncertainty | Safe autonomous action |
| **Operationally trustworthy** | Above properties hold within a measured production envelope, including authorization, freshness and auditability | Reliability outside that envelope |

That distinction is the backbone of this dossier.

## Field taxonomy and analytical task taxonomy

**AI analytics taxonomy.** Several commonly grouped terms describe genuinely different things; others are primarily product-positioning labels. The distinction matters because evidence about one should not be generalized to another.

| Term | What it materially denotes | Distinct capability or mostly label? | Principal trust question |
|---|---|---|---|
| **Text-to-SQL / natural-language-to-SQL** | Translation from a linguistic request plus database context into SQL | Substantially the same core technical task | Did the program express the intended query? |
| **Conversational analytics** | Multi-turn interface over data, often including query generation, answers, charts and explanations | Broader capability | Did conversational context improve or corrupt interpretation? |
| **AI data analyst** | System that may plan, inspect files/tables, write SQL/Python, calculate, visualize and narrate | Broad product/category label | Can the entire analytical workflow be validated, not merely one query? |
| **Analytical copilot** | Human-in-the-loop assistant for analytical work | Mainly interaction/delegation label | Which steps are reviewed by a competent human? |
| **Data agent** | Tool-using system that selects and invokes databases, code, catalogs or other analytical tools over multiple steps | Material architectural distinction | Does planning/tool selection introduce new failure modes? |
| **Semantic layer** | Representation of business entities, dimensions, measures, relationships and related semantics above physical schemas | Material infrastructure capability | Are meanings, grains and joins correctly modeled? |
| **Metric layer / governed metrics** | Narrower mechanism defining reusable calculations and permissible dimensional slicing; “governed” adds ownership/control/versioning | Material control mechanism | Is the requested concept mapped to the intended metric/version? |
| **Verified/golden query** | Reviewed question–query pair used for retrieval/grounding or evaluation | Control/evaluation artifact, not intelligence | Is it actually correct, current and sufficiently similar to the new question? |
| **AI-generated dashboard** | Automated selection/composition of queries, charts and layouts | Output-generation category | Are queries, visual encodings and aggregation levels correct? |
| **Narrative analytics** | Generated natural-language description or explanation of analytical results | Distinct output capability | Does prose stay within what the data establish? |
| **Anomaly explanation** | Narrative or feature search around an unusual observation | Often ambiguously named capability | Is it reporting correlates, hypotheses, or causal evidence? |
| **Forecasting assistance** | Method selection, coding, feature/context assistance or explanation around a forecast | Broad support capability | Is the numerical forecast generated by a validated forecasting method? |
| **Root-cause analysis** | Attempt to establish why an outcome changed | Genuine diagnostic/causal task, often overclaimed by product terminology | Is causal identification possible from available evidence? |
| **Agentic analytics** | AI plans multi-step analytical work with some autonomy | Delegation/architecture distinction | How much unchecked discretion exists between request and result? |
| **Autonomous analytical agent** | Agent also entrusted to reach conclusions or trigger downstream actions with limited human gating | Material risk/delegation distinction | What validates the reasoning and constrains the action? |

Current semantic-layer implementations illustrate why the layer is materially different from raw text-to-SQL. dbt's current semantic models explicitly represent entities, dimensions, time dimensions and metrics; MetricFlow resolves and queries those objects rather than requiring every consumer to reconstruct metric SQL independently. citeturn14search0turn14search1 Snowflake's Verified Query Repository similarly adds reviewed question–SQL examples to a semantic model, while Looker grounds conversational analytics in its semantic modeling layer. Yet Looker's own documentation still warns that the conversational system can produce plausible but factually incorrect output and recommends validating it. citeturn13search15turn13search3

That is strong evidence against the proposition that **“semantic layer present” ⇒ “answer trustworthy.”** It reduces some classes of uncertainty; it does not make intent, source quality, causal interpretation, freshness, or model behavior unambiguous. This is a synthesis from the documented architectures and limitations, not a direct benchmark result. citeturn14search1turn13search3

**Analytical task taxonomy.** The following ratings are a synthesis of the benchmark and methodology evidence below. “Higher” never means safe for an unconstrained chat model: it means that the task admits comparatively strong controls when definitions and data are governed.

| Task | Analytical class | Evidence-based reliability potential | Why / principal boundary |
|---|---|---:|---|
| Finding an existing metric | Descriptive / retrieval | **Higher** | Catalog/semantic lookup is bounded when metric identity and access are explicit. Ambiguous synonyms remain a semantic problem. citeturn14search0turn13search15 |
| Retrieving a known value | Descriptive | **Higher** | Can reduce to deterministic metric execution; freshness and filter interpretation remain consequential. |
| Filtering and grouping | Descriptive | **Higher–conditional** | Reliable when dimensions, entities, grain and relationships are modeled; wrong joins/grain can silently change values. citeturn14search1turn13search18 |
| Generating SQL | Translation | **Conditional** | Benchmark performance ranges enormously with schema complexity, metadata, interaction and task setting. citeturn0academia12turn0academia13turn11view1 |
| Calculating derived metrics | Descriptive | **Conditional–higher if deterministic** | Formula execution is straightforward; inventing the formula or denominator is a semantic decision. |
| Comparing periods | Descriptive | **Higher–conditional** | Arithmetic is deterministic; date boundaries, timezone, fiscal calendar, comparable populations and late data are not automatically resolved. |
| Explaining a metric definition | Retrieval | **Higher if retrieval-grounded** | Strong when response is constrained to an authoritative, versioned definition; weak if the model synthesizes an undocumented meaning. |
| Producing a chart | Descriptive presentation | **Conditional** | Simple rendering can be deterministic after the query; deciding chart type, axes, aggregation and omissions introduces judgment. |
| Describing an observed change | Descriptive | **Conditional–higher** | “Revenue is 8% lower” can be checked; language must not silently become “because X happened.” |
| Detecting an anomaly | Diagnostic precursor | **Conditional** | Dedicated statistical/change-detection procedures can be validated; an LLM noticing “unusual” values is a different and weaker claim. |
| Generating hypotheses | Diagnostic support | **Useful, not truth-bearing** | LLMs can enumerate candidate explanations, but plausibility is not evidence that a hypothesis occurred. |
| Explaining why something happened | Diagnostic / causal | **Low without causal evidence** | General LLMs remain sensitive to confounding, encoding and incomplete causal structure. citeturn4search1turn4search3 |
| Forecasting | Predictive | **Conditional with dedicated models; weaker for direct LLM prediction** | Evidence on general-purpose LLM numerical forecasting is mixed; validated statistical/ML/time-series models and horizon-specific backtests remain the relevant oracle. citeturn5search0turn5search2turn5academia25 |
| Recommending an intervention | Prescriptive | **Low/strongly conditional** | Requires objectives, causal assumptions, costs, constraints and counterfactual reasoning not contained in a descriptive query result. |
| Executing follow-up actions | Action | **Separate safety problem** | Even a correct analysis does not establish authority or safety to mutate systems; authorization must be independently enforced. citeturn7search3turn8search7 |

The important pattern is monotonic only in broad terms, not as a rigid ladder. **Descriptive work is easier to validate because it often has a deterministic oracle. Diagnostic, predictive and prescriptive work require progressively more assumptions that cannot be checked by rerunning SQL.** A highly complex descriptive query can still be harder than a simple forecast, but moving from “what?” to “why?” to “what will happen?” to “what should we do?” changes the epistemic problem.

## End-to-end reliability chain and SQL evidence

The appropriate unit of analysis is the entire chain in the brief, not the generated query.

```mermaid
flowchart LR
    A[User question] --> B[Business interpretation]
    B --> C[Metric / definition]
    C --> D[Data-source selection]
    D --> E[Permissions]
    E --> F[Query / code generation]
    F --> G[Execution]
    G --> H[Calculation]
    H --> I[Validation]
    I --> J[Interpretation]
    J --> K[Explanation]
    K --> L[Visualization]
    L --> M[Decision / action]

    B -. ambiguity .-> X[Clarify / abstain]
    C -. conflicting definitions .-> X
    D -. stale or unsuitable source .-> X
    I -. validation failure .-> X
    J -. causal claim unsupported .-> X
```

This is effectively a **series system**: a perfect query does not repair an incorrect interpretation upstream, and a correct result does not repair an unjustified causal narrative downstream. That “weakest consequential stage” framing is an analytical synthesis rather than an empirical formula.

| Stage | Characteristic silent failure | What can provide evidence |
|---|---|---|
| User question | “Customers,” “revenue,” “active,” “last month” are underspecified | Clarification/refusal tests; corpus of real ambiguous questions |
| Business interpretation | Model silently chooses one of several legitimate readings | Intent labels, expected clarifications, SME review |
| Metric/definition | Wrong revenue metric or historical definition | Versioned metric registry, owner, validity dates, semantic tests |
| Source selection | Correct concept queried from wrong table/snapshot | Catalog and lineage assertions |
| Permissions | AI sees or returns data beyond user entitlement | Infrastructure-enforced access tests, adversarial security tests |
| Query generation | Wrong join, grain, predicate, null rule, date boundary | Execution/result tests plus targeted SQL invariants |
| Execution | Dialect error, timeout, truncation, partial result | Engine logs/query ID/status |
| Calculation | Model makes arithmetic/statistical error | Deterministic code/functions and numerical unit tests |
| Validation | “Gold” is itself wrong or leaks into generation | Independent held-out oracle, adjudication, test suites |
| Interpretation | Association becomes explanation | Claim typing and causal-evidence requirements |
| Explanation | Fluent prose adds unsupported details | Claim-level grounding/provenance checks |
| Visualization | Misleading denominator, axis, binning, aggregation | Chart-spec/result consistency tests |
| Decision/action | Valid observation becomes invalid intervention | Decision rules, causal evidence, constraints, independent authorization |

### What text-to-SQL benchmarks actually show

The most useful evidence is not a single leaderboard score but the enormous variance across evaluation settings.

The original Spider 2.0 study was explicitly designed around real enterprise workflows, with 632 problems, BigQuery and Snowflake databases, schemas that can exceed 1,000 columns, complex metadata/documentation, multiple SQL dialects and queries that can run beyond 100 lines. Its reported o1-preview agent baseline solved only **17.0%** of Spider 2.0 tasks, compared with much higher reported performance on older Spider 1.0 and BIRD settings. citeturn0academia12

By contrast, the current Spider 2.0-Snow leaderboard—an intentionally self-contained Snowflake setting with prepared metadata—shows top systems around **96.7%** as of the research snapshot. The benchmark itself describes Snow as 547 examples with well-prepared metadata and documentation. These numbers are not contradictory measurements of the same thing: architecture, model generation, task packaging and available context differ materially. citeturn10search8turn11view1

BIRD broadened earlier text-to-SQL work using 12,751 question-SQL pairs over 95 databases, 33.4 GB of data and 37 domains, deliberately incorporating dirty values and external knowledge. Its original paper reported 40.08% execution accuracy for ChatGPT versus 92.96% for humans in its evaluation. citeturn10academia47 Current systems are substantially stronger on the benchmark, but BIRD's own site also documents data-cleaning effects and newer tests intended to address ambiguity, large schemas, business-rule drift and interaction. citeturn11view0

The crucial stress test is **BIRD-Interact**. It turns database interaction into a dynamic problem involving ambiguity and user interaction rather than assuming that one static utterance fully specifies the task. The reported GPT-5 success rates were only **8.67% in conversational interaction and 17.00% in agentic interaction** on the benchmark's respective settings. citeturn0academia13 This is strong evidence that high one-shot SQL benchmark performance does not transfer automatically to the full business-question problem.

End-to-end data-analysis evaluations are similarly sobering. DA-Code evaluates agents using both Python and SQL on realistic data-analysis/wrangling problems; its strongest reported baseline achieved **30.5%** accuracy. citeturn12search1 Newer benchmarks such as DataSciBench and UniDataBench explicitly expand evaluation beyond SQL toward natural analytical prompts, files, multi-source data, summaries and recommendations, underscoring that database translation is only one component of analytical capability. citeturn12search2turn12search3

### Four levels of “SQL correctness”

**Syntactically valid SQL** means only that a parser accepts it.

**Executable SQL** means the database can run it. A query may still join the wrong population, apply the wrong period, or double-count entities.

**Result-equivalent SQL** means it returns the expected result under an evaluation oracle. This is substantially stronger, but a single database instance can hide semantic differences. Zhong, Yu and Klein's test-suite work was specifically motivated by weaknesses in exact SQL matching and single-database execution evaluation. citeturn9search1

**Business-correct SQL** means the query embodies the intended definition, population, grain, time semantics, exclusions, source and relevant organizational rules. Standard text-to-SQL execution accuracy does not establish this final property. Spider 2.0's enterprise context requirements and BIRD-Interact's ambiguity findings are direct evidence that context outside SQL syntax is material. citeturn0academia12turn0academia13

Recent research is moving toward verification rather than unconstrained generation. VET makes intermediate execution semantics inspectable; PV-SQL probes databases and applies rule-based verification; LatentRefusal explicitly treats underspecified or unanswerable questions as cases where a system should refuse rather than manufacture an executable answer. These approaches improve particular benchmark outcomes, but none converts execution success into proof of business correctness. citeturn0search1turn0search2turn3search2

## Business semantics, data quality, and deterministic computation

### Business-semantics requirements

Raw schema access is insufficient whenever the intended concept is not uniquely encoded in physical table and column names. Enterprise text-to-SQL work explicitly finds schema grounding, external metadata and business knowledge consequential at large schema sizes. citeturn0academia12turn3search10

A minimally meaningful semantic context for business analytics generally contains the following, although there is **no research-supported universal token count or document-size threshold**:

| Artifact | Question it answers | What remains unresolved |
|---|---|---|
| Metric definition | “Exactly what is revenue / ARR / active customer?” | Which version/domain is appropriate for this question? |
| Dimensions | “Along what attributes may this metric be sliced?” | Are values consistent historically? |
| Entities and keys | “What is a customer/order/account and how are objects related?” | Incorrect source keys can still contaminate the model |
| Grain | “One row per what?” | Hidden fanout may still arise downstream |
| Relationship semantics | “Which joins are valid and with what cardinality?” | Metadata can itself be incorrect |
| Business glossary | “What do users' terms and synonyms mean?” | Natural language can remain context-dependent |
| Table/column descriptions | “What does this field physically represent?” | Descriptions do not establish metric policy |
| Verified queries | “How was this known question correctly answered?” | Nearby question may differ in one consequential assumption |
| Data contracts | “What schema, quality/SLA and ownership expectations apply?” | A contract does not prove current compliance |
| Lineage | “Where did this output come from?” | Lineage describes derivation, not correctness |
| Ownership | “Who can resolve ambiguity and approve changes?” | Ownership is organizational, not an automatic semantic oracle |
| Version/effective time | “Which definition applied at the time?” | Historical restatements may require domain policy |

dbt's semantic specification makes entities, dimensions, aggregation-time dimensions and metrics explicit; its command layer can validate configs and query metrics with their permitted dimensions. citeturn14search0turn14search1 Open Data Contract Standard likewise treats schema, quality, SLA and role/ownership information as explicit contractual metadata rather than something downstream consumers should infer from values alone. citeturn3search7

**Two teams defining “revenue” differently is therefore not a language-model problem to solve probabilistically.** Unless organizational context selects one definition deterministically, the correct analytical behavior is to expose the competing definitions or seek disambiguation. LatentRefusal is relevant evidence that answerability detection can itself be modeled and benchmarked rather than treating refusal as a product failure. citeturn3search2

Verified queries help, but they are not ground truth by virtue of being stored. Snowflake's documentation describes them as reviewed natural-language-question/SQL pairs that can improve grounding; its evaluation mechanism temporarily removes selected verified queries so that the same examples do not directly leak into the evaluation being scored. citeturn13search15turn13search7 This distinction—**runtime exemplar versus evaluation oracle**—is important for credible reliability measurement.

### Data-quality dependencies

AI generally inherits unobserved data defects. It can detect a defect only when the data, metadata, constraints, tests, or statistical evidence make the defect identifiable. An LLM cannot infer that 12% of events never arrived if nothing in the accessible system states what should have arrived.

| Defect | Can an AI sometimes detect it? | Why it can remain silent |
|---|---|---|
| Missing values | Yes, when nulls or coverage gaps are visible | Missing records leave no row to inspect |
| Duplicates | Yes, if keys/expected uniqueness are known | Legitimate repeated events can resemble duplicates |
| Stale data | Only with refresh timestamps/SLAs or external comparison | Stale values can look perfectly plausible |
| Schema drift | Often, if schema/version contract changes are observable | Semantic drift can occur without physical schema change |
| Inconsistent dimensions | Sometimes via distribution/category checks | AI does not know which spelling/mapping is canonical without policy |
| Wrong joins | Sometimes through cardinality/fanout tests | Executable SQL rarely signals that the relationship is conceptually wrong |
| Incorrect grain | With explicit key/grain metadata or invariant tests | Aggregates can be numerically plausible |
| Slowly changing dimensions | With effective-date semantics | A naive current-state join may run correctly but rewrite history |
| Broken transformation | With tests/lineage/reference expectations | Internally consistent bad upstream output can propagate silently |
| Late-arriving data | With watermarks/completeness rules | Latest period may simply appear lower |
| Incomplete history | With coverage metadata | “No earlier rows” does not reveal whether history never existed or was omitted |

Benchmark curation itself demonstrates the issue. Research on noise in BIRD found erroneous or noisy benchmark questions and reference SQL capable of affecting measured rankings. citeturn0search4 BIRD's current site reports score changes after dataset quality corrections, while the 2026 ReViSQL preprint reports extensive errors in a studied portion of BIRD training data and materially higher performance after training on cleaned examples. The precise ReViSQL numbers should be treated as preprint evidence rather than settled benchmark fact, but the broader conclusion is strong: **a supposedly authoritative oracle can itself be a data-quality problem.** citeturn11view0turn10academia43

### Deterministic versus model-generated analysis

A critical distinction is between using a model to **choose a computation** and asking the model to **be the computation**.

| Pattern | Where nondeterminism/semantic risk resides | Reproducibility | Evidentiary strength |
|---|---|---:|---:|
| LLM directly states number/result | Interpretation, retrieval, arithmetic and wording may all be generated | Low–medium | Weakest |
| LLM writes SQL/Python; engine executes it | Program selection may be wrong; execution is deterministic | High once program/version/data are captured | Moderate |
| LLM selects governed metric + parameters | Semantic mapping is probabilistic; metric execution fixed | High | Strong for bounded descriptive tasks |
| Verified parameterized query | Question matching may remain probabilistic; computation fixed | Very high | Strong within covered intent |
| Analyst-written SQL | Human interpretation and code may be wrong, but intent is attributable/reviewable | High | Not automatically correct |
| Governed metric query without LLM | Meaning and execution fixed under versioned definitions | Very high | Strongest computational baseline for covered task |

Arithmetic, financial formulas, currency rules, aggregations, date logic, statistical tests and access control generally gain nothing from being re-invented token-by-token when a deterministic implementation is available. This is an architectural inference supported by the gap between query-generation uncertainty and deterministic semantic-layer/query execution in systems such as MetricFlow and Looker. citeturn14search0turn13search18

But deterministic execution solves only one layer. **Deterministically executing the wrong formula is still reliably wrong.**

## Self-service boundaries, human judgment, causality, and forecasting

### Meaningful self-service

Natural-language access lowers the skill needed to *ask* a question. It does not necessarily lower the skill needed to *know whether the answer is wrong*.

The strongest self-service boundary supported by current evidence is therefore:

> **Users who cannot independently validate an answer can be given trustworthy self-service only to the extent that the system itself has an independent validation path for the task, knows when the question is ambiguous, and can abstain or escalate outside that envelope.**

This conclusion follows from the large performance gap between bounded SQL settings and interactive ambiguous tasks, together with Google's explicit warning that conversational analytics can return plausible but factually incorrect output. citeturn0academia13turn13search1

The strongest current candidates for low-review self-service are existing-metric discovery, definition retrieval, known metric values, bounded dimensional cuts, simple period comparisons and straightforward visualizations—provided metric semantics, access, freshness and query provenance are controlled. More open-ended cohort design, custom derived metrics, multi-source reconciliation, data-quality judgment, causal explanation, forecasting and intervention selection retain substantial analytical-review requirements.

Whether AI **reduces analyst backlog or creates a review backlog** is presently less well supported by independent production evidence than benchmark claims about query generation. Current research demonstrates reduced friction and stronger benchmark execution in some constrained settings, but that does not establish net review effort across real organizations. This remains an unresolved production-measurement question rather than a defensible universal claim.

### Human-judgment boundaries

Human judgment remains especially consequential where the task requires choosing among legitimate interpretations rather than executing an already-specified transformation:

| Judgment | Automation trajectory | Present boundary |
|---|---|---|
| Detecting lexical ambiguity | Increasingly automatable | Models can learn to clarify/refuse, but must be evaluated for false confidence. citeturn3search2 |
| Selecting known metric from context | Increasingly automatable | Hard when definitions conflict across organizational domains |
| Applying known calculation | Highly automatable | Prefer deterministic code |
| Identifying suspicious data | Partly automatable | Deciding whether it invalidates a conclusion requires context |
| Choosing comparison baseline | Partly automatable | Often depends on business purpose and seasonality |
| Interpreting an anomaly | Useful AI assistance | Context and alternative mechanisms remain open |
| Distinguishing correlation from causation | Technically formalizable but difficult in open business settings | Requires assumptions/data/design absent from ordinary BI queries |
| Communicating uncertainty | Automatable in format | Underlying uncertainty must first be quantified correctly |
| Choosing intervention | Weakly automatable in general | Requires goals, causal effects, constraints and side effects |
| Accepting accountability for action | Not merely an analytical capability | Organizational/legal authority remains external to the model |

AgentDS reports that, on its set of industry-oriented data-science challenges, AI-only performance remained at or below median human performance while human-AI combinations performed best; as a 2026 preprint/competition-style result, it is suggestive rather than decisive evidence for broad production settings. citeturn12academia24

### Narrative analytics and the causality ladder

Generated explanations should be separated into five epistemically distinct levels:

| Statement type | Example | Minimum evidence that supports it |
|---|---|---|
| **Observed fact** | “Revenue decreased 8% month over month.” | Verified calculation |
| **Association** | “The decrease was concentrated among SMB customers.” | Correct segmented analysis |
| **Hypothesis** | “Higher churn may have contributed.” | Evidence of temporal/quantitative plausibility; clearly labeled hypothesis |
| **Causal explanation** | “Higher churn caused the revenue decrease.” | Causal identification assumptions/design plus supporting evidence |
| **Intervention claim** | “Reducing onboarding time will restore revenue.” | Estimated treatment effect, mechanism, costs/constraints and external-validity assumptions |

LLMs are especially dangerous when fluent language compresses those five categories into one sentence. CausalGraph2LLM evaluated more than 700,000 causal queries and found strong sensitivity of models including GPT-4 and Gemini to causal-graph encoding and contextual presentation. citeturn4search1 NoisyCausal similarly finds difficulties separating causal structure from noisy, confounded or partially observed information, while showing that explicit causal structure can improve performance. citeturn4search3

There is also positive evidence: specialized post-training can produce very high results on controlled causal-reasoning benchmarks. That does **not** establish reliable root-cause discovery from arbitrary enterprise observational tables because the benchmark may supply causal structure or task regularities that the enterprise problem does not. citeturn4search5

Anomaly analysis has the same category error:

**anomaly detection → change localization → correlation search → hypothesis generation → causal identification → root-cause confirmation**

are six different operations.

For example, Microsoft's documented “explain increase/decrease” functionality searches dimensions for categories associated with a change. That can be a useful diagnostic shortcut; it is not equivalent to experimental or quasi-experimental causal identification. citeturn4search7

A credible generated narrative should therefore expose which layer each statement occupies and, for causal-sounding conclusions, either cite the identification evidence or explicitly retain alternative explanations.

### Forecasting

The forecasting evidence is genuinely mixed because papers test substantially different architectures.

LLMTime showed that representing time series as token sequences could produce surprisingly competitive zero-shot results in some benchmark settings, while also reporting model/tokenization sensitivities and weak uncertainty calibration. citeturn5search0

The NeurIPS 2024 study *Are Language Models Actually Useful for Time Series Forecasting?* challenged the idea that pretrained language representations themselves were providing the benefit: removing LLM components often did not hurt and could improve forecasting, and pretraining was not consistently superior to training from scratch. citeturn5search2

More recent evaluations continue to favor a distinction between **LLM as analytical planner/interface** and **LLM as direct numerical forecaster**. MUSE-Bench reports stronger overall performance from dedicated time-series foundation models and inconsistent benefit from general-purpose LLM refinement, while TFRBench finds off-the-shelf LLMs struggling with direct numerical forecasting and time-series reasoning unless more structured grounding is supplied. citeturn5academia25turn5academia24

The resulting evidence map is:

| LLM role | Evidence position |
|---|---|
| Explain a forecast already produced | Useful, subject to grounding |
| Generate forecasting code | Useful if code and method are validated |
| Suggest candidate methods/features | Useful as hypothesis/planning support |
| Select among models using measured backtests | Plausible orchestration role |
| Rank/model external context | Potentially useful, but bad context can degrade results |
| Directly emit future numeric values | Weak/inconsistent basis for production trust |
| Emit calibrated uncertainty from prose alone | Not established |

Forecast validity itself should be assessed against naive and domain-relevant baselines using chronological/rolling-origin evaluation, not random train/test splits that leak the future. Hyndman and Athanasopoulos describe rolling forecasting-origin cross-validation and emphasize forecast distributions/prediction intervals because uncertainty generally grows with forecast horizon. citeturn15search0turn15search1

Thus “AI forecasting” is too broad to evaluate meaningfully. A trustworthy system needs a stated **forecast horizon, baseline, backtest period, error metric, uncertainty interval and regime assumptions**, irrespective of whether an LLM helped write or select the forecasting model.

## Freshness, provenance, permissions, and security

### Freshness is a separate correctness dimension

A stale answer can be **perfectly reproducible and mathematically correct for yesterday's snapshot** while being wrong for today's decision.

An AI cannot infer freshness from the plausibility of a value. It requires temporal metadata such as source refresh time, ingestion watermark, semantic-model version, metric effective dates, cache creation time or a service-level freshness rule. dbt, for example, models source freshness explicitly using warn/error thresholds against observed source timestamps; this is infrastructure metadata, not an LLM judgment. citeturn3search1turn3search3

A freshness-aware analytical claim needs at least:

`as-of time → source snapshot/version → latest successful refresh → freshness policy → query execution time → cache status`

Caching creates a distinct failure mode. A semantically correct cached answer becomes invalid when any dependency relevant to that answer changes: underlying data, metric definition, semantic model, access policy, business rule, timezone/calendar policy, or possibly the question's contextual interpretation. Treating only data refresh as an invalidation signal is therefore insufficient; this is an inference from the dependency structure.

Snowflake's Verified Query Repository itself includes `verified_at` metadata, illustrating that even semantic exemplars have a temporal dimension. citeturn13search15

### Provenance and reproducibility

A user should be able to distinguish **computed facts** from **generated interpretation**.

For a consequential analytical answer, reconstructability means retaining enough evidence to reproduce at least:

| Provenance element | Why it matters |
|---|---|
| Original user question | Establishes requested intent |
| Clarified interpretation | Reveals any semantic transformation |
| Metric/semantic definition + version | Establishes business meaning |
| Data source/snapshot identifiers | Establishes evidence base |
| Freshness/as-of timestamp | Establishes temporal validity |
| User/role authorization context | Establishes permitted view |
| Exact SQL/code/tool calls | Reproduces transformation |
| Database/engine/dialect | Explains execution semantics |
| Query/run ID and execution status | Links narrative to actual run |
| Parameters, timezone, date boundaries | Prevents hidden temporal differences |
| Result/validation artifacts | Shows what was checked |
| Chart specification | Enables visual reproduction |
| Model/system version | Identifies generative component |
| Generated narrative boundaries | Separates computation from interpretation |
| Upstream lineage | Enables source/transformation tracing |

OpenLineage provides a useful infrastructure analogy: its specification records run events, jobs, input/output datasets and explicit lineage edges, including event time and producing system; its dataset lineage facet can record the exact jobs/datasets/fields from which a dataset derives. citeturn13search0turn13search2turn13search5

That does not by itself prove an answer correct, but it makes failures inspectable. **Provenance converts an opaque claim into a falsifiable claim.**

### Permission and security boundaries

Authorization must be distinguished sharply from model alignment.

A prompt saying “only show rows this user is allowed to see” is not an access-control mechanism. Access should be enforced by the database, semantic platform, policy engine or similarly trusted infrastructure before data become available to model-controlled tools.

Current enterprise products illustrate this architecture. Databricks documents row filters and column masks enforced through Unity Catalog for Genie-related data access; Snowflake documents Cortex Analyst's use of existing role-based access controls. These are vendor capability statements, not independent reliability evidence, but they exemplify the correct trust boundary: **the model generates a request; infrastructure decides what it may access.** citeturn7search6turn7search5

Required security boundaries extend beyond read permissions:

| Boundary | Why prompt-only control is inadequate |
|---|---|
| Dataset access | Model can choose a tool/source unexpectedly |
| Row-level security | Prompt reasoning is probabilistic |
| Column-level masking | Sensitive attributes may appear through joins or summaries |
| Tenant isolation | One retrieval error can cross organizational boundaries |
| Read-only tooling | A hallucinated write command must be structurally impossible |
| Query/runtime limits | Agent loops can create runaway compute/cost |
| Spending limits | Token/query/warehouse expenditure is externally measurable |
| Action permissions | Analysis and mutation should not share unrestricted credentials |
| Metadata exposure | Table/column names can themselves reveal sensitive structure |
| Prompt injection | Retrieved content can attempt to redirect agent behavior |
| Data exfiltration | Tool use may move information into model-visible/output channels |

OWASP's generative-AI guidance distinguishes direct and indirect prompt injection and explicitly includes sensitive-data disclosure, unauthorized functionality and manipulated decisions among the risks. citeturn7search3turn7search1 Research on schema-inference attacks further shows that text-to-SQL interfaces can leak substantial information about underlying database schemas through probing even without returning arbitrary raw table data. citeturn3search8

Consequently, **correct authorization behavior must be penetration-tested as infrastructure behavior, not scored by asking a cooperative model whether it followed its system prompt.**

## Evaluation, benchmark map, and contradictions

### What a trustworthy evaluation actually has to measure

No single scalar such as “SQL accuracy” is sufficient. The evaluation target must match the analytical task.

| Evaluation dimension | Suitable question |
|---|---|
| SQL syntax | Does generated SQL parse? |
| Execution accuracy | Does it execute to the reference result? |
| Result correctness | Are returned values correct under the intended oracle? |
| Metric correctness | Did it use the intended governed metric/version? |
| Source selection | Did it choose authoritative data? |
| Join/grain correctness | Did it preserve intended population/cardinality? |
| Ambiguity recognition | Does it detect questions with multiple legitimate interpretations? |
| Clarification behavior | Does it ask the right question rather than guess? |
| Abstention | Does it refuse unanswerable requests? |
| Citation/provenance accuracy | Do cited sources actually support individual claims? |
| Reproducibility | Can another execution reconstruct the answer? |
| Data-quality awareness | Does it detect conditions that invalidate the answer? |
| Freshness | Is the answer valid for its claimed as-of time? |
| Narrative factuality | Is every factual sentence supported by computed/retrieved evidence? |
| Causal discipline | Are association, hypothesis and causation distinguished? |
| Chart correctness | Does visualization faithfully encode the verified result? |
| Forecast accuracy | Does it beat appropriate baselines at the stated horizon? |
| Uncertainty calibration | Do intervals/confidence statements behave as claimed? |
| Permission compliance | Can adversarial requests escape entitlements? |
| Latency/cost | Is reliability achieved within operating constraints? |
| Human-review effort | How many answers require intervention, and why? |

NIST's AI Risk Management Framework and generative-AI profile reinforce the broader principle that “valid and reliable” behavior is contextual and must be measured under specified conditions rather than inferred from generic model capability. citeturn8search17turn8search7

**Evaluation should also use the same distribution of ambiguity, dirty data, schema complexity and organizational semantics that the production system will face.** This is where many public benchmarks remain incomplete.

### Benchmark map

| Benchmark / evidence source | What it adds | Important blind spot |
|---|---|---|
| **Spider 1.0** | Classic cross-domain text-to-SQL | Too far from many modern enterprise workflows; increasingly saturated |
| **BIRD** | Larger DBs, values, external knowledge, dirty-data aspects | Static questions; benchmark/gold quality has required correction. citeturn10academia47turn0search4 |
| **Spider 2.0** | Enterprise schemas, metadata, Snowflake/BigQuery, workflow complexity | Still benchmark-defined tasks rather than an organization's live semantic ambiguity. citeturn0academia12 |
| **Spider 2.0-Snow** | Controlled, self-contained enterprise-like Snowflake task | High scores should not be generalized to unconstrained enterprise analytics. citeturn10search8turn11view1 |
| **BIRD-Interact** | Ambiguity, dynamic user interaction, agent setting | Newer and smaller than established static corpora. citeturn0academia13 |
| **LiveSQLBench** | Large schemas, evolving/business-rule-drift orientation, contamination-aware evaluation | Still SQL-centered and relatively new. citeturn11view0 |
| **DA-Code** | SQL + Python + data-analysis workflow | Limited coverage of enterprise governance/authorization. citeturn12search1 |
| **DataSciBench** | More open-ended data-science tasks and uncertain ground truth | Evaluation of open-ended analysis remains intrinsically difficult. citeturn12search2 |
| **UniDataBench** | Structured, semi-structured and unstructured multi-source analysis | New; not a mature proxy for all enterprise environments. citeturn12search3 |
| **CausalGraph2LLM / NoisyCausal** | Explicit causal reasoning stress tests | Controlled causal tasks differ from real causal identification in companies. citeturn4search1turn4search3 |
| **LLMTime / newer forecasting benchmarks** | Tests direct/general LLM time-series capabilities | Results vary with representation, model class and forecasting architecture. citeturn5search0turn5search2turn5academia25 |

No benchmark in this set simultaneously captures all of: changing enterprise schemas, dirty data, multiple conflicting metric definitions, authorization, temporal freshness, adversarial inputs, real conversational ambiguity, multi-step analysis, causal claims, visualization, forecast uncertainty and real downstream actions. That is a synthesis from their published scopes, not a claim made by any one benchmark.

### Contradictions and how to interpret them

**Very low Spider 2.0 performance versus roughly 97% Spider 2.0-Snow.**  
This is not evidence that one paper is wrong. The original Spider 2.0 setting exposed broad enterprise-workflow complexity, while Snow is a narrower, self-contained Snowflake track with prepared context; systems also improved substantially from 2024 to 2026. The disagreement is primarily **task definition + architecture + model capability + available context**, not conflicting evidence. citeturn0academia12turn11view1

**High modern static-SQL scores versus GPT-5's 8.67%/17% BIRD-Interact results.**  
Again, the target changed. Interaction introduces ambiguous requests, clarification decisions and dynamic state that one-shot execution accuracy does not measure. This is mainly **benchmark/task-definition disagreement**. citeturn0academia13turn11view0

**Semantic-layer claims of “trusted self-service” versus vendor warnings that answers can be factually wrong.**  
These statements can coexist: semantic models constrain business meaning, but the language model can still misinterpret the request or produce incorrect output. Google's own documentation contains both the governed-semantic-layer positioning and the warning to validate generated output. This is **architecture versus residual model-error**, not literal contradiction. citeturn13search3

A 2026 paired preprint comparing text-to-SQL with and without compact hand-authored semantics reports improvements on the order of 17–23 percentage points for tested models, yet semantic-layer conditions still remained well below perfect correctness. Because this is a preprint on a small controlled benchmark, it is supportive rather than definitive evidence. citeturn9academia25

**LLMTime's positive forecasting results versus later skepticism about LLM forecasting.**  
LLMTime tests a particular tokenized zero-shot formulation; later ablation work asks whether language-model pretraining is actually responsible for forecasting quality; newer benchmarks compare different specialized and general architectures. The disagreement arises from **model architecture, representation and benchmark design**, not merely replication failure. citeturn5search0turn5search2turn5academia25

**General LLM causal weakness versus very high specialized causal benchmark scores.**  
Structured causal training can teach models to solve formalized benchmark tasks. That does not establish that an LLM can identify the causal structure of a business process from ordinary observational warehouse tables. The difference is **availability of causal structure + post-training + benchmark scope**. citeturn4search3turn4search5

**Human-level or near-human scores on cleaned subsets versus materially lower full-benchmark results.**  
ReViSQL's strong results on expert-verified subsets and BIRD's broader leaderboard should not be treated as the same measurement. Dataset cleaning, subset selection, oracle quality and evaluation conditions matter significantly. This is principally **data quality + benchmark scope**. citeturn10academia43turn11view0

The general lesson is that benchmark scores require a five-part descriptor:

> **task × dataset × context supplied × system architecture × evaluation metric**

A percentage without those qualifiers is weak evidence for production trust.

## Failure catalogue, claim-to-source ledger, unresolved questions, and frontier

### Failure-mode catalogue

The following catalogue captures the most important ways an AI analytics system can produce an answer that looks competent while being untrustworthy.

| Failure class | Example | Why ordinary SQL execution may not catch it |
|---|---|---|
| Intent substitution | “customers” interpreted as paying accounts instead of end users | Both are queryable |
| Metric substitution | Booked revenue used instead of recognized revenue | Both return plausible numbers |
| Definition-version error | Current churn logic applied historically | Query is valid |
| Wrong data source | Replica/extract used instead of authoritative table | Values may be similar |
| Staleness | Yesterday's incomplete snapshot returned today | No syntax/runtime error |
| Wrong grain | Order-line rows treated as orders | Aggregates inflate silently |
| Join fanout | Customer ↔ orders ↔ events duplicates revenue | Query executes normally |
| Join omission | Customers without activity disappear | Results can look plausible |
| Wrong population | Internal/test users included | No database error |
| Predicate error | `>` instead of `>=`, wrong status list | Often tiny but material discrepancies |
| Null semantics | `NOT IN`/null treatment changes population | Valid SQL |
| Time-boundary error | UTC instead of local/fiscal period | Valid SQL |
| Slowly-changing-dimension error | Current segment applied to historical transactions | Historically misleading but consistent |
| Late data | Current period appears to collapse | Correct for incomplete snapshot |
| Arithmetic hallucination | Model computes ratio in prose incorrectly | Query may have returned correct inputs |
| Gold-query contamination | Evaluation example also available as runtime exemplar | Inflated benchmark score |
| Bad reference answer | Benchmark gold SQL itself incorrect | Correct model may be marked wrong |
| Unsupported narrative | “Churn caused decline” from correlation | SQL cannot test causal identification |
| Cherry-picked explanation | One correlate reported without alternatives | Factual subset can still mislead |
| Forecast leakage | Future information enters features/evaluation | Excellent historical score, invalid future estimate |
| Uncalibrated forecast | Point estimate presented as certainty | No query error |
| Chart misrepresentation | Wrong denominator/axis/aggregation | Underlying numbers may be correct |
| Permission bypass | Model tool sees rows user should not | Correct query, prohibited result |
| Indirect prompt injection | Retrieved metadata/document instructs agent to leak or act | Model sees attack as context |
| Schema leakage | Probing reveals sensitive database structure | Raw rows need not be returned |
| Autonomous-action overreach | Analytical conclusion triggers write/action without sufficient evidence | Analytics can be right while action is unsafe |

### Claim-to-source table

“Supported” means the cited evidence directly supports the claim in its own tested/documented context. “Conditional” means extrapolation is restricted. “Unresolved” identifies evidence gaps. All sources below were inspected for this dossier on **2026-10-03**.

| Exact claim | Evidence type / analytical task / context | Source date | Direct source | Status and limitation |
|---|---|---:|---|---|
| Enterprise text-to-SQL remains much harder than legacy academic settings when schemas, metadata and workflows become realistic; Spider 2.0 reported 17.0% for its o1-preview agent baseline. | Empirical benchmark; SQL generation; enterprise BigQuery/Snowflake workflows | 2024 | [Spider 2.0, arXiv](https://arxiv.org/abs/2411.07763) citeturn0academia12 | **Supported.** Baseline is historical and not current frontier performance. |
| A controlled/self-contained Spider 2.0-Snow track now has top scores around 96.7%. | Official benchmark leaderboard; SQL generation | 2026 snapshot | [Spider 2.0 official site](https://spider2-sql.github.io/) citeturn11view1 | **Supported.** Must not be equated with unrestricted enterprise analytics. |
| BIRD was designed to add larger data, dirty values and external knowledge beyond classic text-to-SQL datasets. | Peer-reviewed/benchmark paper; SQL generation | 2023 | [BIRD, arXiv](https://arxiv.org/abs/2305.03111) citeturn10academia47 | **Supported.** Still primarily static text-to-SQL. |
| Interactive ambiguity remains far harder: BIRD-Interact reports GPT-5 success of 8.67% and 17.00% in its two interaction settings. | Empirical benchmark; conversational/agentic SQL | 2025/2026 | [BIRD-Interact, arXiv](https://arxiv.org/abs/2510.05318) citeturn0academia13 | **Supported.** Specific benchmark/system configuration. |
| Executing SQL is not a sufficient semantic evaluation criterion; richer test suites reduce failures of exact-match/single-instance evaluation. | Peer-reviewed methodology; SQL evaluation | 2020 | [DOI 10.18653/v1/2020.emnlp-main.29](https://doi.org/10.18653/v1/2020.emnlp-main.29) citeturn9search1 | **Supported.** Evaluates SQL semantics, not organizational business meaning. |
| Systems can be trained to recognize underspecified/unanswerable database questions and abstain rather than generate misleading programs. | Peer-reviewed benchmark/method; ambiguity/refusal | 2026 | [DOI 10.18653/v1/2026.findings-acl.1007](https://doi.org/10.18653/v1/2026.findings-acl.1007) citeturn3search2 | **Supported.** Benchmark refusal performance does not guarantee deployment calibration. |
| Verification/probing improves text-to-SQL reliability but does not reach universal correctness. | Peer-reviewed method; SQL verification | 2026 | [PV-SQL DOI 10.18653/v1/2026.findings-acl.1286](https://doi.org/10.18653/v1/2026.findings-acl.1286) citeturn0search2 | **Supported.** Improvement over baseline, not proof of business correctness. |
| End-to-end data analysis is materially harder than SQL execution alone; DA-Code's strongest baseline reported 30.5% accuracy. | Peer-reviewed benchmark; SQL+Python analytics | 2024 | [DOI 10.18653/v1/2024.emnlp-main.748](https://doi.org/10.18653/v1/2024.emnlp-main.748) citeturn12search1 | **Supported.** Benchmark scope is not all enterprise analytics. |
| A semantic layer represents metrics, entities and dimensions that can be queried independently of raw physical schema. | Official documentation; semantic governance | Living docs | [dbt semantic models](https://docs.getdbt.com/docs/build/semantic-models) citeturn14search1 | **Supported as documented capability.** Not independent reliability proof. |
| Verified query repositories use reviewed NL question/SQL pairs as grounding and can also supply evaluation ground truth, with selected test VQs removed during evaluation. | Official documentation; semantic grounding/evaluation | Living docs | [Snowflake VQR](https://docs.snowflake.com/user-guide/snowflake-cortex/cortex-analyst/verified-query-repository) and [evaluations](https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-analyst-evaluations) citeturn13search15turn13search7 | **Supported as product behavior.** Vendor documentation is not independent proof of achieved accuracy. |
| A governed semantic model does not guarantee correct conversational output; Google explicitly warns that plausible but factually incorrect results can occur. | Official limitation documentation; conversational analytics | Living docs | [Looker Conversational Analytics](https://docs.cloud.google.com/looker/docs/conversational-analytics-overview) citeturn13search3 | **Supported.** Product-specific but directly refutes blanket semantic-layer guarantees. |
| General LLM causal reasoning is sensitive to representation/context and remains unreliable under noisy/confounded settings. | Peer-reviewed empirical studies; causal reasoning | 2025–2026 | [CausalGraph2LLM DOI 10.18653/v1/2025.findings-naacl.110](https://doi.org/10.18653/v1/2025.findings-naacl.110); [NoisyCausal DOI 10.18653/v1/2026.acl-long.1833](https://doi.org/10.18653/v1/2026.acl-long.1833) citeturn4search1turn4search3 | **Supported.** Formal causal QA is not identical to applied causal inference. |
| Specialized causal post-training can dramatically improve controlled causal-reasoning benchmarks. | Peer-reviewed empirical study | 2026 | [DOI 10.18653/v1/2026.findings-acl.839](https://doi.org/10.18653/v1/2026.findings-acl.839) citeturn4search5 | **Conditional.** Does not demonstrate autonomous root-cause identification in arbitrary enterprise data. |
| Evidence on direct LLM time-series forecasting is mixed; later ablations question whether language-model pretraining provides the forecasting benefit. | Peer-reviewed forecasting studies | 2023–2024 | [LLMTime DOI 10.52202/075280-0861](https://doi.org/10.52202/075280-0861); [NeurIPS 2024 DOI 10.52202/079017-1922](https://doi.org/10.52202/079017-1922) citeturn5search0turn5search2 | **Supported as disagreement.** Methods and representations differ. |
| Forecast evaluation should respect temporal order and uncertainty; rolling-origin cross-validation and prediction intervals address these separately. | Academic methodology | Current 3rd ed. | [Forecasting: Principles and Practice — CV](https://otexts.com/fpp3/tscv.html); [prediction intervals](https://otexts.com/fpp3/prediction-intervals.html) citeturn15search0turn15search1 | **Supported methodology.** Does not prescribe one universal metric/model. |
| Indirect prompt injection can produce sensitive disclosure or unauthorized behavior in tool-using LLM systems. | Security standard/guidance | 2025 | [OWASP GenAI Prompt Injection](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) citeturn7search3 | **Supported risk class.** Attack success depends on architecture and defenses. |
| Text-to-SQL interfaces can leak schema information through probing. | Peer-reviewed security research | 2025 | [DOI 10.18653/v1/2025.findings-naacl.386](https://doi.org/10.18653/v1/2025.findings-naacl.386) citeturn3search8 | **Supported.** Specific attack/setup; not universal compromise. |
| Lineage systems can record explicit run, job and dataset dependencies needed for reconstructability. | Open specification | v1.53 / 2026 inspection | [OpenLineage API](https://openlineage.io/apidocs/openapi/); [dataset lineage](https://openlineage.io/docs/spec/facets/dataset-facets/lineage/) citeturn13search0turn13search5 | **Supported capability.** Lineage improves auditability; it does not establish analytical correctness. |
| Trustworthiness should be validated for specified intended conditions rather than assumed from generic model behavior. | Government risk/validation framework | 2023–2024 | [NIST AI RMF 1.0 DOI 10.6028/NIST.AI.100-1](https://doi.org/10.6028/NIST.AI.100-1); [NIST GAI Profile DOI 10.6028/NIST.AI.600-1](https://doi.org/10.6028/NIST.AI.600-1) citeturn8search17turn8search7 | **Supported framing.** Framework is deliberately use-case agnostic. |

### Unresolved questions

The evidence base is advancing quickly, but several questions remain materially unresolved.

**Production business correctness.** There is still no widely accepted benchmark that starts with authentic enterprise user questions and simultaneously scores interpretation, metric choice, source selection, SQL, freshness, permissions, narrative correctness and decision relevance. Spider 2.0 and BIRD-Interact move meaningfully toward this goal but cover only parts of the chain. citeturn0academia12turn0academia13

**Semantic-layer effect size.** It is intuitively and architecturally plausible that well-designed semantics improve reliability, and emerging paired studies support sizeable gains, but robust independent multi-company estimates across models, schema types and query distributions remain sparse. citeturn9academia25

**Human review economics.** Published evidence remains inadequate to establish whether conversational analytics reduces total analyst labor after semantic curation, benchmark creation, exception handling, user support and answer review are counted.

**Longitudinal drift.** Current benchmarks rarely measure a deployed agent across months of metric-definition changes, schema migrations, historical restatements and evolving business vocabulary.

**Confidence calibration.** There is no generally validated mapping from an LLM's verbal confidence or self-evaluation to business-answer correctness. Trust should therefore be based on external evidence rather than how certain the model sounds.

**Automatic detection of data-quality invalidity.** It remains unclear what fraction of real analytical data defects an agent can independently detect when the defect is not already encoded as a test, constraint, contract, lineage event or explicit anomaly.

**Real causal analytics.** Strong performance on formal causal questions does not answer whether an agent can reliably discover confounding structures, construct defensible identification strategies and communicate limitations from messy enterprise observational data. citeturn4search3turn4search5

**Safe autonomous action.** Evidence that an agent can produce correct descriptive analytics is not evidence that it can safely choose and execute consequential interventions. This remains a distinct research problem spanning causal inference, planning, authorization and organizational governance. citeturn8search7turn7search3

### Research frontier

The field appears to be shifting from **“make the model better at SQL”** toward **“make the analytical process verifiable.”** Several research directions support that interpretation.

**Answerability and abstention** move the objective from always producing a query to recognizing when the information supplied does not determine one. LatentRefusal is a representative example. citeturn3search2

**Execution-grounded verification** decomposes queries into checkable intermediate steps or probes the database before accepting a program, as in VET and PV-SQL. citeturn0search1turn0search2

**Interactive evaluation** introduces the missing behavior of asking users for information rather than silently resolving ambiguity. BIRD-Interact demonstrates both the importance and present difficulty of this shift. citeturn0academia13

**Large-schema and schema-robustness research** is exposing a deeper limitation: logically equivalent data represented under different schemas can cause materially different model behavior, so raw model “reasoning ability” cannot be cleanly separated from schema representation and grounding. citeturn3academia36turn3search10

**Semantic grounding** increasingly treats business metrics, entities, relationships and verified exemplars as first-class model context rather than hoping an LLM reconstructs organizational meaning from raw DDL. Current vendor architectures and emerging independent experiments point in this direction, although the magnitude of improvement remains context-specific. citeturn14search1turn13search15turn9academia25

**AI-native database functions** are expanding evaluation beyond conventional SQL. Spider 2.0-AIFunc, for example, evaluates SQL programs containing AI functions over verified database tasks and still reports substantial error rates in areas including predicate construction, schema grounding and function parameterization. citeturn0academia15

**Full analytical-agent benchmarks** are moving beyond a single SQL answer toward data engineering, Python, multiple data modalities, uncertain ground truth and multi-step analysis. DA-Code, DataSciBench, UniDataBench and related work are early evidence that this broader target is considerably harder. citeturn12search1turn12search2turn12search3

The most defensible synthesis as of **3 October 2026** is therefore:

> **AI analytics is becoming credible first where ambiguity can be bounded, business semantics are externalized, computations are deterministic, permissions and freshness are infrastructure-enforced, and correctness has a machine- or human-verifiable oracle.**
>
> **Human analytical judgment remains most necessary where the task itself is underspecified, the data may be unfit, definitions legitimately conflict, causal structure must be inferred, future regimes matter, or an observation must be converted into an intervention.**
>
> **The decisive evidence of trust is not that the model produced plausible SQL or prose. It is that the complete question-to-answer chain has been evaluated against the failures that could make that particular analytical use wrong.**