# Empirical Reliability of LLM-Based Systems for Analytics over Structured Company Data

**Research snapshot: October 3, 2026**

## Executive synthesis and analytical task construct map

The most defensible conclusion from the empirical literature is narrower than the phrase “AI analyst” suggests:

> **Current evidence supports LLMs as useful components in bounded analytical workflows, especially for translating explicit questions into queries, orchestrating deterministic computation, and drafting descriptions from already-correct results. It does not establish that present systems are reliably correct end to end when they must infer an organization’s business meaning, select the right data and grain, recover from ambiguity, perform multi-step investigation, and produce a faithful business interpretation.**

The strongest evidence for that distinction is the gap between mature academic benchmarks and more realistic successors. Spider contains 10,181 questions, 5,693 distinct SQL queries across 200 databases and 138 domains, and was deliberately designed as a cross-domain challenge; contemporary systems have subsequently reached very high performance on it. Yet Spider 2.0, built around 632 enterprise-style workflows with BigQuery and Snowflake, databases that often exceed 1,000 columns, metadata/documentation search, data transformations, and sometimes more than 100 lines of SQL, reported only **17.0% task success for an o1-preview code-agent configuration**, despite reporting 91.2% on Spider 1.0 and 73.0% on BIRD in its comparison. The difference is primarily a difference in construct and environment, not evidence that models suddenly forgot SQL. citeturn0search9turn1search4turn1search1

BIRD moved substantially toward harder database-grounded evaluation—12,751 question–SQL pairs, 95 large databases, 33.4 GB of data and more than 37 professional domains—but it still fundamentally evaluates a curated mapping from a question to an intended query/result. BEAVER goes further by using private enterprise data warehouses, actual query logs and explicitly decomposing failures into table retrieval, join-key detection, column mapping, domain-knowledge extraction and query decomposition; its current public description covers 9,128 queries and 812 tables across 19 domains. These benchmarks are better evidence about enterprise querying, but neither by itself validates the complete business-analytics chain. citeturn1search2turn3search0turn3search6

The right reliability construct is therefore:

**business intent → business definition → data/schema grounding → grain/join choice → query/computation → result validation → statistical interpretation → narrative → decision**

Every arrow can fail independently, and success at one stage does not identify reliability at another.

### Analytical task construct map

| Task class | What is actually being tested | Evidence state as of snapshot | What a good score does **not** establish |
|---|---|---|---|
| **Schema linking** | Mapping linguistic entities to tables/columns/relationships | Substantial benchmark evidence; retrieval/ranking often helps on Spider/BIRD-style tasks, but large-schema enterprise grounding remains difficult. citeturn7academia38turn7search0turn3search6 | Correct business metric, grain, SQL or interpretation |
| **Text-to-SQL** | Translating sufficiently specified intent plus schema/context into executable SQL | Strong evidence on curated benchmarks; sharply weaker evidence on enterprise workflows. citeturn0search9turn1search2turn1search4 | Production analytical correctness |
| **SQL correction/debugging** | Diagnosing and repairing an existing SQL issue | Separate and still difficult skill: BIRD-CRITIC reports only 38.87% for o3-mini on its PostgreSQL set and 33.33% on the multi-dialect setting; SQL-IQ also shows correction materially weaker than ordinary generation for many models. citeturn15search8turn15search6 | Reliable self-correction of an agent’s own semantic mistakes |
| **Table QA** | Extracting/filtering/comparing values from one or more tables/text documents | Well studied; large remaining gaps on arithmetic and multi-hop tasks, especially in older controlled benchmarks. citeturn9search0turn9search1turn9search2 | Schema discovery or organizational semantics |
| **Numerical reasoning** | Arithmetic, ratios, percentages, aggregation and symbolic operations | Direct language generation is less reliable than external computation; program/code execution consistently helps. citeturn11search0turn11search1turn11academia27 | Correct selection of the numbers or business formula |
| **Multi-table reasoning** | Combining evidence across tables/documents and multiple operations | Demonstrated but materially harder than single-table cases. citeturn9search2turn3search6 | Correct warehouse grain or avoidance of fanout |
| **Metric selection** | Choosing the organization-specific definition of “revenue,” “active,” “margin,” etc. | Thin independent academic evidence; promising small controlled semantic-layer/provider benchmarks. citeturn17search6turn17search11turn17search17 | That the metric definition itself is authoritative |
| **Chart reasoning** | Reading values, relations and derived facts from a rendered visualization | Benchmark capability is real but highly sensitive to benchmark realism: a reported Claude Sonnet 3.5 result falls from 90.5% on ChartQA to 55.81% on ChartQAPro. citeturn19search6turn19search26 | Faithful explanation of why a trend occurred |
| **Chart generation** | Turning data/intent into a chart/specification | Technically distinct from chart QA; production business correctness is inadequately established in the evidence reviewed | Correct metric, denominator, grain or visualization choice |
| **Anomaly detection** | Detecting statistically unusual observations or changes | No strong evidence found here that a general LLM should replace dedicated statistical detectors | That an anomaly is meaningful or causal |
| **Anomaly explanation** | Proposing reasons for an observed anomaly | Much weaker construct; overlaps hypothesis generation and causal reasoning | That a proposed reason caused the anomaly |
| **Forecasting** | Predicting future values | Mixed-to-negative evidence for the LLM component itself; strong language ability is not forecasting evidence. citeturn20search1turn20academia20 | Forecast calibration, causal understanding or decision usefulness |
| **Causal reasoning** | Inferring effects/causal structure under assumptions | Models can improve under algorithmic prompting, but benchmark evidence still shows major weakness distinguishing causal claims from correlational premises. citeturn20academia19turn20academia21 | Autonomous causal inference from arbitrary company data |
| **Multi-step data analysis** | Iterative query → inspect → hypothesize → query → synthesize | Bounded SQL agents work, but realistic workflow success is far below simpler benchmark scores; end-to-end open-ended analysis remains inadequately validated. citeturn1search4turn7search0 | Reproducible autonomous analyst behavior |
| **Analytical narrative generation** | Converting correct data/results into prose | Language quality is strong; factual/causal faithfulness remains a separate evaluation problem. New chart-explanation datasets explicitly treat explanation as its own task. citeturn19search34turn16search10 | Truth, causality, significance or completeness |

This separation matters empirically. SQL-IQ, for example, evaluates seven SQL-related constructs separately and shows major within-model differences: its current leaderboard has models that perform substantially better at text-to-SQL or equivalence judging than at SQL correction. That is direct evidence against treating “SQL intelligence,” let alone “analytics intelligence,” as one scalar capability. citeturn15search6

A useful four-level distinction for query evaluation is also essential:

| Evaluation level | What counts as correct | Main weakness |
|---|---|---|
| **Exact match** | Generated SQL structurally/textually matches reference SQL | Rejects semantically equivalent SQL |
| **Execution accuracy** | Generated query executes and matches the reference result on the benchmark database | Can accept semantically wrong queries that happen to return the same result on one database state |
| **Result/semantic equivalence** | Alternative queries are tested more carefully for equivalent behavior, often using additional test databases or richer adjudication | Still asks whether the query matches the benchmark’s intended semantics |
| **Business correctness** | Query answers what the organization actually means, with correct metric definition, scope, grain, temporal convention and authorization | Rarely measured directly in public benchmarks |

Test Suite Accuracy was introduced precisely because earlier Spider evaluation produced false negatives for semantically equivalent queries; across 21 Spider systems, manual verification of a sample found an average 2.5% false-negative rate under the older metric, reaching 8.1% for the worst system. FLEX later showed that ordinary execution accuracy also produces false positives and false negatives; its expert-oriented evaluator raised agreement with human experts from Cohen’s κ=0.62 to 0.8704 and could change system rankings. Neither development solves the further question of whether the benchmark’s intended SQL embodies the organization’s correct business definition. citeturn5search0turn5search4turn2search30

**Evidence boundary:** deterministic execution solves *computation once the computation has been specified*. It does not make the specification deterministic.

## Text-to-SQL, benchmark validity, schema grounding and business semantics

### Text-to-SQL evidence map

The historical progression of benchmarks is more informative than any single leaderboard.

| Benchmark | Scale/schema | What it adds | Ambiguity & business semantics | Dirty/dynamic data & permissions | Valid inference |
|---|---|---|---|---|---|
| **Spider** | 10,181 NL questions; 5,693 unique SQL; 200 DBs; 138 domains. citeturn0search9 | Cross-domain database split, structurally complex SQL | Annotator-created questions target a known intended SQL; organizational ambiguity is not the main construct | Static DBs; no user-specific permission construct in the benchmark definition | Strong evidence for cross-domain semantic parsing on relatively bounded schemas; weak evidence for production analytics |
| **BIRD** | 12,751 pairs; 95 DBs; 33.4 GB; >37 professional domains. citeturn1search2 | Larger databases, contents and external knowledge | More business/domain context than Spider, but still paired with target SQL | Static benchmark snapshot; authorization not central | Better database-grounded SQL evidence, not end-to-end organizational interpretation |
| **KaggleDBQA** | Real web databases rather than cleaned benchmark-only schemas | Real formatting, domain-specific data types, natural database documentation. citeturn5search6 | More naturally occurring domain knowledge | More realistic messiness; still static | Useful robustness step toward real DBs |
| **Dr.Spider** | Perturbation suite over Spider | 17 robustness perturbations across question/schema/SQL dimensions | Tests invariance, not latent corporate metric meaning | Simulated perturbation rather than live warehouse drift | Strong evidence that benchmark accuracy is brittle: even the most robust evaluated model lost 14.0% overall and 50.7% on the hardest perturbation. citeturn5search1turn5search17 |
| **Spider 2.0** | 632 enterprise-workflow problems; schemas often >1,000 columns | BigQuery/Snowflake, metadata search, project code, long transformations, multi-query workflows | Substantially more realistic contextual interpretation | More realistic platform complexity, but still curated/static tasks rather than live authorization/drift | Strong evidence that Spider/BIRD success does not transfer directly to enterprise workflows. citeturn1search1turn1search4 |
| **BEAVER** | Current site: 9,128 queries, 812 tables, 19 domains | Private enterprise warehouses, query logs, business/domain knowledge; table retrieval/join-key/column mapping subtasks | Explicitly addresses enterprise domain knowledge | Stronger private-data realism; still benchmark evaluation rather than longitudinal operations | One of the strongest available bridges to enterprise SQL; off-the-shelf LLMs remain poor even with prompt engineering/RAG. citeturn3search0turn3search6 |
| **BIRD-CRITIC / SWE-SQL** | 530 PostgreSQL + 570 multi-dialect tasks in the research paper; project includes PostgreSQL, MySQL, SQL Server, Oracle variants | Realistic user-reported SQL debugging rather than generation | Intent supplied through an issue/debugging context | Multi-dialect and executable environments | Shows production-like SQL repair is a distinct unsolved capability. citeturn15search1turn15search8 |
| **PRACTIQ** | Conversational ambiguous/unanswerable cases | Explicit clarification and unanswerability | **Directly tests ambiguity**; four-turn interaction | Not a live enterprise environment | Particularly important because it exposes a construct omitted from conventional benchmarks. SOTA baselines struggled. citeturn13academia39 |
| **AIGT2S** | 18k Q–SQL pairs, 113 DBs; 55% “Ultra” complexity | Very high structural complexity | Generated from expert seeds with verification | Synthetic/LLM-assisted construction | Useful complexity stress test; maximum reported baseline execution accuracy was 56%, but synthetic construction limits business-realism inference. citeturn2search13 |
| **Spider 2.0-AIFunc** | 465 verified tasks, 125 real-world DBs, six Snowflake AI functions | AI-native SQL workflows and function parameterization | NL was deliberately refined to reduce ambiguity | Cloud-dialect realism; still curated | Current frontier systems reach only about 67–70% execution accuracy; strongest open model reported 58.1%. citeturn1search23 |
| **SQL-IQ** | 17 models × seven SQL task families in current benchmark | Generation, conversational SQL, judging, equivalence, error classification, correction, dialect translation | Holistic SQL capability rather than one score | Does not itself solve org semantics | Excellent evidence that capability is task-dependent; current best macro-average shown is below 60. citeturn15search6 |

A central validity problem is **question construction**. Traditional text-to-SQL fundamentally requires an intended SQL target. That encourages datasets in which the intended interpretation is already sufficiently specified for annotators to map it to one answer. PRACTIQ makes this limitation explicit: its motivation is that previous text-to-SQL datasets and systems focused primarily on questions with clear intentions that are answerable, whereas real questions may be ambiguous or unanswerable. citeturn13academia39

Consequently, a benchmark question like “What was revenue last quarter?” normally cannot function as a conventional one-gold-SQL item until someone has implicitly or explicitly decided what “revenue” and “last quarter” mean. The genuinely organizational version might require resolving gross versus net, booked versus recognized, reporting currency, fiscal versus calendar quarter, entity scope and exclusions. **I found no published audit that quantifies, across Spider/BIRD/Spider 2.0, what percentage of questions require such unresolved organizational semantics.** The empirical basis for saying current models “understand revenue,” “understand churn” or “understand the business” is therefore much weaker than the basis for saying they map specified concepts to SQL.

### Schema grounding and the large-schema effect

Schema grounding does improve with engineered context selection. RESDSQL explicitly decouples schema linking from SQL skeleton generation and ranks schema items before generation, improving Spider and robustness variants. CHESS combines information retrieval, schema selection, candidate generation and unit testing; on large schemas its schema selector reportedly improved accuracy by about two percentage points while reducing token use approximately fivefold. citeturn7academia38turn7search0

But the evidence does **not** support the stronger proposition that retrieval “solves” large-schema analytics. Spider 2.0 combines thousand-column schemas with documentation, metadata and code-navigation requirements and remains very difficult end to end. BEAVER explicitly identifies table retrieval, column mapping and join-key detection as separate enterprise subtasks. Spider 2.0-AIFunc reports schema grounding among leading error categories and found that traditional schema-retrieval/table-selection agents did not produce large gains over a simpler agent in that AI-native setting. citeturn1search4turn3search6turn1search23

This also explains an apparent contradiction in the literature: **retrieval can help when the problem is context dilution, while failing when the remaining bottleneck is semantic selection, join logic, dialect behavior or workflow reasoning.** “Retrieved the right table” and “constructed the right analysis” are not equivalent events.

### Business semantics and semantic layers

The strongest evidence that structured semantic metadata helps is promising but notably less mature than the text-to-SQL literature.

A reproducible 2026 Cube community/provider benchmark compares raw DDL with the same DDL plus a small hand-authored semantic guide covering default fact tables, snapshot semantics, calculated metrics, join paths, geography conventions and time anchoring. Across 100 hand-authored Contoso retail questions and three frontier models, the reported gain was **17–23 percentage points in analytical accuracy**, with paired exact McNemar **p < .002 for every model**; the semantic configurations converged near 68%, versus roughly 46–51% without the semantic guide. This is unusually useful because it holds the model roughly constant while manipulating semantic context. But it is a small, author-created benchmark, not a peer-reviewed multi-company study, and 68% remains far from a reliability result one would call “solved.” citeturn17search6

dbt’s separately implemented provider benchmark compares raw text-to-SQL, minimal semantic-layer access, modeled semantic-layer access and text-to-SQL over modeled data. Its 2026 experiment uses an ACME Insurance model, only 11 questions but repeats each 20 times. For questions supported by the layer, several model/effort configurations reached 95–100% semantic-layer accuracy while raw SQL generation was often around 50–62.5%. Again, this is useful controlled evidence of an effect, but it is provider-authored, tiny in question count and specifically constructed around its semantic-layer technology. It should not be treated like an independent cross-industry clinical trial for semantic layers. citeturn17search11turn17search17

Thus the defensible statement is:

**Structured metric definitions, modeled entities, join relationships and business conventions can materially improve correctness in controlled tests. Evidence that semantic layers eliminate hallucinations or guarantee analytical correctness is absent.**

That distinction is strengthened by BEAVER: genuine enterprise tasks require domain knowledge beyond raw schema shape, and off-the-shelf prompt/RAG approaches still struggle. citeturn3search0turn3search6

### Join, grain and aggregation reliability

Ordinary text-to-SQL benchmarks unquestionably contain joins and aggregations. BEAVER goes further by separately evaluating multi-table retrieval and join-key identification, while AIGT2S deliberately pushes structural query complexity. citeturn3search6turn2search13

What remains poorly isolated are the warehouse-specific failure modes that experienced analytics engineers worry about most: **fact-to-fact fanout, many-to-many bridge expansion, duplicated additive measures, mixed event/snapshot grains, slowly changing dimensions, late-arriving facts and apparently valid aggregations over the wrong business grain**. The reviewed major benchmarks do not report dedicated fanout/grain stress-test scores comparable to their overall execution metrics.

This is a major construct gap because execution accuracy is particularly vulnerable here: a query can execute perfectly, look sophisticated, and return plausible numbers while being analytically wrong. A test database may even fail to expose an incorrect join if the particular data state lacks duplicates that would reveal the fanout. Test-suite evaluation improves semantic equivalence testing of SQL but cannot supply missing organizational grain semantics. citeturn5search0turn2search30

The semantic-layer experiment is suggestive precisely because its metadata includes snapshot-table caveats and default fact-table rules; the large gain indicates that these conventions matter. But that remains evidence from a small benchmark rather than a general reliability estimate for fanout prevention. citeturn17search6

### Ambiguity and clarification

This is one of the clearest places where benchmark design is misaligned with production analytics.

PRACTIQ deliberately constructs conversations around ambiguous and unanswerable database questions and asks systems to classify the situation, request clarification and then generate SQL. Its authors report that state-of-the-art LLM baselines struggle with these cases. citeturn13academia39

AmbiSQL separately implements interactive ambiguity detection and multiple-choice clarification. On its ambiguous-query dataset, it reports **87.2% precision for ambiguity detection** and a **50% improvement in SQL exact-match accuracy** when its clarification process is integrated into text-to-SQL systems. This is strong evidence for the *value of clarification under deliberately ambiguous conditions*, although it is not yet evidence that a production agent can reliably identify all economically important ambiguities in arbitrary company terminology. citeturn13academia40

Conventional one-shot benchmarks create the wrong incentive for this construct: when every test item is assumed to have one answer, asking “Do you mean recognized or booked revenue?” can be treated as failure to answer rather than correct analytical behavior. A production-grade evaluation should sometimes reward **abstention, clarification or returning multiple interpretations**, rather than forcing a guess.

## Numerical, tabular, chart, narrative, causal, anomaly and forecasting evidence

### Numerical and table reasoning

Table and financial QA benchmarks demonstrate that retrieving the right values and performing the correct arithmetic is its own problem even after SQL generation is removed.

TAT-QA combines tables and text from financial reports and includes addition, subtraction, multiplication, division, counting, comparison/sorting and composed operations. Its original TAGOP model achieved **58.0 F1 versus 90.8 F1 for expert humans**, illustrating the historical size of the gap on hybrid financial reasoning. citeturn9search0turn9search4

FinQA similarly pairs financial-report questions with explicit reasoning programs, making it possible to separate the reasoning/computation chain from mere answer generation. MultiHiertt adds multiple hierarchical tables, longer surrounding text, supporting-fact selection and multi-step symbolic reasoning, and its baselines remained substantially behind human performance. citeturn9search1turn9search5turn9search2

For analytical-system design, however, the most important finding is not a particular benchmark score but the comparison between **language-generation arithmetic and executable computation**.

Program of Thoughts asks an LLM to produce executable programs while an external Python interpreter performs the numerical computation. Across mathematical and financial QA datasets including FinQA, TAT-QA and ConvFinQA, the authors report an average improvement of roughly 12 percentage points over chain-of-thought prompting. PAL similarly delegates execution to a Python runtime and, in one headline comparison, Codex+PAL exceeded a much larger PaLM-540B chain-of-thought system on GSM8K by 15 percentage points. Toolformer independently demonstrates the broader principle that models can learn to call calculators and other tools for operations where pure language modeling is weak. citeturn11search0turn11search11turn11search1turn11academia27

The empirical takeaway is strong:

**For arithmetic, aggregation and repeatable numerical transformation, external code, a calculator, SQL or another deterministic execution engine is better evidence than asking the model to “reason out” the number in prose.**

But a subtle boundary remains: external execution makes `123,456 × 0.17` deterministic; it does not decide whether 17% is the correct margin definition or whether 123,456 is the right revenue denominator.

### Chart reasoning and analytical narrative

ChartQA contains 9.6k human-written questions plus 23.1k questions generated from human-written chart summaries and tests both visual extraction and logical reasoning. It became sufficiently saturated that ChartQAPro was explicitly created to increase diversity and realism. ChartQAPro contains 1,341 charts from 157 sources and 1,948 questions including conversational, hypothetical and unanswerable cases. Across 21 evaluated models, the authors report a striking drop for Claude Sonnet 3.5 from **90.5% on ChartQA to 55.81% on ChartQAPro**. citeturn19search6turn19search10turn19search26

That is another example of benchmark validity dominating leaderboard interpretation. High chart-QA accuracy on a saturated benchmark does not imply robust understanding of dashboards, unusual chart forms, ambiguous labels or unanswerable executive questions.

Explanation itself is also becoming a separately recognized construct. ChartQA-X provides 30,799 chart samples with questions, answers and explanations, specifically because generating a correct answer and generating an adequate explanation are not the same problem. citeturn19search34

A further caution comes from work on model reasoning traces: reasoning models do not always faithfully verbalize which prompt information drove their answer. In experiments over multiple reasoning hints, reported chain-of-thought disclosure of used hints was often below 20%. That is not a chart-analytics study, but it is direct evidence against equating a fluent generated explanation with a faithful causal account of the underlying computation. citeturn16search10

Accordingly, analytical narrative generation should be decomposed into at least four checks:

**observation correctness** — did it describe what the numbers actually show?

**statistical correctness** — did it represent uncertainty, sampling and significance appropriately?

**causal restraint** — did it avoid converting association into causal explanation?

**coverage** — did it ignore materially plausible alternative explanations?

Existing chart/table benchmarks cover the first of these far better than the latter three. citeturn19search26turn19search34

### Anomaly detection versus anomaly explanation

The evidence reviewed does not justify collapsing these into one task.

Anomaly **detection** can often be specified as a statistical problem: identify deviations relative to a baseline, forecast distribution, seasonality model or peer group. A language model may orchestrate that method, but success in text-to-SQL, table QA or narrative generation is not evidence that the language model itself is a competitive anomaly detector.

Anomaly **explanation** is much harder. “Sales are 18% below forecast” is descriptive. “Sales fell because the price increase reduced conversion” is causal. The second claim requires evidence about competing explanations, confounding and often an explicit experimental or quasi-experimental identification strategy. The causal-reasoning evidence below does not establish that present LLMs can autonomously do this from observational business telemetry. citeturn20academia19turn20academia21

The legitimate current role is therefore closer to **hypothesis generation**—for example, proposing region, product, funnel stage, pricing, availability or acquisition mix as follow-up dimensions—than certifying why the anomaly occurred.

### Causal reasoning and “why” questions

The literature makes the distinction between causal vocabulary and causal inference particularly important.

Corr2Cause-style tasks give models correlational/conditional-independence information and ask them to infer causal conclusions. A 2025 analysis reports **GPT-4 F1 of 29.08 versus a random-uniform baseline of 20.38** on Corr2Cause, characterizing modern LLMs as still unreliable at distinguishing causal conclusions from correlation structure. Structured intermediate reasoning improves performance—Qwen3-32B in that study increased from 32.71 to 48.26 F1—but remains far from evidence of general causal competence. citeturn20academia21

Another study improves five LLMs by decomposing the same problem into explicit steps corresponding to the PC causal-discovery algorithm. That finding is important in two directions: it shows that LLMs can usefully execute parts of a structured causal procedure, but it also indicates that performance comes from **imposing an algorithmic causal scaffold**, not from trusting unconstrained narrative intuition. citeturn20academia19

Nothing in these results validates the leap from:

> “Enterprise customers with characteristic X have higher churn”

to:

> “X caused higher churn.”

Nor does ordinary SQL or table QA identify interventions, counterfactuals, selection effects, reverse causality or unobserved confounding.

For observational company data, current evidence supports using LLMs to **generate candidate hypotheses, explain a supplied causal model, help formulate an analysis and perhaps execute a specified statistical procedure**. It does **not** establish autonomous causal interpretation of arbitrary “Why did KPI X change?” questions.

### Forecasting

Forecasting provides one of the clearest examples of why language-model capability must not be inferred from language fluency.

The NeurIPS 2024 Spotlight paper *Are Language Models Actually Useful for Time Series Forecasting?* ablated the LLM components of three popular LLM-based time-series forecasting approaches. Removing the LLM, replacing it with a basic attention layer or using models trained from scratch generally did **not** reduce forecasting accuracy and often improved it. The authors conclude that pretrained LLMs in those architectures did not meaningfully model the relevant sequential dependencies or provide few-shot advantages commensurate with their computational cost. citeturn20search1turn20search9

A separate evaluation of LLMTIME against standard statistical models found that zero-shot language-model forecasting can work on some datasets but deteriorates on more diverse signals and on combinations of trend, periodicity and complex frequency structure. citeturn20academia20

This does not imply that every language-assisted forecasting workflow is useless. It does imply that the defensible comparison is:

**naive baseline → statistical forecasting → specialized ML/time-series model → human/domain forecast → LLM-assisted model selection/interpretation**

rather than “the LLM sounds analytically sophisticated, therefore it can forecast.”

Using an LLM to select, run and explain a dedicated forecasting model is a different construct from asking the LLM itself to predict the series. The former can be useful; the reviewed evidence does not establish a general advantage for the latter.

## Multi-step systems, correction, calibration, provenance, robustness and security

### Multi-step and agentic analytics

Agent frameworks clearly improve some bounded SQL tasks. CHESS uses separate retrieval, schema-selection, candidate-generation and unit-testing stages and reports strong BIRD performance with markedly fewer calls than other high-compute agent configurations. citeturn7search0turn7search15

But Spider 2.0 is a more revealing end-to-end test. Its tasks can require locating metadata, understanding project structure, producing multiple queries and transformations, and operating cloud warehouse dialects; the reported o1-preview code-agent success was only 17.0%. This is much closer to an analytical workflow than Spider 1.0, and it demonstrates substantial error accumulation across stages. citeturn1search4

SQL repair is not an automatic escape hatch. BIRD-CRITIC/SWE-SQL, built from realistic SQL issues replayed in executable environments, reports only **38.87% success for o3-mini on its PostgreSQL tasks and 33.33% on multi-dialect tasks**. citeturn15search8

SQL-IQ reinforces that point with current models: its best reported SQL-correction result is 41.3% for Gemini-3.0-flash, while some models with considerably stronger text-to-SQL scores have correction scores in the 20s or low 30s. An agent that can generate SQL is therefore not necessarily an agent that can correctly diagnose its own SQL. citeturn15search6

Spider 2.0-AIFunc provides another caution against “more agents = more reliability”: its authors found that traditional schema-retrieval/table-selection agent machinery offered limited benefit for the AI-native SQL tasks, with a relatively minimal architecture matching or outperforming more elaborate alternatives in several comparisons. citeturn1search23

The evidence supports **structured decomposition plus executable verification**. It does not support the stronger claim that autonomous loops reliably recover from semantic error.

For the full workflow requested in the research question—

**question → query → inspect → hypothesize → query again → synthesize**

—the public evidence is notably weaker than for single-query generation. Most high-quality benchmarks still have a pre-specified terminal target. Few measure whether an agent chose a useful *next analytical question*, whether its intermediate hypothesis was justified, whether it explored alternative explanations, and whether the final synthesis remains faithful to every intermediate result.

Cost and variance are also underreported relative to accuracy. CHESS explicitly studies call efficiency, which is valuable, but many benchmark comparisons permit different numbers of model calls, retries, samples or self-correction rounds. A system receiving five candidate generations plus execution feedback is not a like-for-like baseline against one-shot SQL generation. citeturn7search0

### Calibration, abstention and knowing when to ask

The best task-specific evidence currently comes from ambiguity benchmarks rather than ordinary leaderboards. PRACTIQ shows that models struggle when the correct action is to recognize an ambiguous or unanswerable question; AmbiSQL shows that explicitly detecting ambiguity and interacting with the user can substantially improve downstream SQL accuracy. citeturn13academia39turn13academia40

Conventional exact-match/execution leaderboards generally do not evaluate **selective risk**: accuracy conditional on the system choosing to answer, coverage at a given error tolerance, calibration error, or the quality of abstention decisions. That means a model with 80% ordinary accuracy and no knowledge of its failures could be less operationally useful than a 70% model that reliably flags its difficult 30%.

I found no strong enterprise-analytics evidence establishing that an LLM’s free-form statement such as “I am 95% confident” is sufficiently calibrated to serve as an analytical control. Confidence should therefore not be treated as validation unless empirically calibrated against held-out tasks from the actual deployment distribution.

### Data-quality robustness

The strongest direct robustness result remains Dr.Spider. Across 17 realistic perturbation types, the most robust evaluated system still suffered a **14.0% aggregate performance decline**, and the hardest perturbation caused a **50.7% decline**. citeturn5search1turn5search17

KaggleDBQA improves realism with naturally occurring web databases, domain-specific data types, original formatting and database documentation, which addresses part of the gap between clean benchmark schemas and real data. citeturn5search6

BEAVER further introduces private enterprise schemas and genuine domain knowledge. Yet the standard benchmark regime remains overwhelmingly **static**: one schema snapshot, one database state, one metadata state. citeturn3search0turn3search6

Real organizational environments also contain missing observations, broken pipelines, stale dimensions, undocumented schema changes, misleading column names, contradictory definitions and metadata that is simply wrong. No major benchmark reviewed here provides a comprehensive longitudinal stress test of all these conditions.

There is analogous evidence from dynamic RAG evaluation that factual performance decreases when facts are more dynamic, obscure or complex—CRAG reports only 63% hallucination-free answers for state-of-the-art industry RAG systems in its setup—but that is an adjacent warning rather than direct structured-company-data evidence. citeturn21search33

### Provenance, generated SQL and traceability

Showing the SQL is useful for **auditability**, but auditability is not reliability.

The evaluation literature makes that distinction concrete. A query can look different yet be equivalent, which motivated Test Suite Accuracy; a query can execute to the right result accidentally, which motivates richer semantic evaluation such as FLEX. citeturn5search0turn2search30

Generated SQL therefore provides a trace that a skilled reviewer can inspect, reproduce and test. What I did **not** find is a controlled study establishing that simply exposing SQL to users materially increases detection of subtle business-semantic, grain or join errors in realistic analytics workflows.

That question matters because SQL visibility is asymmetric:

A senior analytics engineer may recognize a missing effective-date predicate, wrong join or fanout.

A business user who asked natural language specifically because they do not know SQL may gain almost no error-detection power from seeing a 40-line query.

And even an expert cannot detect an incorrect metric definition from SQL alone if the organizational definition is absent or wrong.

Thus **query provenance is a necessary feature for serious review, not evidence that the answer is correct.**

The same caveat applies to free-form reasoning explanations. Recent work on chain-of-thought faithfulness demonstrates that generated reasoning need not faithfully reveal all factors that actually determined an answer. citeturn16search10

### Security

Security vulnerabilities are not merely theoretical for tool-using and retrieval-augmented LLM systems. The OWASP GenAI security work explicitly covers prompt injection, sensitive-information disclosure, excessive agency and related tool/application risks, based on demonstrated failure modes across deployed and experimental LLM applications. citeturn18search24

The relevant analytics threat model includes data values, comments, catalog descriptions, documentation or retrieved metadata containing adversarial instructions that compete with the system prompt; agents with query/export/write tools can then convert a language-model failure into unauthorized tool use or data disclosure. These are instances of the broader demonstrated **indirect prompt-injection** problem.

Two important research artifacts in this area are Greshake et al., *Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection* (`https://arxiv.org/abs/2302.12173`) and AgentDojo, a benchmark environment for prompt-injection attacks and defenses in tool-using agents (`https://arxiv.org/abs/2406.13352`). The broader vulnerability category is recognized in the current OWASP GenAI taxonomy. citeturn18search24

What is **not** established is the incident rate for real enterprise analytics deployments or the effectiveness of any single defense under every warehouse/tool configuration. The evidence supports treating retrieved company data and metadata as potentially untrusted input, not assuming that “read-only analytics” makes prompt injection irrelevant.

## Human collaboration and claim audit

### Human–AI collaboration

The direct evidence base for **professional analysts working with AI over real enterprise warehouses** is surprisingly thin relative to the volume of text-to-SQL benchmark work.

The broader 2026 ILO review of empirical GenAI-at-work studies concludes that productivity benefits are real but uneven and often incompletely verified, drawing on experiments, firm data, platform studies and worker surveys. That broader literature is useful context, but improvements in writing, coding, customer support or consulting cannot simply be re-labeled evidence that financial or product analytics becomes more correct. citeturn16search17

For analytics specifically, the benchmark literature establishes that AI can reduce the mechanical barrier to forming SQL, particularly with good schema/context support. What is insufficiently established is whether this increases **decision-quality-adjusted productivity** after counting wrong analyses, review time, debugging, downstream decisions and failures that users do not notice.

Expertise is likely to produce two opposing effects that must be measured rather than assumed. A novice receives more capability uplift from natural-language access but has less ability to detect subtle mistakes. An expert can inspect SQL, distributions and metric definitions better but may save less time on tasks already easy for them. Current text-to-SQL leaderboards do not answer which effect dominates in field use.

Similarly, “human in the loop” is not itself an empirical safety result. Human review works only when the reviewer has sufficient expertise, sees enough provenance, allocates enough attention, is not over-relying on the AI, and has a practical way to recompute or challenge the answer. The reviewed enterprise analytics literature does not establish those conditions by default.

### Claim audit

| Common industry claim | Assessment | What the evidence legitimately says |
|---|---|---|
| **“Modern LLMs can replace routine analysts.”** | **Unsupported** | There is strong component-level capability but no comparable evidence of analyst-level end-to-end correctness across business semantics, data selection, querying, iterative investigation and interpretation. Spider 2.0’s 17% enterprise-workflow result is particularly inconsistent with broad replacement claims. citeturn1search4 |
| **“Natural-language analytics democratizes data.”** | **Plausible but inadequately tested** | It clearly lowers the SQL-expression barrier. Whether it democratizes *correct* analysis without increasing undetected errors among non-experts requires human field studies not provided by SQL benchmarks. citeturn0search9turn16search17 |
| **“Semantic layers solve AI hallucinations over company data.”** | **Contradicted as an absolute claim; conditionally supported as an improvement claim** | Small controlled provider/reproducible experiments report large improvements, but one still levels around 68% accuracy and independent multi-company validation is lacking. citeturn17search6turn17search17 |
| **“AI can reliably generate production SQL.”** | **Conditionally supported only in bounded settings** | High benchmark scores are real for curated tasks, but enterprise workflows, SQL debugging and private-schema tasks remain substantially harder. citeturn1search4turn15search8turn3search0 |
| **“Showing generated SQL makes the system trustworthy.”** | **Unsupported** | SQL increases traceability; no strong reviewed experiment shows that display alone causes users to catch business-semantic/grain errors. Evaluation studies show that even automated SQL correctness judgments are nontrivial. citeturn5search0turn2search30 |
| **“AI can explain why metrics changed.”** | **Unsupported for causal claims; conditionally supported for descriptive hypotheses** | Models can describe patterns and generate hypotheses. Causal benchmarks remain difficult even when the causal premises are explicitly provided. citeturn20academia19turn20academia21 |
| **“AI agents can autonomously perform multi-step analysis.”** | **Mixed / conditionally supported** | Agents solve bounded multi-step SQL workflows, but realistic end-to-end success is low and open-ended hypothesis-selection/synthesis is much less validated. citeturn7search0turn1search4 |
| **“AI-generated forecasts are useful.”** | **Mixed** | LLM-based forecasters sometimes work, but strong ablations show the LLM component frequently adds no forecasting value. Tool-assisted interpretation of a proper forecasting model is a different claim. citeturn20search1turn20academia20 |
| **“Larger models eliminate text-to-SQL problems.”** | **Contradicted** | More capable models improve results, but Spider 2.0, BIRD-CRITIC, BEAVER and current SQL-IQ retain large failure rates. citeturn1search4turn15search8turn15search6turn3search0 |
| **“Retrieving schema documentation solves grounding.”** | **Contradicted as an absolute claim; conditionally supported as one intervention** | Schema selection/retrieval can improve accuracy and efficiency, yet enterprise grounding remains difficult and Spider 2.0-AIFunc found conventional retrieval machinery offered limited benefit. citeturn7search0turn1search23 |
| **“Human review is sufficient to make AI analytics safe.”** | **Plausible but inadequately tested, and unsafe as a blanket assumption** | Review can catch errors, but analytics-specific evidence on detection rates, expertise effects and automation bias is insufficient; SQL visibility does not guarantee comprehensibility. citeturn16search17turn2search30 |

The pattern is striking: **the strongest supported industry claim is not “AI replaces analysis,” but “AI can lower the cost of several analytical sub-tasks when surrounded by explicit context, deterministic execution and verification.”**

## Study table and contradictory evidence

### Core empirical studies and benchmarks

| Study / status / date | System, task, dataset and scale | Baseline | Main outcome | Limitations and professional transferability | DOI / direct URL |
|---|---|---|---|---|---|
| **Spider — EMNLP 2018** citeturn0search9 | Cross-domain text-to-SQL; 10,181 questions, 5,693 unique SQL, 200 DBs, 138 domains; original neural parsers | Contemporary semantic parsers; human annotation | Original best model only 9.7% exact match; dataset subsequently became standard benchmark | Foundational but clean/static and intent-defined; **low–medium transfer** to modern enterprise analytics | https://doi.org/10.18653/v1/D18-1425 |
| **BIRD — NeurIPS-era benchmark, 2023** citeturn1search2 | Large DB-grounded text-to-SQL; 12,751 pairs, 95 DBs, 33.4 GB, >37 domains | Modern text-to-SQL/LLM systems | Much harder, content- and knowledge-grounded evaluation than Spider | Better scale, but curated gold intents and static data; **medium transfer** | https://bird-bench.github.io/ |
| **Test Suite Accuracy — EMNLP 2020** citeturn5search0turn5search4 | SQL semantic evaluation; 21 Spider models; 100 manually verified examples in metric study | Exact match / single execution | Old metric average 2.5% false negatives; worst 8.1% | Improves SQL-equivalence validity; does not test business correctness | https://doi.org/10.18653/v1/2020.emnlp-main.29 |
| **KaggleDBQA — ACL 2021** citeturn5search6 | Realistic text-to-SQL using naturally occurring web DBs, formatting, types and documentation | Existing parsers | Demonstrated substantial realism gap from standard academic datasets | Still static/public databases; **medium transfer** for messy-schema robustness | https://aclanthology.org/2021.acl-long.176/ |
| **Dr.Spider — ICLR 2023** citeturn5search1turn5search17 | Robustness; 17 perturbation families on Spider components | Leading text-to-SQL models | Best model dropped 14.0% overall; up to 50.7% on hardest perturbation | Perturbations are controlled, not longitudinal warehouse drift; strong robustness warning | https://arxiv.org/abs/2301.08881 |
| **Spider 2.0 — ICLR 2025 oral / 2024 preprint** citeturn1search4turn1search1 | Enterprise SQL workflow; 632 tasks, BigQuery/Snowflake, schemas often >1,000 columns; o1-preview code agent | Same broad model generation on Spider/BIRD; agent workflows | 17.0% reported versus 91.2% Spider 1.0 and 73.0% BIRD | One of the highest-transfer public evaluations; still curated and static rather than live organization | https://arxiv.org/abs/2411.07763 |
| **BEAVER — preprint/benchmark, 2024–26** citeturn3search0turn3search6 | Private enterprise warehouses/query logs; current site 9,128 queries, 812 tables, 19 domains | Off-the-shelf LLMs, prompt engineering, RAG | Systems struggle substantially; benchmark decomposes table, join, column and domain-knowledge errors | Proprietary portions reduce reproducibility; **high conceptual transfer** | https://arxiv.org/abs/2409.02038 ; https://beaverbench.github.io/ |
| **FLEX — NAACL 2025** citeturn2search30 | Evaluation metric for text-to-SQL | Standard execution accuracy; human experts | Human agreement κ improved from 0.62 to 0.8704; rankings changed | LLM evaluator is itself fallible; evaluates intended SQL semantics, not business intent | https://doi.org/10.18653/v1/2025.naacl-long.228 |
| **AIGT2S — Findings IJCNLP 2025** citeturn2search13 | 18k pairs, 113 DBs, 55% ultra-complex; GPT-4o, Llama3, RESDSQL, MAC-SQL | Multiple neural/LLM systems | Best baseline maxed at ~56% execution; reported 87.5% annotator agreement | Synthetic/LLM-assisted question construction reduces external validity | https://doi.org/10.18653/v1/2025.findings-ijcnlp.143 |
| **CHESS — preprint / MAS@ICML 2025** citeturn7search0turn7search15 | Multi-agent BIRD SQL: IR → schema selection → generation → unit test | Other BIRD agents / less selective context | Reported 71.10% BIRD test at high-compute configuration; schema selection ~+2 points and ~5× token reduction | Strong workflow evidence but public benchmark; weaker transfer than Spider 2.0/BEAVER | https://arxiv.org/abs/2405.16755 |
| **BIRD-CRITIC / SWE-SQL — NeurIPS 2025** citeturn15search8turn15search1 | SQL debugging; 530 PostgreSQL and 570 multidialect realistic issues in paper; o3-mini and other models | Direct reasoning models and SQL repair systems | o3-mini: 38.87% PG, 33.33% multidialect | Strong evidence correction is not solved; realistic debugging has **high transfer** | https://arxiv.org/abs/2506.18951 |
| **PRACTIQ — preprint 2024** citeturn13academia39 | Conversational SQL with ambiguous and unanswerable questions; LLM category classification + clarification | LLM baselines | SOTA systems struggle with correct handling of ambiguous/unanswerable intent | Synthetic conversations inspired by real questions; crucial construct-validity evidence | https://arxiv.org/abs/2410.11076 |
| **AmbiSQL — preprint/demo 2025** citeturn13academia40 | Interactive ambiguity detection + clarification | Text-to-SQL without interactive clarification | 87.2% ambiguity-detection precision; SQL exact match +50% after clarification | Dataset-specific; still important causal intervention on ambiguity | https://arxiv.org/abs/2508.15276 |
| **Spider 2.0-AIFunc — preprint 2026** citeturn1search23 | 465 tasks, 125 DBs, 6 Snowflake AI functions; frontier and open models | Simple vs retrieval-heavy agents | Top proprietary systems ~67–70% execution; best OSS 58.1%; retrieval agents not consistently superior | AI-native but intentionally clarified NL; does not test full business ambiguity | https://arxiv.org/abs/2607.06229 |
| **SQL-IQ — benchmark artifact, current 2026** citeturn15search6 | 17 models, 7 SQL capabilities | Same model families across generation/judging/debugging/translation | Current best macro score 59.5; best correction 41.3, demonstrating capability fragmentation | Benchmark project rather than mature enterprise field study | https://sql-iq.github.io/SQL-IQ/ |

### Numerical, chart, semantic and forecasting studies

| Study / status / date | System, task, dataset and scale | Baseline | Main outcome | Limitations and professional transferability | DOI / direct URL |
|---|---|---|---|---|---|
| **TAT-QA — ACL 2021** citeturn9search0turn9search4 | Financial table+text QA with arithmetic; TAGOP | Neural QA; expert humans | TAGOP 58.0 F1 vs expert-human 90.8 F1 | Real financial reports, but not internal warehouse semantics | https://aclanthology.org/2021.acl-long.254/ |
| **FinQA — EMNLP 2021** citeturn9search1turn9search5 | Financial QA with gold reasoning programs | Neural reasoning systems / humans | Large machine–expert gap in original evaluation | Strong numerical-program construct; semantics supplied by documents | https://aclanthology.org/2021.emnlp-main.300/ |
| **MultiHiertt — ACL 2022** citeturn9search2 | Multi-hierarchical-table + long-text multi-step reasoning | Contemporary QA systems / humans | Baselines substantially behind experts | Good multi-table reasoning evidence; not SQL/schema discovery | https://arxiv.org/abs/2206.01347 |
| **Program of Thoughts — TMLR 2023** citeturn11search0turn11search11 | LLM generates executable programs; evaluated on math + FinQA/ConvFinQA/TAT-QA | Chain-of-thought using same/similar LM | ~12-point average gain over CoT across evaluated datasets | Strong evidence for execution tools; program selection can still be wrong | https://arxiv.org/abs/2211.12588 |
| **ChartQA — Findings ACL 2022** citeturn19search6turn19search10 | 9.6k human + 23.1k generated chart questions | Chart/VL QA systems | Established visual+logical chart QA benchmark | Now partially saturated; limited dashboard diversity | https://doi.org/10.18653/v1/2022.findings-acl.177 |
| **ChartQAPro — preprint 2025** citeturn19search26 | 1,341 charts, 157 sources, 1,948 diverse questions; 21 models | Older ChartQA | Claude Sonnet 3.5: 90.5% ChartQA → 55.81% ChartQAPro | Strong evidence of benchmark saturation/realism gap | https://arxiv.org/abs/2504.05506 |
| **Cube semantic-layer benchmark — provider/community artifact, 2026** citeturn17search6 | 100 Contoso questions; Opus 4.7, Sonnet 4.6, GPT-5.4; raw DDL vs semantic guide | Paired within-model raw-schema baseline | +17–23 pp analytical accuracy; exact McNemar p<.002; semantic runs ~68% | Reproducible but author-created, one dataset, non-peer-reviewed; **promising not definitive** | https://github.com/cubedevinc/semantic-layer-benchmark |
| **dbt semantic-layer benchmark — provider artifact, 2026** citeturn17search11turn17search17 | ACME Insurance, 11 questions × 20 runs, several frontier models | Text-to-SQL vs minimal/modeled semantic layer | Some supported-question configurations 95–100% SL vs ~50–62.5% raw SQL | Very small N and vendor-authored; useful intervention evidence, weak broad external validity | https://github.com/dbt-labs/dbt-llm-sl-bench |
| **Are Language Models Actually Useful for Time Series Forecasting? — NeurIPS 2024 Spotlight** citeturn20search1turn20search9 | Ablations of three LLM-based forecasting approaches | Remove LLM, random attention, train from scratch | LLM removal usually did not hurt and frequently improved results | Directly undermines claims that language pretraining itself drives forecasting performance | https://arxiv.org/abs/2406.16964 |
| **Prompting Strategies for Causation from Correlation — preprint 2024** citeturn20academia19 | Five LLMs on Corr2Cause using PC-algorithm subquestions | Ordinary prompting | Structured causal decomposition improves all tested models | Formal toy/benchmark premises are much easier than messy observational company data | https://arxiv.org/abs/2412.13952 |
| **Structured Thinking Matters — preprint 2025** citeturn20academia21 | GPT-4 benchmark result plus Qwen3-32B structured approach on Corr2Cause | Random uniform; direct prompting | GPT-4 F1 29.08 vs random 20.38; Qwen3 32.71 → 48.26 with structure | Improved causal reasoning remains far from autonomous business causal inference | https://arxiv.org/abs/2505.18034 |

### Contradictory evidence is mostly construct shift, not true contradiction

| Apparent contradiction | What explains it |
|---|---|
| **Spider appears nearly solved; Spider 2.0 is extremely difficult.** | Spider measures cross-domain NL→SQL under bounded benchmark conditions; Spider 2.0 adds very large schemas, warehouse dialects, metadata discovery, project navigation and multi-query workflows. It is a harder construct. citeturn0search9turn1search4 |
| **Schema retrieval helps in CHESS/RESDSQL; retrieval does little in Spider 2.0-AIFunc.** | Retrieval helps when irrelevant context is the binding constraint. Once correct context is available, predicate semantics, function parameters and workflow reasoning can dominate. citeturn7search0turn7academia38turn1search23 |
| **Semantic metadata produces very large gains; enterprise benchmarks remain poor.** | Metadata removes some semantic ambiguity but does not solve every source of error—schema scale, query decomposition, joins, dirty data and workflow planning remain. Also, semantic-layer experiments are smaller and more controlled. citeturn17search6turn3search0 |
| **Agents outperform simple prompting on BIRD; more elaborate agents are not always better.** | Performance depends on whether added stages actually address the current bottleneck. More calls can add recovery opportunities but also error, latency and context complexity. citeturn7search0turn1search23 |
| **Older chart benchmarks suggest very strong reasoning; ChartQAPro shows large deficits.** | Dataset saturation and limited visual/question diversity made earlier scores optimistic for real charts. citeturn19search26 |
| **LLM-based forecasting papers report competitive forecasts; ablations say the LLM is unnecessary.** | Forecast quality can come from patching, architecture, preprocessing or attention rather than linguistic pretraining. The proper baseline is the same forecasting pipeline without the LLM. citeturn20search1 |
| **Models appear able to discuss causality fluently; formal causal benchmarks remain difficult.** | Natural-language causal vocabulary and algorithmically valid causal inference are different skills; performance rises when a formal algorithmic scaffold is supplied. citeturn20academia19turn20academia21 |
| **Self-correction sounds intuitively easier than first-pass generation; SQL debugging remains weak.** | Correcting a plausible semantic error requires diagnosing *why* it is wrong, not merely generating another plausible query. BIRD-CRITIC and SQL-IQ isolate this difference. citeturn15search8turn15search6 |

The recurring lesson is methodological: disagreement between papers should first be attributed to **different benchmark distributions, prompts, context access, numbers of attempts, execution feedback, agent decomposition and scoring metrics** before being interpreted as evidence that one paper is “right.”

## What the evidence does not support, research gaps and the frontier

### What current evidence does not support

As of October 3, 2026, the literature reviewed here does **not** legitimately establish any of the following broad propositions.

It does not establish a stable percentage called **“AI analyst accuracy.”** The constructs are too heterogeneous, and systems that are good at generation can be poor at correction, ambiguity recognition, chart reasoning or causal inference. citeturn15search6turn13academia39turn19search26

It does not establish that high Spider or BIRD execution accuracy predicts reliability on an enterprise warehouse. Spider 2.0 and BEAVER are direct counter-evidence to that extrapolation. citeturn1search4turn3search0

It does not establish that an executable query is analytically correct. Execution evaluation can misclassify semantic correctness, and neither execution equivalence nor test-suite equivalence supplies missing business intent. citeturn5search0turn2search30

It does not establish that models autonomously know an organization’s definitions of revenue, churn, active user, margin, cohort or fiscal period. Enterprise/domain metadata helps, but the strongest semantic-layer evidence remains small-scale and largely provider/community produced. citeturn17search6turn17search17

It does not establish robustness to live schema evolution, stale metadata, incorrect documentation, permission changes or arbitrary dirty data. Dr.Spider shows meaningful degradation even under controlled perturbation, while mainstream benchmarks remain static. citeturn5search1

It does not establish reliable avoidance of fanout, wrong grain and many-to-many aggregation errors under realistic warehouse schemas. Existing benchmarks contain joins, but these warehouse pathologies are not sufficiently isolated and reported as first-class reliability constructs. citeturn3search6

It does not establish that models reliably know when a business question requires clarification. Purpose-built benchmarks show the opposite: ambiguity and unanswerability materially challenge state-of-the-art systems, while explicit clarification mechanisms improve results. citeturn13academia39turn13academia40

It does not establish that free-form LLM arithmetic is suitable as the computational substrate of analytics. Program-execution research instead strongly supports delegating exact computation to deterministic tools. citeturn11search0turn11search1

It does not establish autonomous causal explanations for business KPI changes. Even formal causal benchmarks with explicit premises remain difficult. citeturn20academia19turn20academia21

It does not establish a general advantage for LLMs as time-series forecasters. A strong ablation study finds that removing the LLM often leaves forecasting unchanged or better. citeturn20search1

It does not establish that a fluent narrative is faithful to the data or to the system’s actual reasoning. Chart explanations are now evaluated separately, and broader reasoning-trace research shows generated rationales need not faithfully disclose causal influences on model outputs. citeturn19search34turn16search10

It does not establish that human review automatically closes these gaps. The empirical workplace literature finds heterogeneous productivity effects, while analytics-specific studies of review effectiveness, expertise and automation bias remain insufficient. citeturn16search17

And it does not establish that exposing SQL converts an unreliable answer into a trustworthy one. It creates provenance; whether the reviewer understands and checks that provenance is another empirical question.

### The most important research gaps

| Gap | Experiment or field study that would meaningfully resolve it |
|---|---|
| **End-to-end business correctness** | Recruit multiple real organizations across SaaS, commerce, finance and operations. Sample genuine historical analyst tickets before the AI study. Provide live warehouse snapshots, real metric definitions and documentation. Compare AI, human analyst and human+AI. Blind senior-domain analysts should adjudicate *business correctness*, not SQL similarity. Score every stage: interpretation → tables → grain → SQL → result → narrative. |
| **Organizational ambiguity** | Create paired versions of real questions with deliberately omitted metric, time, currency and scope definitions. Randomize systems into forced-answer versus clarification/abstention policies. Primary outcome: confidently wrong answers per 100 requests, not raw answer rate. |
| **Join and grain errors** | Build a dedicated benchmark containing fact-to-fact joins, many-to-many bridges, mixed grains, snapshots, slowly changing dimensions and duplicate-sensitive aggregations. Generate multiple database instances so incorrect joins cannot pass merely because one snapshot lacks revealing duplicates. |
| **Independent semantic-layer efficacy** | Pre-register a multi-company crossover study with the same model under four conditions: raw DDL; DDL+descriptions; full governed semantic layer; semantic layer+verified queries. Evaluate unseen real analyst questions and adjudicate organizational correctness. This would separate metric-layer value from model-generation improvement. |
| **Schema complexity scaling law** | Hold questions constant while systematically adding irrelevant tables/columns, alternative join paths and semantically similar entities. Measure table recall, column grounding, join selection and end-to-end correctness separately as schema size increases. |
| **Human review and automation bias** | Seed realistic answers with subtle business-definition, grain, aggregation and causal errors. Randomize novice and expert analysts to AI answer only, answer+SQL, answer+SQL+result provenance, or deterministic BI baseline. Measure detection rate, review time, confidence and whether explanations increase or decrease overreliance. |
| **Longitudinal data-quality robustness** | Run systems against a warehouse for six to twelve months while deliberately introducing realistic schema migrations, stale docs, renamed fields, delayed pipelines, missing partitions and conflicting definitions. Measure degradation, detection and recovery rather than one-time benchmark accuracy. |
| **Agent reliability and reproducibility** | Re-run identical multi-step analytical cases many times across seeds and model versions. Log every query/hypothesis/tool call. Report end-to-end success, per-stage failure, recovery probability, variance, tokens, latency and cost. Compare against a single-query system and a deterministic workflow using the same tools. |
| **Narrative faithfulness** | Start from independently verified query results. Require executive narratives. Human/statistical adjudicators label every quantitative statement, causal phrase, significance claim, omitted material countertrend and proposed explanation. Score claim-level precision rather than prose quality. |
| **Causal “why” analytics** | Use business settings with known experimental or strong quasi-experimental ground truth. Compare direct LLM causal answers, LLM-assisted formal methods, trained causal analysts and ordinary descriptive analysis. Penalize causal language when identification assumptions are unmet. |
| **Forecasting utility** | Compare naive seasonal forecasts, classical statistics, specialized ML, direct LLM forecasting, LLM-selected forecasting models and human+AI across many business series. Use rolling-origin evaluation and decision-cost metrics, not selected example plots. |
| **Calibration and abstention** | Measure risk–coverage curves: at each confidence/abstention threshold, how many wrong business answers remain? Compare self-reported confidence with ensembles, execution checks, verifier models and calibrated discriminators on genuinely out-of-distribution enterprise tasks. |
| **Security under structured-data retrieval** | Plant adversarial instructions in table values, column comments, catalog descriptions and retrieved docs. Give agents realistic read/export tools and canary-sensitive data. Measure instruction-following, exfiltration and unauthorized tool actions under least-privilege and defense configurations. |
| **Chart generation** | Supply verified analytical results but ambiguous visualization requirements. Score not just syntax/rendering but correct data subset, aggregation, denominator, axis, units, uncertainty representation and whether a human decision-maker reaches the correct conclusion from the generated chart. |
| **Anomaly analysis** | Compare LLM-direct detection with dedicated statistical detectors, then separately evaluate explanation. Supply anomalies with known injected mechanisms so hypothesis ranking can be distinguished from mere plausible storytelling. |

The highest-leverage experiment is the first one. The field currently has many thousands of public question–SQL pairs and remarkably little high-quality evidence on **whether an AI-assisted business answer is actually correct according to the organization that owns the data**.

### Research frontier

The frontier can be divided into three evidence bands.

**Relatively well established:** LLMs can translate explicit natural-language questions into SQL surprisingly well on bounded schemas; schema selection, demonstrations, retrieval and execution feedback can improve this; structured execution is preferable to free-form arithmetic; and richer benchmarks expose large drops relative to older saturated ones. citeturn0search9turn7search0turn11search0turn1search4

**Promising but conditional:** enterprise-domain metadata, metric/semantic layers, clarification mechanisms, agent decomposition and automated repair can materially improve particular failure modes. The evidence is strongest when the intervention is tested against the same model under controlled conditions, but independent, multi-organization replication remains scarce. citeturn17search6turn13academia40turn7search0turn15search8

**Not established at decision-grade reliability:** autonomous understanding of latent organizational semantics; reliable grain/join choice in arbitrary warehouses; calibrated recognition of uncertainty; causal explanations of KPI changes; direct LLM forecasting superiority; faithful executive narratives; robust long-horizon multi-step analysis; and safe correctness under live drift, permissions and adversarial data. The newer enterprise, ambiguity, chart, causal and forecasting results all point toward these remaining gaps rather than closing them. citeturn1search4turn3search0turn13academia39turn19search26turn20academia21turn20search1

The key benchmark-validity judgment is therefore:

> **Success on Spider, BIRD, ChartQA or another component benchmark justifies trust in the component and distribution actually tested. It does not justify trust in a production “AI analyst.”**

Spider 2.0 is especially informative because it shows what happens when apparently modest pieces of production reality—very large schemas, cloud SQL dialects, metadata search, long transformations and multi-step workflows—are put back into the task: performance falls dramatically despite impressive results on the predecessor benchmarks. citeturn1search4

BEAVER adds the missing insight that real enterprise querying contains private domain knowledge and nontrivial table, column and join grounding that public schemas do not reveal. PRACTIQ adds the observation that real requests can be ambiguous or unanswerable. Semantic-layer experiments indicate that explicit organizational knowledge can recover a substantial amount of accuracy, but not all of it. Program-of-Thoughts/PAL show that deterministic tools can recover substantial numerical reliability. Causal and forecasting research, meanwhile, warns against extending gains in language/querying into fundamentally different statistical constructs. citeturn3search0turn13academia39turn17search6turn11search0turn20academia21turn20search1

The empirical frontier is consequently moving away from **“Can an LLM emit valid SQL?”** toward the substantially harder question:

> **Can a system reliably recognize what a business question means, know when that meaning is underspecified, ground it in the correct live data and grain, perform verified computation, distinguish observation from explanation, and communicate only claims justified by the evidence?**

As of the October 3, 2026 snapshot, the answer supported by research is **sometimes, under constrained conditions—not reliably enough to collapse those stages into one construct called “AI analytics.”**