# AI-Ready and Agent-Ready Data: What the Evidence Actually Establishes **Research snapshot:** October 3, 2026
**Scope:** empirical and technical evidence on data-platform requirements for LLM assistants and agents. Vendor material is used only to identify terminology and audit industry claims, not as evidence that an architecture improves AI performance. The review deliberately distinguishes controlled evaluations, benchmark evidence, technical standards, threat models, and commercial assertions. ## Executive synthesis and construct map The central finding is fairly sharp: > **“Agent-ready data” is not an established scientific construct.** The literature supports a narrower collection of mechanisms—schema grounding, selective metadata, retrieval, temporal grounding, well-described tools, constrained interfaces, and especially security controls around untrusted context and actions. It does **not** establish a new, general data-platform architecture that agents require. Research does show that LLM systems are unusually sensitive to **how information is exposed**. Schema size matters; relevant database content matters; descriptions can help; irrelevant or excessive descriptions can hurt; tool descriptions measurably affect tool choice; current information must often be retrieved externally; and untrusted data can be mistaken for instructions. Those are real empirical findings. [1](https://arxiv.org/html/2306.00739v4) The strongest genuinely **agent-specific** result is not about semantic layers, data contracts, lineage, or data products. It is about **authority and instruction boundaries**. Once a model can act through tools, data returned by those tools becomes a potential control channel. AgentDojo explicitly evaluates agents whose external-tool results contain adversarial instructions; InjecAgent reports substantial attack success across more than a thousand indirect-prompt-injection cases; capability/policy-based systems such as CaMeL show that separating trusted control from untrusted data can materially improve security, though generally with utility and deployment assumptions attached. [2](https://arxiv.org/abs/2406.13352) The strongest evidence for **metadata** is similarly narrower than the rhetoric. SQL-PaLM found that query-specific database content improved BIRD execution accuracy from 55.15% to 58.80%, but supplying full column descriptions reduced it to 54.69%. Soft schema selection could make modest gains while aggressive hard pruning sometimes made performance worse. The same work reports degradation as schemas become wider. The implication is not “more metadata is always better”; it is **relevant, correctly retrieved context is useful, while context overload and recall failures are costly**. [1](https://arxiv.org/html/2306.00739v4) Enterprise realism also changes the picture dramatically. Spider 2.0 contains 632 enterprise-style data tasks over systems including BigQuery and Snowflake, with roughly 812 columns per database on average and tasks that can require metadata, dialect documentation, data-model descriptions, and project code. Early agentic systems achieved only roughly the high teens to low twenties in task success depending on track/setup—far below results on classic Spider—showing that “LLM can do text-to-SQL” does not imply “LLM can reliably navigate an enterprise data platform.” [3](https://arxiv.org/abs/2411.07763) ### Construct and terminology map
Term Status in research What can safely be inferred
**AI-ready data** **No stable scientific definition found.** Current definitions are primarily enterprise/vendor umbrella definitions combining quality, accessibility, governance, context, and traceability. DataHub, IBM and others define overlapping but non-identical properties. [4](https://datahub.com/blog/ai-ready-data/) Treat as a business label unless an author operationalizes the exact properties being tested.
**Agent-ready data** **Even less standardized.** Recent practitioner definitions add real-time access, context, guardrails and tool/API accessibility. [5](https://www.cloudera.com/events/webinars/agent-ready-data-to-power-your-ai.html) Not an independently validated data-management maturity level.
**Semantic layer** Longstanding BI/data architecture concept, but current implementations vary in metrics, entities, dimensions, joins and access semantics; dbt and Databricks describe overlapping but product-specific constructs. [6](https://www.getdbt.com/blog/semantic-layer-introduction) A meaningful engineering category, but not one standardized intervention in AI research.
**Metadata-rich data** Descriptive phrase rather than a distinct research construct. Evidence must be decomposed into descriptions, types, keys, examples, values, timestamps, provenance, etc.
**Data product** Established practitioner/data-mesh concept; “AI-ready data product” is an emerging extension rather than a benchmarked AI construct. The BDVA publication explicitly presents AI-ready data products as a new paradigm. [7](https://bdva.eu/news/ai-ready-data-products-publication/) Claims of improved AI consumption remain mostly extrapolation.
**Knowledge graph** Established technical/research representation used extensively in KGQA and graph-RAG research. ToG, for example, has LLMs explicitly explore KG entities and relations. [8](https://arxiv.org/abs/2307.07697) Benefits are task-dependent; existence of a mature KG literature does not establish that enterprise agents require one.
**Ontology** Established formal knowledge-representation family; RDF and related Semantic Web work provide explicit model-theoretic semantics. [9](https://www.w3.org/TR/rdf-mt/) Much more formally grounded than “AI-ready data,” but ontology usefulness to an LLM still requires empirical evaluation.
**Context layer** Mainly contemporary architecture/vendor terminology. No stable intervention or benchmark corresponds to it.
**Governed metrics** Practitioner semantic-model terminology describing centrally defined metric logic. Logical value is clear where business questions depend on metric definitions, but AI-specific controlled evidence is weak.
**Provenance** Stable technical construct. W3C PROV models entities, activities, agents, derivation, revisions and time, explicitly supporting traceability, trust assessment and reproducibility. [10](https://www.w3.org/TR/prov-primer/) Do not conflate provenance with evidence that an LLM knows how to use provenance correctly.
**Lineage** Mature data-management concept closely related to process/derivation provenance. Direct experiments showing enterprise agents perform better when lineage is supplied are conspicuously scarce.
**Data contract** Data/software engineering practice for machine-enforced expectations. No strong direct evidence located that contracts are a prerequisite for agentic AI.
**MCP** A defined interoperability protocol, not a data-quality or semantic model. Its own documentation calls it a standard for connecting AI applications to external data, tools and workflows. [11](https://modelcontextprotocol.io/docs/2026-07-28/getting-started/intro) Connectivity is not equivalent to correctness, governance, semantic consistency or security.
This distinction is crucial because several industry arguments effectively take a chain of plausible propositions—*agents need context; metadata provides context; our semantic/metadata product provides metadata; therefore agents require our architectural layer*—and present the conclusion with greater certainty than the empirical literature warrants. The first two links have evidence. The last one usually does not. ## Structured-data evidence: schemas, metadata, semantics, graphs, quality, freshness and provenance ### AI as a data consumer There is now substantial empirical evidence for LLMs consuming relational schemas, table contents, database values, API descriptions, retrieved documents, knowledge graphs and structured tool definitions. The evidence base is strongest for **text-to-SQL**, **table reasoning**, **retrieval**, **KG question answering**, and **tool calling**; it is much weaker for the holistic enterprise data-platform abstractions described by vendors. [12](https://arxiv.org/abs/2305.03111) BIRD was an important step beyond classic Spider: it contains 12,751 text-to-SQL pairs over 95 databases, about 33.4 GB of data and 37 domains, including dirty database content and questions requiring external/domain knowledge. Its initial ChatGPT baseline achieved 40.08% execution accuracy versus 92.96% for humans. BIRD therefore established that database content and contextual knowledge are first-class parts of realistic text-to-SQL rather than mere SQL-syntax generation. [12](https://arxiv.org/abs/2305.03111) Spider 2.0 goes further toward enterprise workflows. Its tasks involve large schemas, multiple SQL dialects and real enterprise-derived databases, while context may include schema definitions, data-model documentation, repository/code context and external dialect documentation. This is strong evidence that an AI data consumer must often **find and combine information that a human data engineer would also need**. It is not evidence that any particular commercial semantic architecture is required. [3](https://arxiv.org/abs/2411.07763) ### Schema and metadata evidence The schema evidence is one of the strongest areas in the review, but it undermines simplistic “metadata-rich is better” messaging. SQL-PaLM provides unusually useful controlled ablations. On BIRD, adding query-specific database content raised execution accuracy from **55.15% to 58.80%**. Yet adding all available column descriptions dropped performance from **58.80% to 54.69%**. Retrieval-based and program-aided schema selection sometimes recovered small gains when relevant columns were described without aggressively removing the rest; oracle selection reached materially higher performance, confirming that identifying the right schema context is itself a significant bottleneck. [1](https://arxiv.org/html/2306.00739v4) The same study found that performance falls as the number of available columns grows. Different selection mechanisms work better in different width regimes, and hard pruning can perform worse because a necessary column omitted from context cannot subsequently be recovered. [1](https://arxiv.org/html/2306.00739v4) This leads to a fairly robust conclusion: **Table names, columns, relevant values and schema links matter; relevance management matters at least as much as metadata volume.** The available evidence favors *selective contextualization* over dumping a catalog into the model context. [1](https://arxiv.org/html/2306.00739v4) Evidence by metadata type is less complete than the marketing language implies:
Metadata Evidence assessment
**Table/column names** Directly important; schema grounding is foundational to text-to-SQL. Wider and ambiguous schemas degrade performance. [1](https://arxiv.org/html/2306.00739v4)
**Column/table descriptions** **Mixed/conditional.** Relevant descriptions can help; exhaustive descriptions can hurt through context overload. [1](https://arxiv.org/html/2306.00739v4)
**Database values / examples** **Directly supported.** Query-specific content improved SQL-PaLM; BIRD explicitly includes value grounding as part of the difficulty. [1](https://arxiv.org/html/2306.00739v4)
**Schema linking** Strongly supported as an intermediate reasoning problem; DIN-SQL explicitly decomposes schema linking before generation, and Dr.Spider shows brittleness to schema/question perturbations. [13](https://arxiv.org/abs/2304.11015)
**Types** Routinely supplied in schema representations, but clean controlled isolation of “types versus no types” in frontier LLM enterprise settings is limited.
**Primary/foreign keys and relationships** Architecturally sensible and widely represented in text-to-SQL systems; direct modern LLM ablations isolating their incremental enterprise value are much less common than the general schema-linking literature.
**Human-authored business descriptions** Conditional evidence via text-to-SQL metadata; no evidence that every description is useful.
**Examples / verified queries** Few-shot SQL examples are commonly effective, but this is not equivalent to proving a commercial “verified query repository” as a required platform layer.
Dr.Spider reinforces the brittleness point. It introduced 17 perturbation types over questions, schemas and SQL and found that even the most robust evaluated systems suffered an overall performance drop of about **14 points**, with degradation of up to **50.7 points** under the hardest perturbation. This shows that superficial changes to schemas and phrasing can materially affect correctness. [14](https://arxiv.org/abs/2301.08881) ### Semantic-information and semantic-layer evidence The literature supports **semantics as information**, but it does not yet support **the enterprise semantic layer as a uniquely necessary architecture**. A model clearly benefits when ambiguity is resolved: mappings from natural-language phrases to columns, relevant entity/value information, domain knowledge and examples can all improve results. BIRD, SQL-PaLM, DIN-SQL and CHESS-style systems collectively demonstrate that contextual information beyond bare SQL syntax helps. [12](https://arxiv.org/abs/2305.03111) What is notably missing is the controlled experiment that many industry claims implicitly presuppose: > same model + same enterprise questions + same underlying data, tested against **raw schema**, **raw schema plus documentation**, **semantic model**, **governed metric API**, and **verified-query layer**. I did not find a credible published study performing that comparison across representative enterprise domains. Consequently, evidence for business semantics does **not** currently let us estimate the incremental causal benefit of a commercial semantic layer over simpler, well-retrieved textual metadata. The same problem applies to **governed metrics**. If a question asks for “ARR,” “active customer,” “churn” or “gross margin,” and the organization has a non-obvious definition, the agent logically requires access to that definition somewhere. But research has not established that the definition must reside in a centralized semantic layer rather than documentation, a metric API, SQL exemplars or another machine-retrievable representation. SQL-PaLM's results in fact warn that representation and context selection can matter as much as the presence of additional information itself. [1](https://arxiv.org/html/2306.00739v4) **Semantic-layer evidence audit: plausible value, inadequate controlled evidence.** ### Ontologies and knowledge graphs Knowledge graphs have real empirical evidence behind them—but mostly for tasks where graph structure is naturally relevant. Think-on-Graph has an LLM iteratively navigate entities and relations in a knowledge graph and reported state-of-the-art results on six of nine evaluated KG-reasoning datasets; the authors also found cases where smaller LLMs augmented with graph traversal surpassed much larger models. This establishes that **explicit graph structure can be a useful reasoning/retrieval substrate**. It does not establish necessity. [15](https://arxiv.org/abs/2307.07697) Microsoft's GraphRAG work likewise showed substantial gains over naive vector-RAG on **global sensemaking/query-focused summarization** over corpora around one million tokens. The graph is used to create entity/community structure and hierarchical summaries that a simple local semantic retriever cannot easily provide. [16](https://arxiv.org/abs/2404.16130) But both examples contain an important confound: the graph changes **retrieval and aggregation**, not merely semantic representation. Better results may arise because the graph selects different evidence, supplies precomputed summaries, or provides multi-hop traversal—not simply because facts are “semantically modeled.” Graph-versus-text studies therefore often fail to isolate **representation** from **retrieval strategy**. [16](https://arxiv.org/abs/2404.16130) Moreover, successful text-to-SQL, RAG and tool-using systems operate without enterprise knowledge graphs. That directly falsifies a strong necessity claim. KGs are best described as **conditionally useful for relationship-heavy, multi-hop or corpus-level tasks**, not as a universal prerequisite for reliable agents. [1](https://arxiv.org/html/2306.00739v4) ### Data-quality robustness and quality metadata Evidence here is surprisingly thin relative to the strength of industry claims. BIRD deliberately includes dirty and realistic database contents, establishing that real text-to-SQL cannot assume perfectly clean values. Dr.Spider shows large degradation under controlled perturbations. But neither study provides the kind of factorial quality experiment needed to estimate an agent's sensitivity separately to **nulls, wrong values, duplicates, inconsistent categories, delayed updates and schema drift**. [12](https://arxiv.org/abs/2305.03111) The literature therefore supports the modest conclusion that **LLM analytical systems are not magically robust to bad inputs**. It does not yet provide robust enterprise curves such as “a 5% duplicate rate causes X% answer degradation” or “surfacing a freshness SLA reduces stale-answer errors by Y%.” The related question—whether **quality metadata itself** changes model behavior—is even less studied. I found no convincing controlled enterprise study comparing, for example: `table only` versus `table + quality score / validation failures / completeness / freshness SLA / anomaly flags` while holding the data and task fixed. That is a major evidence gap. Today, assertions that agents “need data quality metadata” are technically plausible, but mostly extrapolated from conventional data governance and from the obvious fact that bad evidence can produce bad conclusions. ### Freshness and temporal information Temporal grounding has substantially better direct evidence. FreshQA was explicitly designed around fast-changing world knowledge and false-premise questions. Across evaluated models, fast-changing questions were difficult; FreshPrompt improved answers by inserting current search evidence. The authors also found that the **quantity and ordering of retrieved evidence** affected correctness. This directly supports external grounding when information changes after model training. [17](https://arxiv.org/abs/2310.03214) STALE, introduced in 2026, targets a related agent problem: a newer observation implicitly invalidates an older memory without explicitly saying “the previous fact is wrong.” Its benchmark contains **400 expert-validated conflict scenarios and 1,200 queries across more than 100 topics**, documenting that belief revision in persistent agent memory is a nontrivial problem. [18](https://arxiv.org/abs/2605.06527) What the literature does **not** yet establish is that attaching a `last_updated_at` field or freshness score is sufficient. Most temporal work compares stale model memory with newer external evidence or evaluates conflicting versions; relatively little cleanly isolates: - no temporal metadata; - timestamps only; - timestamps plus validity intervals; - trusted source freshness; - explicit supersession relationships. Thus, **fresh access is supported; a specific freshness-metadata architecture is not**. [17](https://arxiv.org/abs/2310.03214) ### Provenance and citations Provenance is a mature non-AI construct. W3C PROV explicitly represents derivation, activities, agents, revisions and temporal relationships, with intended uses including reproducibility, source assessment and tracking how outputs were generated. [19](https://www.w3.org/TR/prov-primer/) LLM research supports a narrower claim around **citation-backed verifiability**. ALCE evaluates answer correctness and whether citations actually support generated statements. Even the best systems in the original evaluation lacked complete citation support on roughly **50% of ELI5 answers**, demonstrating that citation generation itself remains imperfect. [20](https://aclanthology.org/2023.emnlp-main.398/) This distinction matters: **Citation/provenance availability ≠ correctness.**
**Citation/provenance availability ≠ correct use of provenance.**
**Citation/provenance availability ≠ calibrated human trust.** ALCE measures answer correctness and citation quality and validates its automatic metrics against human judgments; it does **not** demonstrate that showing citations causes users to detect more AI errors or calibrate trust appropriately. [20](https://aclanthology.org/2023.emnlp-main.398/) Accordingly, provenance is strongly justified for traceability and reproduction independently of AI, while claims that provenance automatically makes agents more trustworthy are **only conditionally supported**. ## Interfaces and access: curation, tool abstraction, machine-readable documentation and protocols ### Curated versus open access The empirical literature argues against the intuitive idea that an agent should simply see everything. SQL-PaLM's schema experiments show the central trade-off particularly clearly. Reducing irrelevant schema can help, but **hard selection** may hurt when necessary columns are accidentally removed. “Soft” approaches that identify or describe likely-relevant elements while retaining recoverability perform better in several settings. [1](https://arxiv.org/html/2306.00739v4) ToolLLM addresses a still larger action space: more than **16,000 real-world APIs**. Rather than putting every API directly in context, its architecture includes an API retriever that selects likely-relevant tools for the task. The very architecture of the benchmark/system reflects the practical impossibility of unconstrained all-tool prompting at that scale. [21](https://arxiv.org/abs/2307.16789) At the document level, τ-Knowledge is particularly sobering. Its 2026 banking environment contains roughly **700 interconnected knowledge documents plus tools**; even frontier models operating with high reasoning budgets achieved only about **25.5% pass@1**, with failures involving retrieval of the right policies and reasoning across them. This is closer to the enterprise problem of “there is lots of relevant organizational knowledge somewhere” than classic single-document RAG. [22](https://arxiv.org/abs/2603.04370) The conclusion is therefore conditional rather than simplistic: > **Curating/retrieving a task-relevant subset usually addresses a genuine model limitation, but premature filtering creates its own failure mode.** The desirable property is not “closed access” but **high recall with controlled distraction**. ### Scale effects Four strands of evidence converge on scale sensitivity. First, SQL-PaLM explicitly reports lower text-to-SQL performance as available column counts increase. [1](https://arxiv.org/html/2306.00739v4) Second, Spider 2.0's roughly 812-column average schemas and frequently thousand-plus-column environments are dramatically harder than classic benchmark schemas. [3](https://arxiv.org/abs/2411.07763) Third, ToolLLM resorts to retrieval in a 16,464-API environment rather than treating every tool as an equally visible candidate. [21](https://arxiv.org/abs/2307.16789) Fourth, τ-Knowledge shows that a few hundred interdependent policy/knowledge documents can remain extremely difficult even for 2026 frontier systems. [22](https://arxiv.org/abs/2603.04370) What is **not** known is a stable scaling law. There is no generally established function mapping “number of tables/tools/metrics/documents” to task failure across model families. Relevant-context density, naming quality, retrieval, model generation and task complexity interact strongly. So the evidence supports **degradation with larger search spaces**, not a universal threshold such as “agents can handle N tables.” ### Tool abstraction Tool-use research clearly establishes that models can interact with typed APIs and that tool selection is itself difficult. τ-bench gives agents domain-specific APIs and business-policy instructions while requiring multi-turn interaction with a simulated user. Even GPT-4o succeeded on fewer than half of tasks in the original evaluation, and repeated-trial reliability was much worse; retail pass^8 fell below 25%. Structured APIs therefore do not eliminate reasoning, policy-following or execution failures. [23](https://arxiv.org/abs/2406.12045) ToolSandbox adds stateful environments, implicit dependencies and conversational interaction and finds persistent difficulties when correct actions depend on changing state, canonicalization or obtaining missing information. [24](https://arxiv.org/abs/2408.04682) What is much less established is the specific claim that **high-level domain tools outperform low-level generic data tools** in a controlled enterprise-data experiment. Few studies hold the underlying capability fixed while comparing something like: `execute_sql(sql)`
versus
`get_net_revenue(period, region)`. The engineering argument for higher-level tools is compelling—smaller parameter spaces and embedded domain logic—but the comparative empirical literature is still thin. This should be classified as **plausible but inadequately tested**, not established fact. ### Machine-readable documentation The best recent evidence concerns **tool descriptions**, not data catalogs. A 2026 empirical study examined **856 tools across 103 MCP servers**. It found that 97.1% of descriptions contained at least one identified “smell,” and 56% failed to clearly state purpose. Augmenting descriptions increased median task success by **5.85 percentage points** and partial goal completion by **15.12%**. But it also increased execution steps by **67.46%** and made performance worse in **16.67% of cases**. [25](https://arxiv.org/abs/2602.14878) This is valuable because it experimentally demonstrates that **interface metadata is behaviorally consequential**. It also reinforces the “more context is not automatically better” finding from SQL-PaLM. [25](https://arxiv.org/abs/2602.14878) However, note what the experiment does *not* prove. MCP tool descriptions are still primarily natural-language descriptions attached to structured tool schemas. The study does not establish that JSON/typed documentation universally outperforms equally complete prose, nor that human-readable catalog descriptions must be redesigned into some specific new machine-oriented format. [25](https://arxiv.org/abs/2602.14878) The data-specific evidence for **structured documentation versus prose containing the same facts** remains sparse. ### Agent protocols MCP is a protocol/interoperability specification. Official documentation describes it as an open-source standard for connecting AI applications to external data sources, tools and workflows—the analogy used is a “USB-C port” for AI applications. [11](https://modelcontextprotocol.io/docs/2026-07-28/getting-started/intro) That establishes **interface standardization**. It does not establish: - semantic consistency between connected sources; - correct metric definitions; - data quality; - source freshness; - safe authorization; - correct tool selection; - resilience to prompt injection; - reliable multi-step execution. Indeed, the MCP tool-description study demonstrates that two MCP-compliant tools can differ enough in documentation quality to cause measurable differences in agent behavior. [25](https://arxiv.org/abs/2602.14878) Therefore: > **“MCP-compatible” is evidence of connectability, not evidence of agent readiness.** The protocol may be commercially and operationally important without being a scientifically validated data-readiness criterion. ## Governance and security: authorization, injection, lineage, contracts, data products and policies ### Authorization and confused authority Here the evidence becomes much more specifically agentic. Traditional analytics mostly expose information to a human or application. An agent can additionally **decide which tool to invoke and cause side effects**. That turns failures of instruction interpretation into authorization failures. AgentDojo was created specifically around this problem: agents use tools over **untrusted data**, and data returned by tools may contain instructions intended to hijack the agent into executing attacker goals. The benchmark confirms both substantial baseline task difficulty and prompt-injection vulnerability. [2](https://arxiv.org/abs/2406.13352) InjecAgent contains **1,054 test cases**, 17 user tools, 62 attacker tools and evaluations across 30 agent configurations. The study reports a roughly **24% vulnerability rate for a ReAct GPT-4 agent** under its attacks, while stronger malicious prompting can substantially increase attack success. [26](https://arxiv.org/abs/2403.02691) This is direct evidence that a system which merely gives an LLM authenticated access to tools does **not** thereby inherit safe authorization behavior. The resulting platform requirement is more defensible than most “AI-ready data” claims: **effective authorization must be enforced outside the model's discretionary natural-language reasoning whenever the consequences matter.** That inference is supported by the repeated empirical failure of models to reliably distinguish legitimate user instructions from hostile instructions embedded in retrieved data. [2](https://arxiv.org/abs/2406.13352) ### Prompt injection through data and metadata The distinction between data and instructions is fundamentally unstable inside a plain language-model context. To the model, both arrive as tokens. AgentDojo directly demonstrates attacks embedded in external tool-returned data. [2](https://arxiv.org/abs/2406.13352) This is qualitatively different from ordinary bad data. A wrong revenue figure may produce a wrong analytical answer; an injected instruction inside a retrieved email, webpage, row or tool result may cause the agent to **change what task it thinks it is executing**. Recent adaptive red-teaming indicates that the issue remains active even as model generations improve. PIMiner's 2026 evaluations report high indirect-prompt-injection success against several frontier agent setups under adaptive attack, including attack-success rates above 50% for some GPT and Gemini configurations on evaluated benchmarks. As a fresh preprint, those exact rates should not be generalized beyond its threat model, but they argue strongly against declaring the problem solved. [27](https://arxiv.org/abs/2608.05108) Mitigation evidence favors **architectural enforcement** over instruction-only defenses. CaMeL separates control and data flow and uses capabilities/policies to prevent untrusted content from controlling sensitive actions; its evaluation reports completion of about **67% of AgentDojo tasks while providing its stronger security guarantees**. [28](https://arxiv.org/abs/2503.18813) NetInjectBench provides another domain-specific example. In a network-operations benchmark, naive agents took unsafe actions in **82.5%** of attack trials; prompt-oriented defenses still had double-digit unsafe-action rates, whereas a metadata-aware policy gate reported **0 unsafe actions in 240 attacks**, subject to the key assumption that the policy metadata itself remains trustworthy. [29](https://arxiv.org/abs/2607.10490) That last qualification is central: machine-readable governance becomes valuable only when its **integrity and enforcement boundary are stronger than the model's susceptibility to untrusted content**. ### Lineage evidence This review found a striking mismatch between industry rhetoric and direct evaluation. Lineage/provenance clearly has conventional value for debugging, reproducibility, dependency analysis and tracing derivations; W3C PROV formalizes many of those relationships. [30](https://www.w3.org/TR/prov-primer/) What is missing is a controlled AI-agent study demonstrating that giving an agent enterprise data lineage improves, for example: - correct source selection; - downstream impact analysis; - root-cause diagnosis; - interpretation of transformed metrics; - debugging of wrong analytical results. Security systems such as CaMeL sometimes track information flow or provenance-like labels, but **security taint/provenance is not the same thing as warehouse lineage** across tables, models and pipelines. [28](https://arxiv.org/abs/2503.18813) Therefore, **“lineage is essential for AI reasoning” is currently unsupported** as an empirical generalization. It is a reasonable research hypothesis and may be particularly useful for debugging and source selection, but the required controlled experiments have largely not been done. ### Data contracts The evidence is even weaker for data contracts. I found no material controlled study in which otherwise identical AI agents were run against: - data with machine-enforced contracts; - documented but unenforced expectations; - no contracts; under controlled schema drift, nullability changes, unit changes or semantic changes. Contracts may obviously prevent broken data from reaching *any* consumer. That is ordinary software/data reliability. It does not establish an **AI-specific** requirement. Accordingly, “data contracts are prerequisites for agentic AI” is **unsupported** by direct evidence. A more defensible claim would be: *machine-enforced contracts can prevent classes of upstream failures before they reach an AI consumer*, a proposition that still needs AI-specific quantification. ### Data products The same distinction applies to data products. Recent industry and ecosystem publications explicitly advocate “AI-ready data products,” combining quality, semantics and accessibility as a design paradigm. [7](https://bdva.eu/news/ai-ready-data-products-publication/) But I found no credible controlled study evaluating agents as consumers of formally managed data products versus otherwise equivalent datasets or APIs. Thus: > **Data-product principles may transfer well to AI because discoverability, ownership, documentation and quality are valuable to many consumers. But the AI-specific benefit is presently extrapolation, not demonstrated causal evidence.** Claims that **every** data product must now become “agent-ready” go beyond the literature. ### Human-readable versus machine-readable governance There is meaningful evidence that simply handing agents the same policies humans read is not enough. τ-bench supplies domain policies together with structured tools, yet frontier systems still fail a large fraction of end-to-end tasks. [23](https://arxiv.org/abs/2406.12045) τ-Knowledge goes further with hundreds of interdependent banking documents and reports only about 25.5% pass@1 for strong models under its more difficult knowledge-intensive setting. [22](https://arxiv.org/abs/2603.04370) In contrast, domain-specific experiments such as NetInjectBench show that an externally enforced policy gate can dramatically outperform merely reminding a model to behave safely, although such results depend on trusted policy metadata and do not automatically transfer to arbitrary enterprise systems. [29](https://arxiv.org/abs/2607.10490) The evidence therefore supports **machine enforcement of critical governance rules** more strongly than it supports wholesale “machine-oriented redesign” of catalogs, glossaries or documentation. ## Benchmark validity, scale and contradictory evidence ### Benchmark map
Benchmark / study family What it tests Enterprise relevance Major blind spots
**Spider** Cross-domain text-to-SQL Foundational schema grounding Small, clean schemas; static; no enterprise permissions/governance.
**BIRD** Large DB text-to-SQL with values and external knowledge Better data-size/content realism; 95 DBs, 37 domains, 33.4 GB. [12](https://arxiv.org/abs/2305.03111) Still benchmark-style, mostly static analytical queries; limited organizational semantics/permissions.
**Spider 2.0** Enterprise-style SQL/data workflows Strongest public match in this set: hundreds of tasks, large schemas, multiple industrial DB systems. [3](https://arxiv.org/abs/2411.07763) No full real-enterprise organizational environment; limited live permission, ownership and temporal change.
**Dr.Spider** Robustness to schema/question/SQL perturbations Strong evidence against brittle schema assumptions. [14](https://arxiv.org/abs/2301.08881) Perturbations are designed experiments, not production-quality incidents.
**FreshQA** Current knowledge, changing facts, false premises Strong temporal-grounding evidence. [17](https://arxiv.org/abs/2310.03214) Web factual QA rather than warehouse freshness or table versions.
**STALE** Detecting newer information that invalidates agent memory Relevant to supersession and persistent agent context. [31](https://arxiv.org/abs/2605.06527) Personal/everyday memory scenarios, not enterprise temporal data governance.
**ALCE** Correctness and support of generated citations Relevant to provenance/verifiability. [20](https://aclanthology.org/2023.emnlp-main.398/) Not structured-data provenance; not a user trust-calibration experiment.
**ToG/KGQA** Multi-hop reasoning over explicit KGs Tests whether graph traversal can aid reasoning. [8](https://arxiv.org/abs/2307.07697) Public/static KGs; graph and retrieval effects often confounded.
**GraphRAG evaluations** Global synthesis over large text collections Relevant where cross-document relationships matter. [16](https://arxiv.org/abs/2404.16130) Not relational analytics; graph construction itself uses LLMs and introduces cost/error.
**ToolBench/ToolLLM** API retrieval and invocation across 16k+ APIs Strong tool-search-scale evidence. [21](https://arxiv.org/abs/2307.16789) Many instructions/tasks generated synthetically; not enterprise data governance.
**ToolSandbox** Stateful conversational tool use Better approximation of dynamic applications. [24](https://arxiv.org/abs/2408.04682) Still sandboxed and narrow relative to enterprise systems.
**τ-bench** Stateful tools + user interaction + policy constraints Particularly relevant to business agents. [23](https://arxiv.org/abs/2406.12045) Few domains and simulated users/environments.
**τ-Knowledge** Hundreds of connected enterprise-style knowledge documents + tools Good evidence on document/policy scale. [22](https://arxiv.org/abs/2603.04370) Banking benchmark, not heterogeneous live corporate infrastructure.
**AgentDojo** Indirect prompt injection through tool-returned data Highly relevant to agent security. [2](https://arxiv.org/abs/2406.13352) Synthetic/sandboxed attack environment.
**InjecAgent** Large-scale indirect prompt-injection cases Strong attack-surface evidence. [26](https://arxiv.org/abs/2403.02691) Constructed attack scenarios; model results age quickly.
The major shared limitation is that even the better benchmarks rarely combine **large schemas + messy data + live temporal change + conflicting documentation + row/column permissions + multiple identity domains + side-effecting actions + malicious content + organization-specific metrics** in one environment. That is the environment implied by much “agent-ready enterprise data” rhetoric, and it remains largely **unbenchmarked**. ### Contradictory evidence Several apparent disagreements in the literature disappear once the intervention is examined carefully. **More metadata helps / more metadata hurts.** SQL-PaLM finds both. Relevant database content gives a substantial improvement, while full column descriptions reduce accuracy. Soft schema highlighting can help slightly; hard filtering can hurt badly when recall is lost. The contradiction is explained by a trade-off among relevance, context length and schema recall—not by metadata being inherently good or bad. [1](https://arxiv.org/html/2306.00739v4) **More retrieved evidence helps / more context distracts.** FreshPrompt finds the number and order of current search evidences affect answer correctness; SQL-PaLM and large-schema results show that indiscriminate context can degrade reasoning. These are compatible: marginal evidence is useful only while its information value exceeds its distraction/context cost. [17](https://arxiv.org/abs/2310.03214) **Knowledge graphs improve reasoning / KGs are unnecessary.** ToG and GraphRAG show real gains on graph-suited and global-synthesis tasks. Text-to-SQL and ordinary tool agents demonstrate successful reasoning without KGs. The correct conclusion is task-conditional usefulness, not architectural necessity. [32](https://arxiv.org/abs/2307.07697) **Richer tool descriptions help / richer descriptions increase cost and sometimes hurt.** The MCP-description study observes a median +5.85-point success gain, but 67.46% more execution steps and regressions in 16.67% of cases. This is unusually direct evidence that metadata quality and metadata quantity should not be conflated. [25](https://arxiv.org/abs/2602.14878) **Prompt defenses help / prompt injection remains severe.** Some defenses reduce attack success under fixed attacks, but adaptive red-team studies continue to recover high success rates. More structural approaches—capabilities, policy gates, trusted execution boundaries—show stronger results under their evaluated assumptions. Differences in adversary adaptiveness and enforcement model explain much of the apparent conflict. [28](https://arxiv.org/abs/2503.18813) ### Evidence-quality interpretation The field is dominated by benchmarks rather than deployment experiments. That matters. A result on 100 clean benchmark databases is not a production study of a data mesh. A result using synthetic tool descriptions is not proof of enterprise catalog redesign. A KGQA gain over Freebase does not demonstrate that an ontology program will pay for itself in a Fortune 500 warehouse. A prompt-injection benchmark does, however, demonstrate a **mechanism** that production systems must address, because the attack relies on a fundamental property of language-model context rather than a particular vendor stack. [12](https://arxiv.org/abs/2305.03111) Enterprise transferability is highest when a paper isolates a causal mechanism that clearly survives environmental changes—schema overload, stale knowledge, prompt injection, tool-description ambiguity—and lowest when the “intervention” bundles an entire architecture whose components are not independently ablated. ## Industry claim audit and what current evidence does not support ### Claim audit
Industry claim Classification Evidence judgment
**“AI requires a semantic layer.”** **Contradicted as a universal necessity; conditional benefit is plausible.** Models can and do operate directly over database schemas and values. Semantics/context improve many tasks, but no controlled literature establishes a commercial semantic layer as a necessary condition. [1](https://arxiv.org/html/2306.00739v4)
**“Metadata is the new interface for agents.”** **Conditionally supported.** Schema context and tool descriptions measurably affect performance, but excessive metadata can reduce it. Metadata is an important model-facing interface, not a guarantee of reliability. [1](https://arxiv.org/html/2306.00739v4)
**“Every data product must become agent-ready.”** **Unsupported.** “AI-ready data products” are being advocated as a new paradigm, but direct agent-consumer evaluations were not found. [7](https://bdva.eu/news/ai-ready-data-products-publication/)
**“Data contracts are prerequisites for agentic AI.”** **Unsupported.** No controlled AI evidence establishes contracts as a prerequisite. Benefits are extrapolated from general interface/data reliability.
**“Knowledge graphs are necessary for reliable agents.”** **Contradicted as a necessity.** KGs improve some multi-hop/global tasks, but many successful agent and text-to-SQL systems work without them. [32](https://arxiv.org/abs/2307.07697)
**“Agents need governed metrics.”** **Plausible but inadequately tested.** Agents need correct metric definitions when questions depend on organization-specific semantics, but no controlled evidence establishes a governed metric layer as the required representation.
**“MCP makes data platforms agent-ready.”** **Unsupported as a broad readiness claim.** MCP standardizes connections. It does not provide data quality or semantic correctness, and MCP tool-description quality itself materially affects success. [11](https://modelcontextprotocol.io/docs/2026-07-28/getting-started/intro)
**“Lineage is essential for AI reasoning.”** **Unsupported.** Lineage has clear non-AI value, but direct agent-consumption studies demonstrating reasoning improvements are missing. W3C provenance capabilities should not be mistaken for LLM evaluation evidence. [30](https://www.w3.org/TR/prov-primer/)
**“AI will force companies to modernize data platforms.”** **Unsupported as a technical necessity.** Current studies expose scaling, metadata and security problems; they do not demonstrate that a particular modernization program is causally necessary.
**“Companies with modern data stacks are automatically better positioned for agents.”** **Unsupported.** No benchmark maps “modern stack maturity” to agent task success. Poor tool descriptions and unsafe authority boundaries can exist on modern infrastructure just as easily as legacy infrastructure. [25](https://arxiv.org/abs/2602.14878)
**“AI-ready data requires a fundamentally new architecture.”** **Unsupported.** Most demonstrated requirements—accurate schema, relevant metadata, fresh information, source traceability, authorization and validation—have pre-AI antecedents. The genuinely new pressure is chiefly around model context selection and the security consequences of agents interpreting data while wielding actions. [33](https://www.w3.org/TR/prov-primer/)
The most defensible statement is therefore weaker but more useful: > **Agents raise the value of some existing data-management properties and introduce new interface/security failure modes. They do not, on current evidence, imply a universally new data architecture.** ### What current evidence does support A conservative evidence hierarchy looks like this:
Proposition Evidence strength
LLM analytical accuracy depends on receiving the relevant schema/context. **Strong**. [1](https://arxiv.org/html/2306.00739v4)
Wider schemas and larger choice spaces can degrade reliability. **Strong**. [1](https://arxiv.org/html/2306.00739v4)
Relevant DB values/examples can improve query generation. **Strong**. [1](https://arxiv.org/html/2306.00739v4)
More metadata is not monotonically better. **Strong**. [1](https://arxiv.org/html/2306.00739v4)
Tool-description quality affects selection/execution behavior. **Directly supported.** [25](https://arxiv.org/abs/2602.14878)
Retrieving fresh external information improves changing-world questions. **Strong for factual QA.** [17](https://arxiv.org/abs/2310.03214)
Agents can fail to supersede stale memory correctly. **Direct benchmark evidence.** [31](https://arxiv.org/abs/2605.06527)
Knowledge graphs can improve graph-heavy/multi-hop/global retrieval tasks. **Direct, task-conditional evidence.** [32](https://arxiv.org/abs/2307.07697)
Citations enable inspectable evidence but are themselves often incomplete. **Strong.** [20](https://aclanthology.org/2023.emnlp-main.398/)
Data returned from external systems can prompt-inject tool-using agents. **Very strong empirical security evidence.** [2](https://arxiv.org/abs/2406.13352)
Critical authorization/policy enforcement should not depend solely on model obedience. **Strong inference from security evaluations; supported by enforcement-based mitigations.** [28](https://arxiv.org/abs/2503.18813)
### What current evidence does not support Current evidence does **not** justify treating any of the following as established scientific findings: “Every AI system requires a semantic layer”; “ontology is inherently superior to textual metadata”; “lineage materially improves generic agent reasoning”; “data contracts are a prerequisite”; “quality scores cause agents to abstain appropriately”; “MCP provides reliability”; “a modern data stack predicts agent success”; “data products designed for humans automatically become better AI inputs”; or “AI requires replacing the current enterprise data architecture.” Those ideas range from sensible engineering hypotheses to aggressive commercial positioning, but the causal evidence needed to distinguish them is mostly absent. A particularly important absence is the **semantic representation comparison**. There is abundant work comparing systems, but very little work holding factual information constant while encoding it alternately as: `DDL + prose`
`semantic-model entities/metrics`
`ontology/KG triples`
`high-level metric API` and then measuring correctness across the same tasks and models. Without that experiment, claims about the superiority of a representation often confound **what information was available**, **how it was retrieved**, **how much context was supplied**, and **what reasoning scaffold surrounded the model**. ## Material study table The table below prioritizes studies that materially constrain the answer rather than trying to catalogue every text-to-SQL, RAG or agent paper. ### Core structured-data and semantic studies
Study Publication status / date / direct URL Model / system Task and data environment Intervention / context Baseline / comparator N / metric / result Limitations and enterprise transferability
**Li et al., BIRD** NeurIPS 2023; arXiv 2023. [https://arxiv.org/abs/2305.03111](https://arxiv.org/abs/2305.03111) ChatGPT and contemporary text-to-SQL baselines Large database-grounded text-to-SQL; 95 DBs, 37 domains, 33.4 GB Schemas, DB content, external knowledge Existing text-to-SQL systems; human performance 12,751 pairs; execution accuracy; ChatGPT 40.08%, human 92.96% Much more realistic than Spider but static benchmark; no enterprise authorization/organizational governance. [12](https://arxiv.org/abs/2305.03111)
**Sun et al., SQL-PaLM** arXiv 2023–24. [https://arxiv.org/abs/2306.00739](https://arxiv.org/abs/2306.00739) PaLM-2 Unicorn / SQL-PaLM Spider and BIRD text-to-SQL DB values, schema filtering, column descriptions, execution feedback Same model without each intervention BIRD ablations: query-specific content 55.15→58.80 EX; full descriptions 58.80→54.69; oracle schema selection substantially higher Excellent causal ablations; model generation now old; still not enterprise business semantics. [1](https://arxiv.org/html/2306.00739v4)
**Pourreza et al., DIN-SQL** NeurIPS 2023; [https://arxiv.org/abs/2304.11015](https://arxiv.org/abs/2304.11015) GPT-family prompting pipeline Complex text-to-SQL Explicit schema linking, decomposition, generation, self-correction Less decomposed prompting Improved benchmark execution accuracy Pipeline evidence for schema linking, but components interact and do not isolate metadata types. [13](https://arxiv.org/abs/2304.11015)
**Chang et al., Dr.Spider** ICLR 2023; [https://arxiv.org/abs/2301.08881](https://arxiv.org/abs/2301.08881) Multiple text-to-SQL models Robustness under question/schema/SQL perturbation 17 perturbation types Original unperturbed examples Best systems still ≈14-point aggregate drop; up to 50.7 points on hardest perturbation Strong brittleness evidence; perturbations do not model full production data corruption. [14](https://arxiv.org/abs/2301.08881)
**Lei et al., Spider 2.0** arXiv 2024; [https://arxiv.org/abs/2411.07763](https://arxiv.org/abs/2411.07763) Code/LLM agents incl. frontier models Enterprise-style SQL/data workflows on BigQuery, Snowflake, etc. Large schemas plus metadata/docs/code context Modern text-to-SQL/agent frameworks 632 tasks; avg ≈812 columns; early success roughly 17–21% depending setup/track Best public enterprise approximation, but still curated/static and lacks full organizational permissions/change. [3](https://arxiv.org/abs/2411.07763)
**Sun et al., Think-on-Graph** ICLR 2024; [https://arxiv.org/abs/2307.07697](https://arxiv.org/abs/2307.07697) LLM + iterative KG beam search KG question answering/reasoning Explicit entities/relations and path traversal LLM-only and prior KGQA systems State of art on 6/9 evaluated datasets; smaller augmented models beat GPT-4 in some settings Supports KG usefulness, not necessity; static public KGs unlike enterprise data landscapes. [15](https://arxiv.org/abs/2307.07697)
**Edge et al., GraphRAG** arXiv 2024; [https://arxiv.org/abs/2404.16130](https://arxiv.org/abs/2404.16130) GraphRAG pipeline Global question-focused summarization over ≈1M-token corpora LLM-created entity graph, communities and summaries Naive vector RAG Substantial gains in comprehensiveness/diversity on global questions Graph representation, retrieval and pre-summarization are confounded; not structured enterprise analytics. [16](https://arxiv.org/abs/2404.16130)
**Vu et al., FreshLLMs / FreshQA** Findings of ACL 2024; [https://arxiv.org/abs/2310.03214](https://arxiv.org/abs/2310.03214) Multiple LLMs + FreshPrompt Fast-changing factual QA and false-premise QA Current search evidence Model memory, Self-Ask and search-augmented baselines >50k human judgments; current evidence materially improves FreshQA; evidence number/order matters Strong freshness evidence, but web facts rather than database freshness fields. [17](https://arxiv.org/abs/2310.03214)
**Chao et al., STALE** arXiv preprint, 2026. [https://arxiv.org/abs/2605.06527](https://arxiv.org/abs/2605.06527) LLM agents with persistent memory Detection of implicit supersession/conflict New observations against old memory Static-memory behavior 400 conflict scenarios, 1,200 queries, >100 topics; models struggle with implicit invalidation Important temporal-agent result; does not directly evaluate data-warehouse timestamps or freshness SLAs. [18](https://arxiv.org/abs/2605.06527)
**Gao et al., ALCE** EMNLP 2023; DOI **10.18653/v1/2023.emnlp-main.398**; [https://aclanthology.org/2023.emnlp-main.398/](https://aclanthology.org/2023.emnlp-main.398/) Retrieval + LLM generation Long-form QA with citations Retrieved source evidence and inline citations Multiple retrieval/generation approaches ASQA/QAMPARI/ELI5; even best system lacks complete citation support ~50% on ELI5 Excellent citation-quality benchmark; does not test enterprise lineage or human trust calibration. [20](https://aclanthology.org/2023.emnlp-main.398/)
### Tool, governance and security studies
Study Publication status / date / direct URL Model / system Task and environment Intervention / context Baseline / comparator N / metric / result Limitations and enterprise transferability
**Qin et al., ToolLLM / ToolBench** ICLR-era 2024 work; [https://arxiv.org/abs/2307.16789](https://arxiv.org/abs/2307.16789) ToolLLaMA + API retriever Tool selection/invocation Retrieval over real RapidAPI descriptions General-purpose LLM/tool baselines 16,464 APIs across 49 categories; demonstrates scalable API retrieval/use Many instructions synthetic; API ecosystem ≠ enterprise data estate. [21](https://arxiv.org/abs/2307.16789)
**Yao et al., τ-bench** arXiv 2024; [https://arxiv.org/abs/2406.12045](https://arxiv.org/abs/2406.12045) GPT-4o and other agents Stateful retail/airline tool-agent-user tasks with policies Structured domain APIs + written policy rules Multiple agent/model configurations GPT-4o <50% original task success; retail pass^8 <25% Strong end-to-end reliability evidence; few domains and simulated users. [23](https://arxiv.org/abs/2406.12045)
**ToolSandbox** Findings NAACL 2025; [https://arxiv.org/abs/2408.04682](https://arxiv.org/abs/2408.04682) Multiple tool-using LLMs Stateful conversational execution Dynamic state, dependencies, simulated user Existing tool-use agents Models remain challenged by state dependencies and insufficient-information cases Better interaction realism, but small compared with enterprise tool ecosystems. [24](https://arxiv.org/abs/2408.04682)
**Hasan et al., MCP Tool Descriptions Are Smelly** arXiv preprint, revised May 2026; DOI **10.48550/arXiv.2602.14878**; [https://arxiv.org/abs/2602.14878](https://arxiv.org/abs/2602.14878) FM agents over MCP tools Tool understanding/selection/execution Augmented tool descriptions Original descriptions 856 tools / 103 servers; median +5.85 pp task success, +15.12% partial completion; +67.46% steps; regression in 16.67% cases Excellent real-ecosystem metadata study, but not relational-data semantics; preprint. [25](https://arxiv.org/abs/2602.14878)
**Debenedetti et al., AgentDojo** NeurIPS 2024 Datasets & Benchmarks; [https://arxiv.org/abs/2406.13352](https://arxiv.org/abs/2406.13352) Tool-using LLM agents Legitimate tasks under indirect prompt injection Adversarial instructions embedded in untrusted tool data Benign environment / attack and defense variants Demonstrates significant utility and security failures Sandbox rather than live enterprise; nevertheless directly models the key data-as-instruction vulnerability. [2](https://arxiv.org/abs/2406.13352)
**Zhan et al., InjecAgent** arXiv 2024; [https://arxiv.org/abs/2403.02691](https://arxiv.org/abs/2403.02691) 30 LLM-agent configurations Indirect prompt injection through external tool/content channels Adversarial retrieved content Benign agent runs 1,054 cases, 17 user tools, 62 attacker tools; ReAct GPT-4 ≈24% vulnerable under reported setup Attack construction and older models limit exact current rates; mechanism transfers strongly. [26](https://arxiv.org/abs/2403.02691)
**Debenedetti et al., CaMeL / Defeating Prompt Injections by Design** IEEE SaTML 2026; [https://arxiv.org/abs/2503.18813](https://arxiv.org/abs/2503.18813) Capability/data-flow constrained agent system Prompt-injection-safe tool use Trusted control/data separation and capability policies Standard LLM-agent execution ≈67% AgentDojo utility while providing its stronger security guarantees Utility cost and integration complexity; assumes trusted enforcement layer. [28](https://arxiv.org/abs/2503.18813)
**NetInjectBench** arXiv preprint 2026; [https://arxiv.org/abs/2607.10490](https://arxiv.org/abs/2607.10490) Qwen/Llama/Mistral network agents Network-configuration attacks and authorized changes Prompt defenses vs metadata-aware policy gate Naive agent 240 attacks; naive unsafe 82.5%; policy gate 0/240 under trusted-metadata assumption Narrow domain/small models; unusually clean evidence for external policy enforcement. [29](https://arxiv.org/abs/2607.10490)
The publication-status distinction matters. BIRD, Dr.Spider, DIN-SQL, ToG, ALCE, AgentDojo and CaMeL have stronger publication vetting than the newest 2026 preprints. The 2026 work is useful for mapping the frontier but should not be treated as settled consensus merely because its numbers are recent. ## Research gaps and research frontier ### Most important research gaps **Enterprise semantic-layer trial.**
A serious evaluation should select several real company schemas and freeze the model and questions while varying only the representation: `raw DDL` → `DDL + descriptions` → `retrieved catalog metadata` → `semantic entities/metrics` → `governed metric API` → `verified-query examples`. It should measure execution correctness, *business-semantic* correctness, source selection, abstention, token cost and latency. Without this trial, “semantic layer for AI” remains mostly architecture-led reasoning. **Business-semantic benchmark.**
Current text-to-SQL benchmarks usually have one canonical interpretation. Real businesses often have several technically valid queries but one organizationally correct definition. A useful benchmark would include conflicting plausible definitions of customer, revenue, churn, headcount and attribution, along with effective dates and domain ownership. The central metric should be whether the model chooses the **governed meaning**, not merely executable SQL. **Quality-degradation experiment.**
Take clean enterprise-style datasets and independently vary missingness, wrong values, duplicate rates, inconsistent identifiers, delayed partitions and schema drift. Compare no quality information against machine-readable quality metadata and enforced validation. Measure wrong-answer rate, anomaly detection, abstention and confidence calibration. This would directly answer whether agents benefit from surfaced quality indicators instead of merely inheriting ordinary garbage-in/garbage-out behavior. **Freshness-metadata trial.**
Create parallel historical/current table versions and documents with contradictory values. Compare no timestamps, timestamps, validity windows, explicit supersession links and freshness/SLA metadata. Measure whether agents choose the correct current source. FreshQA and STALE establish the problem but not the optimal data representation. [17](https://arxiv.org/abs/2310.03214) **Lineage-consumption benchmark.**
Build realistic multi-hop transformation DAGs with duplicated sources, intermediate models, metric transformations and injected upstream errors. Test impact analysis, source selection and root-cause diagnosis with no lineage, textual lineage descriptions and graph-structured lineage. Until that exists, “agents need lineage” remains speculative. **Data-contract experiment.**
Inject schema drift, unit changes, nullability changes and renamed fields. Compare: no contract, prose documentation, visible machine contract, and contract enforcement before agent access. This would separate “a contract helps the agent reason” from the much simpler “a contract prevents broken data from entering the system.” **Graph-versus-text semantics experiment.**
Encode **identical facts** as prose, relational metadata, triples/ontology and a graph API, with equivalent retrieval recall. Current KG and GraphRAG work often improves the retrieval mechanism together with the representation, making the incremental causal value of graph semantics difficult to estimate. [32](https://arxiv.org/abs/2307.07697) **Controlled tool-abstraction experiment.**
Provide exactly the same underlying capabilities through low-level SQL/CRUD tools and high-level domain functions. Vary number of tools and overlapping scopes. Measure tool-selection accuracy, parameter errors, policy violations and end-to-end success. This would test the common claim that agent-facing platforms need higher-level domain APIs. **Scale curves rather than single benchmarks.**
Systematically scale tables, columns, metrics, tools, documents and domains while keeping task difficulty fixed. SQL-PaLM and Spider 2.0 establish degradation, but the field lacks reusable scaling curves across model generations. [1](https://arxiv.org/html/2306.00739v4) **Authorization benchmark with real permission structure.**
Enterprise evaluations should include user identity, role changes, row-level security, column-level restrictions, delegated authority, cross-system scopes, read-versus-write privileges and indirect prompt injection. Current security benchmarks demonstrate the vulnerability very convincingly, but not the full complexity of enterprise IAM. [2](https://arxiv.org/abs/2406.13352) **Human trust-calibration study for provenance.**
Independently manipulate answer correctness, citation correctness and citation presence, then measure whether users detect errors and calibrate reliance. ALCE establishes that citations can be objectively evaluated, but not whether they improve human decision quality. [20](https://aclanthology.org/2023.emnlp-main.398/) **Longitudinal production studies.**
The field badly needs real before/after deployment data: agent incident rates, semantic errors, permission violations and successful task completion before and after metadata/semantic/governance changes. Architecture case studies without a counterfactual cannot establish which layer caused the improvement. ### Research frontier The frontier is uneven. **Independent evidence is moving relatively fast** in four areas: large-schema grounding, retrieval/context selection, tool-interface quality, and prompt-injection/authorization security. SQL-PaLM, Spider 2.0, τ-bench, AgentDojo and the MCP-description study all expose measurable failure mechanisms and provide at least some controlled comparisons. [1](https://arxiv.org/html/2306.00739v4) **Practitioner architecture is moving substantially faster than independent evidence** around enterprise semantic layers for agents, “context layers,” AI-ready data products, governed-metric APIs as an AI prerequisite, agent use of lineage, agent-oriented data contracts, and wholesale machine-oriented redesign of catalogs. Vendor definitions of AI-ready and agent-ready data are already broad and prescriptive, while the research literature generally evaluates the individual mechanisms rather than those architecture bundles. [4](https://datahub.com/blog/ai-ready-data/) **MCP is another case where adoption is ahead of causal evidence.** MCP provides a useful common connectivity protocol, but the empirical work so far is already showing that the quality of the descriptions carried through that interface can materially change agent behavior. Protocol standardization therefore removes integration friction without eliminating the underlying intelligence, semantic or security problems. [11](https://modelcontextprotocol.io/docs/2026-07-28/getting-started/intro) The deeper pattern is that most experimentally demonstrated requirements do not demand a wholly new kind of enterprise data architecture. They demand that existing information be made **selectively retrievable, unambiguous enough for the task, temporally grounded, and enforceably governed**, while recognizing that language models can confuse irrelevant context, stale evidence and hostile instructions with useful context. [1](https://arxiv.org/html/2306.00739v4) The one important architectural discontinuity is **agency**. When a model only answers a question, bad data mainly threatens correctness. When a model can call tools, modify systems, send information or spend authority, untrusted data can alter the model's control decisions. That turns the traditional “data quality” problem into a **control-plane security problem**, for which external authorization, capability boundaries and trusted policy enforcement have substantially stronger empirical justification than simply adding another metadata or semantic layer. [2](https://arxiv.org/abs/2406.13352) The current evidence therefore supports a narrower—and less commercially convenient—definition of what makes data usable by agents: **Agents benefit from accurate and task-relevant schemas, values and semantics; they need mechanisms to retrieve rather than indiscriminately ingest large environments; they need access to current information when facts change; tool interfaces and their descriptions materially affect performance; and action-taking agents require security boundaries that prevent untrusted data from becoming authority.** It does **not** follow from the evidence that organizations require a semantic layer, knowledge graph, data contracts, lineage system, data-product operating model, MCP, or fundamentally new stack as universal prerequisites. Those components may be good engineering choices in particular environments. As of October 3, 2026, treating them as empirically established constituents of “agent-ready data” would run ahead of the research.