# AI-Assisted Data Engineering: From Copilot to Autonomous Changes

**Research snapshot:** 2026-10-03<br>
**Research question:** Which parts of the data-engineering lifecycle can AI reliably accelerate or automate, and where must deterministic controls, tests, contracts, and human review remain in control?

This dossier follows the scope and evidence rules in the supplied research brief: it treats assistance, candidate generation, evaluation, repository mutation, production mutation, and incident repair as distinct capability classes; prioritizes empirical and primary evidence; and does not infer production reliability from code-generation ability alone. [1](https://chatgpt.com/g/g-p-6ac129ef43c08191acf988f750b7a6da/c/6ac142a2-b758-83eb-8180-242aa6856e09)

## Executive synthesis and evidence frame

### Bottom line

The evidence supports a **large useful zone for AI-assisted data engineering, a narrower zone for autonomous repository work, and little credible evidence for generally safe autonomous mutation of production data systems**.

The strongest current use cases are not those where the model is trusted to be correct. They are those where the model can cheaply generate, search, explain, transform, or iterate **inside an environment whose correctness is determined elsewhere**: compilation, execution against non-production data, fixed tests, independently specified invariants, schema contracts, reconciliation queries, cost limits, permissions, branch rules, and human review. Recent text-to-SQL research strengthens this conclusion: execution-grounded methods materially outperform unaided generation, yet still fall far below a level that would justify assuming semantic correctness on enterprise workloads. VET reached 37.04% execution accuracy on Spider 2.0-lite; Spider 2.0 itself reports very low success on much more realistic project-level enterprise workflows. [2](https://aclanthology.org/2026.findings-acl.1544/)

The distinction that matters is therefore not **“copilot versus agent.”** It is:

> **Who decides what correct means, who can mutate persistent state, and what independent mechanism can stop or reverse a wrong change?**

That produces five substantive findings.

**First, generation is substantially ahead of verification.** Models can frequently produce plausible SQL, Python, tests, documentation, and repository changes, but real enterprise data work remains difficult. Spider 2.0 includes 632 enterprise-level workflow problems, databases often containing more than 1,000 columns, multiple SQL dialects, project codebases, metadata, and documentation; its abstract reports only 17.0% task success for an o1-preview-based code-agent framework, compared with much higher performance on older SQL benchmarks. [3](https://arxiv.org/abs/2411.07763) Enterprise debugging is harder still: OurBench's best evaluated model solved 36.46% of syntax-error cases and 32.17% of semantic-error cases on long enterprise ETL SQL, with most models below 20%. [4](https://arxiv.org/abs/2601.18119)

**Second, passing checks is evidence, not proof.** In generic software engineering, METR found roughly half of AI-written patches that passed SWE-bench Verified's automated grader would still not have been accepted by maintainers. A September 2026 formal-verification benchmark found counterexamples in roughly one-quarter to one-half of test-passing patches for the two frontier models tested. [5](https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/?utm_source=chatgpt.com) This is especially consequential in data engineering because code can execute successfully while producing a plausible but semantically wrong dataset.

**Third, AI-generated tests are not automatically an independent evaluator.** A 2026 experiment found that when an LLM sees buggy code, the code can “misguide” test generation: the model becomes more likely to write tests that encode the buggy behavior and less likely to generate tests that expose it. Specification-based test generation mitigated the effect. [6](https://arxiv.org/abs/2607.22883?utm_source=chatgpt.com) The relevant control is therefore not “AI generated code plus AI generated tests,” but **independent specification plus independently executable evidence**.

**Fourth, context is essential but non-monotonic.** Enterprise SQL increasingly requires schemas, repository structure, metadata, lineage, dialect information, and business semantics. Spider 2.0 explicitly demonstrates this dependence. [3](https://arxiv.org/abs/2411.07763) Yet “give the model everything” is not a reliable strategy: in SWE-PRBench, all eight tested frontier models performed worse as code-review context expanded from a diff to fuller repository context, despite structured context enrichment. [7](https://arxiv.org/abs/2603.26130) The emerging design problem is **context selection and authority**, not context-window size alone.

**Fifth, the boundary around production mutation is visible in real product architectures.** Google's current BigQuery Data Engineering Agent can generate and modify Dataform pipeline code, inspect metadata and sample data, and validate compilation, but Google explicitly states that the agent cannot execute pipelines; a user must review and run or schedule them. [8](https://docs.cloud.google.com/gemini/data-agents/data-engineering-agent/agent-overview?utm_source=chatgpt.com) GitHub's cloud coding agent similarly constrains writes to a branch, applies repository protections and required checks, and under its documented cloud-agent workflow cannot approve or merge its own PR. [9](https://docs.github.com/en/copilot/concepts/security-governance-and-network-settings/risks-and-mitigations?utm_source=chatgpt.com) Datadog's database investigation agent can gather evidence and propose SQL changes but deliberately does not connect to the database or execute that SQL. [10](https://docs.datadoghq.com/database_monitoring/database_investigator/?utm_source=chatgpt.com) These boundaries are documented product capabilities rather than independent reliability studies, but their design choices are informative: sophisticated providers are themselves separating **agentic reasoning from unrestricted production authority**.

### Evidence hierarchy used

The evidence base is uneven, so statements below distinguish:

<table header-row="true">
	<tr>
		<td>Evidence class</td>
		<td>Meaning</td>
		<td>Weight in this dossier</td>
	</tr>
	<tr>
		<td>**Production evidence**</td>
		<td>Measured deployment, maintainer review, or operational evaluation on real systems</td>
		<td>Highest for demonstrated operational behavior, subject to sample/generalizability limits</td>
	</tr>
	<tr>
		<td>**Controlled empirical evidence**</td>
		<td>Experimental study with explicit comparisons or interventions</td>
		<td>High when task resembles the target use case</td>
	</tr>
	<tr>
		<td>**Benchmark**</td>
		<td>Standardized model/task evaluation</td>
		<td>Useful for capability boundaries; weak evidence of production safety</td>
	</tr>
	<tr>
		<td>**Documented capability**</td>
		<td>Current primary product/platform documentation</td>
		<td>Reliable for what a system is designed or permitted to do, not for claimed productivity/reliability</td>
	</tr>
	<tr>
		<td>**Provider claim**</td>
		<td>Vendor study, case study, or marketing result</td>
		<td>Establishes a reported result, not independent evidence</td>
	</tr>
	<tr>
		<td>**Interpretation**</td>
		<td>Synthesis from multiple sources</td>
		<td>Explicitly treated as inference</td>
	</tr>
	<tr>
		<td>**Hypothesis**</td>
		<td>Important proposition for which direct evidence is missing</td>
		<td>Not treated as established fact</td>
	</tr>

</table>

A recurring limitation is that **direct empirical evidence specifically measuring end-to-end AI data-engineering changes in production is still sparse**. The strongest data-specific studies are SQL generation/debugging benchmarks; the strongest evidence about coding agents, review, tests, and autonomous workflows often comes from generic software engineering. That transfer is useful but must remain conditional because the failure surface is different.

## Taxonomy, lifecycle map, and autonomy spectrum

### AI-assisted data-engineering taxonomy

The labels currently used in products collapse materially different authority levels.

<table header-row="true">
	<tr>
		<td>Role</td>
		<td>What the AI actually does</td>
		<td>Representative data-engineering tasks</td>
		<td>Main source of value</td>
		<td>Main risk</td>
	</tr>
	<tr>
		<td>**Autocomplete**</td>
		<td>Predicts local code/text</td>
		<td>SQL expressions, Python, YAML</td>
		<td>Keystrokes</td>
		<td>Locally plausible error</td>
	</tr>
	<tr>
		<td>**Coding assistant**</td>
		<td>Generates or edits requested artifacts</td>
		<td>SQL, dbt, Python, orchestration</td>
		<td>Faster production/refactoring</td>
		<td>Semantic mismatch</td>
	</tr>
	<tr>
		<td>**Analytical assistant**</td>
		<td>Queries/explains available information</td>
		<td>Schema exploration, query explanation, lineage</td>
		<td>Faster comprehension</td>
		<td>Incorrect interpretation</td>
	</tr>
	<tr>
		<td>**Documentation assistant**</td>
		<td>Converts structural context into prose</td>
		<td>model/column docs, lineage summaries</td>
		<td>Removes documentation toil</td>
		<td>Invented business meaning</td>
	</tr>
	<tr>
		<td>**Reviewer**</td>
		<td>Critiques an existing artifact</td>
		<td>SQL logic, joins, code style</td>
		<td>Additional review coverage</td>
		<td>False negative / false confidence</td>
	</tr>
	<tr>
		<td>**Debugging assistant**</td>
		<td>Collects evidence and proposes causes/fixes</td>
		<td>failed pipeline, query error</td>
		<td>Faster search/hypothesis generation</td>
		<td>Plausible but false diagnosis</td>
	</tr>
	<tr>
		<td>**Migration assistant**</td>
		<td>Designs/translates transformation</td>
		<td>schema migration, SQL dialect migration</td>
		<td>Mechanical transformation</td>
		<td>Incomplete compatibility analysis</td>
	</tr>
	<tr>
		<td>**Incident assistant**</td>
		<td>Correlates telemetry and proposes remediation</td>
		<td>logs, failed jobs, lineage</td>
		<td>Reduced investigative toil</td>
		<td>Acting on uncertain RCA</td>
	</tr>
	<tr>
		<td>**Workflow agent**</td>
		<td>Uses tools and iterates against feedback</td>
		<td>edit → compile → test → repair</td>
		<td>Automates closed-loop mechanical work</td>
		<td>Evaluator becomes the bottleneck</td>
	</tr>
	<tr>
		<td>**Autonomous engineering agent**</td>
		<td>Persists changes without per-step human action</td>
		<td>PR iteration, conditional merge</td>
		<td>Removes workflow latency</td>
		<td>Scope expansion / weak gate</td>
	</tr>
	<tr>
		<td>**Production operator**</td>
		<td>Changes running system or durable data</td>
		<td>DDL, backfill, rollback, repair</td>
		<td>Potentially large operational leverage</td>
		<td>Irreversible or silent corruption</td>
	</tr>

</table>

The last three categories are qualitatively different from ordinary code generation because they add **persistence, feedback loops, and authority**.

### Autonomy spectrum

The requested eleven levels are useful as a description of authority, but they should **not** be interpreted as a maturity ladder: moving upward is not intrinsically desirable. A level appropriate for formatting a dbt YAML file may be inappropriate for a billion-row backfill.

<table header-row="true">
	<tr>
		<td>Level</td>
		<td>Capability</td>
		<td>Qualitative change in risk</td>
	</tr>
	<tr>
		<td>**1**</td>
		<td>Autocomplete</td>
		<td>No independent action</td>
	</tr>
	<tr>
		<td>**2**</td>
		<td>Conversational copilot</td>
		<td>Advice may influence human judgment</td>
	</tr>
	<tr>
		<td>**3**</td>
		<td>Generate code on request</td>
		<td>Produces an executable candidate</td>
	</tr>
	<tr>
		<td>**4**</td>
		<td>Generate code plus tests</td>
		<td>Generator begins producing its own evidence</td>
	</tr>
	<tr>
		<td>**5**</td>
		<td>Propose repository changes</td>
		<td>**Persistent artifact boundary**</td>
	</tr>
	<tr>
		<td>**6**</td>
		<td>Open a pull request</td>
		<td>Change enters institutional workflow</td>
	</tr>
	<tr>
		<td>**7**</td>
		<td>Review/repair failed CI</td>
		<td>Model consumes evaluator feedback</td>
	</tr>
	<tr>
		<td>**8**</td>
		<td>Iterate autonomously until checks pass</td>
		<td>**Closed-loop optimization boundary**</td>
	</tr>
	<tr>
		<td>**9**</td>
		<td>Merge under defined conditions</td>
		<td>**Human review / integration boundary**</td>
	</tr>
	<tr>
		<td>**10**</td>
		<td>Modify production systems or data</td>
		<td>**Production-state boundary**</td>
	</tr>
	<tr>
		<td>**11**</td>
		<td>Diagnose and repair production incidents</td>
		<td>**Production-state + epistemic-uncertainty boundary**</td>
	</tr>

</table>

```

```

The critical boundaries are **4→5, 6→7/8, 8→9, 9→10, and 10→11**. GitHub's `/pr auto` illustrates the distinction between levels 8 and 9 particularly cleanly: Copilot can iterate over review feedback, conflicts, and failing CI until the PR is green, but `/pr auto` itself does not merge. Its separate auto-merge behavior delegates the eventual merge to GitHub only after configured requirements are satisfied. [11](https://docs.github.com/en/copilot/how-tos/copilot-cli/use-copilot-cli/manage-pull-requests?utm_source=chatgpt.com)

### Lifecycle map

The following “autonomy ceiling” is an evidence-based synthesis, not a statement of what APIs technically permit.

<table header-row="true">
	<tr>
		<td>Activity</td>
		<td>Current AI capability</td>
		<td>Evidence quality</td>
		<td>Typical failure</td>
		<td>Justified autonomy today</td>
		<td>Necessary context / controls</td>
	</tr>
	<tr>
		<td>Requirements understanding</td>
		<td>Summarize, question, draft acceptance criteria</td>
		<td>Indirect / generic</td>
		<td>Missing business assumption</td>
		<td>**1–3**</td>
		<td>Human business owner; explicit definitions</td>
	</tr>
	<tr>
		<td>Explore models / unfamiliar repo</td>
		<td>Strong search, explanation, dependency tracing when context is retrievable</td>
		<td>Benchmark + documented capability</td>
		<td>Missed dependency; stale context</td>
		<td>**1–4**</td>
		<td>Code, manifest, lineage, ownership</td>
	</tr>
	<tr>
		<td>Generate SQL</td>
		<td>Strong on bounded/common tasks; sharply weaker on complex enterprise workflows</td>
		<td>Strong data-specific benchmarks</td>
		<td>Wrong table, predicate, join, grain</td>
		<td>**3–6**, **7–8 only sandboxed**</td>
		<td>Schema, samples, semantics, execution, reconciliation [3](https://arxiv.org/abs/2411.07763)</td>
	</tr>
	<tr>
		<td>Refactor SQL</td>
		<td>Often mechanically useful</td>
		<td>Indirect benchmark evidence</td>
		<td>Semantics changed despite equivalent-looking code</td>
		<td>**3–6**</td>
		<td>Differential results, query plans, regression data</td>
	</tr>
	<tr>
		<td>Generate dbt models</td>
		<td>Productized; project-context-aware agents exist</td>
		<td>Documented capability + Spider project tasks</td>
		<td>Wrong `ref`, materialization, grain, incremental behavior</td>
		<td>**3–6**</td>
		<td>Manifest, DAG, tests, model contracts, sample data [3](https://arxiv.org/abs/2411.07763)</td>
	</tr>
	<tr>
		<td>Generate Python</td>
		<td>Broad coding capability</td>
		<td>Generic SE evidence</td>
		<td>Edge cases, dependency/API errors</td>
		<td>**3–7**</td>
		<td>Repo, tests, type checks, sandbox</td>
	</tr>
	<tr>
		<td>Generate orchestration/config</td>
		<td>Good at templates and local changes</td>
		<td>Generic / documented</td>
		<td>Wrong schedule, retry, dependencies, secrets</td>
		<td>**3–6**</td>
		<td>DAG, environment config, policy</td>
	</tr>
	<tr>
		<td>Define schemas</td>
		<td>Can infer likely types/shape</td>
		<td>Documented capability</td>
		<td>Conflates observed samples with intended contract</td>
		<td>**2–5**</td>
		<td>Producer contract, consumer requirements</td>
	</tr>
	<tr>
		<td>Write tests / quality checks</td>
		<td>Easy to generate; quality of oracle variable</td>
		<td>Controlled empirical</td>
		<td>Tests reproduce implementation's wrong assumption</td>
		<td>**3–5**</td>
		<td>Independent specification, historical failures, mutation/adversarial tests [6](https://arxiv.org/abs/2607.22883?utm_source=chatgpt.com)</td>
	</tr>
	<tr>
		<td>Write contracts</td>
		<td>Can draft structure</td>
		<td>Indirect</td>
		<td>Encodes current implementation rather than intended interface</td>
		<td>**2–5**</td>
		<td>Human-owned interface semantics</td>
	</tr>
	<tr>
		<td>Documentation</td>
		<td>Strong for structural summaries</td>
		<td>Documented + limited empirical</td>
		<td>Invented business semantics</td>
		<td>**2–5**</td>
		<td>Catalog + validated semantic definitions</td>
	</tr>
	<tr>
		<td>Lineage explanation</td>
		<td>Strong if machine lineage is supplied</td>
		<td>Documented capability</td>
		<td>Treats inferred lineage as authoritative</td>
		<td>**2–4**</td>
		<td>Manifest/catalog/runtime lineage</td>
	</tr>
	<tr>
		<td>Code review</td>
		<td>Useful second reviewer; incomplete defect detection</td>
		<td>Controlled benchmark</td>
		<td>Misses contextual defects</td>
		<td>**2–4 as reviewer**, not sole gate</td>
		<td>Diff-first context, tests, specialist human [7](https://arxiv.org/abs/2603.26130)</td>
	</tr>
	<tr>
		<td>Performance optimization</td>
		<td>Can suggest rewrites/indexing/partitioning</td>
		<td>Benchmark + documented tools</td>
		<td>Correct but more expensive query, altered semantics</td>
		<td>**2–6**</td>
		<td>Explain plans, historical query telemetry, cost cap</td>
	</tr>
	<tr>
		<td>Warehouse-cost optimization</td>
		<td>Good candidate generation; cost must be measured</td>
		<td>Documented platform controls</td>
		<td>Massive scan / repeated agent probes</td>
		<td>**2–6**</td>
		<td>Dry run, quotas, max bytes/credits</td>
	</tr>
	<tr>
		<td>Schema evolution</td>
		<td>Can draft DDL/migration</td>
		<td>Documented capability; weak AI-specific empirical evidence</td>
		<td>Breaking downstream compatibility</td>
		<td>**2–6 proposal only**</td>
		<td>Dependency graph, contracts, versioning, staging</td>
	</tr>
	<tr>
		<td>Backfills</td>
		<td>Can draft plan/query</td>
		<td>Very weak autonomous evidence</td>
		<td>Wrong period, duplicate/rewrite history, extreme cost</td>
		<td>**2–6 proposal only**</td>
		<td>Immutable source snapshot, partition bounds, reconciliation, rollback</td>
	</tr>
	<tr>
		<td>Migration</td>
		<td>Strong at syntax translation; semantic migration remains conditional</td>
		<td>Benchmark/documented</td>
		<td>Partial compatibility, hidden dialect differences</td>
		<td>**3–6**</td>
		<td>Golden outputs, dual-run, downstream tests</td>
	</tr>
	<tr>
		<td>Incremental-model design</td>
		<td>Can draft patterns</td>
		<td>Data-specific complexity evidence; little autonomy evidence</td>
		<td>Watermark, late data, idempotency errors</td>
		<td>**2–6**</td>
		<td>Replay tests, historical windows, state model</td>
	</tr>
	<tr>
		<td>Debugging</td>
		<td>Good evidence search/hypothesis generation; weak on hard semantic ETL bugs</td>
		<td>Data-specific benchmark</td>
		<td>Plausible wrong fix</td>
		<td>**2–7 read-only/sandbox**</td>
		<td>Error, code, data sample, lineage, job history [4](https://arxiv.org/abs/2601.18119)</td>
	</tr>
	<tr>
		<td>Incident triage</td>
		<td>Promising for evidence compression and ranked hypotheses</td>
		<td>Some production evidence outside DE</td>
		<td>Premature RCA</td>
		<td>**2–6; read-only tools preferred**</td>
		<td>Logs, metrics, traces, lineage, runbooks</td>
	</tr>
	<tr>
		<td>Root-cause confirmation</td>
		<td>AI can assist; independent evidence still necessary</td>
		<td>Limited production evidence</td>
		<td>Confuses correlation with cause</td>
		<td>**2–5**</td>
		<td>Reproduction, counterfactual evidence</td>
	</tr>
	<tr>
		<td>Pipeline repair</td>
		<td>Agent can prepare patch and iterate in CI</td>
		<td>Generic SE + documented capability</td>
		<td>Green CI but wrong data</td>
		<td>**5–8 pre-production**</td>
		<td>Independent data checks and human review</td>
	</tr>
	<tr>
		<td>Dependency upgrades</td>
		<td>Well suited to branch/CI iteration</td>
		<td>Generic agent evidence</td>
		<td>API behavior/regression not tested</td>
		<td>**5–8**, occasionally **9** for low-risk policies</td>
		<td>Lockfiles, full CI, canary</td>
	</tr>
	<tr>
		<td>CI/CD</td>
		<td>Agents can execute/repair checks in controlled workflows</td>
		<td>Strong documented capability</td>
		<td>Gaming incomplete checks</td>
		<td>**7–8**</td>
		<td>Protected checks, immutable policies</td>
	</tr>
	<tr>
		<td>Merge</td>
		<td>Mechanically automatable after policy gates</td>
		<td>Documented capability</td>
		<td>Gate set fails to capture semantics</td>
		<td>**9 only for bounded classes**</td>
		<td>Required reviews/checks; path/risk policy</td>
	</tr>
	<tr>
		<td>Deployment</td>
		<td>Technically automatable; little evidence AI should decide deployment safety</td>
		<td>Weak</td>
		<td>Blast radius</td>
		<td>**Deterministic deployment, AI advisory**</td>
		<td>staged rollout, approvals, canary</td>
	</tr>
	<tr>
		<td>Rollback</td>
		<td>AI can recommend; deterministic mechanisms are preferable</td>
		<td>Platform capability</td>
		<td>Rollback incomplete or itself destructive</td>
		<td>**Policy automation &gt; model discretion**</td>
		<td>known-good version, retained history</td>
	</tr>
	<tr>
		<td>Monitoring</td>
		<td>Strong summarization/anomaly explanation</td>
		<td>Operational/product evidence</td>
		<td>Alert narrative mistaken for diagnosis</td>
		<td>**1–5**</td>
		<td>deterministic telemetry and alert thresholds</td>
	</tr>

</table>

The Google BigQuery Data Engineering Agent is a useful real-world calibration point. It generates and edits Dataform code and automatically attempts to fix compilation errors, but its documentation explicitly says it **cannot execute pipelines**; users must review and run or schedule them. [8](https://docs.cloud.google.com/gemini/data-agents/data-engineering-agent/agent-overview?utm_source=chatgpt.com) That corresponds roughly to levels 5–8 depending on workflow—not level 10—despite the “Data Engineering Agent” label.

## Why data engineering is different, and what context AI needs

### The central reliability problem is semantic data state

Generic software failures often manifest as an exception, failed assertion, broken API, or obvious UI behavior. Data-engineering failures frequently have a more dangerous outcome: **the job succeeds and the output is wrong**.

Spider 2.0 demonstrates why apparently strong SQL-generation results do not transfer cleanly to enterprise transformations: its workloads have large schemas, multiple dialects, external documentation, project context, long SQL, nested structures, and multistep interaction. [3](https://arxiv.org/abs/2411.07763) SchemaScope adds a narrower finding: text-to-SQL performance deteriorates sharply as the required join path becomes deeper, with wrong join-path selection becoming a dominant failure on harder cases. [12](https://aclanthology.org/2026.surgellm-1.17/)

That matters because several data-engineering invariants exist outside syntax:

**Grain.** A query can use real columns, compile, and return plausible rows while representing the wrong entity grain.

**Join cardinality.** A valid join can fan out rows and silently inflate sums, counts, or attribution.

**Temporal state.** Incremental transforms depend on what was processed previously, not only on the code currently visible.

**History.** A backfill can retroactively alter metrics already consumed by reports, models, billing, or audit processes.

**Late arrival and replay.** Watermarks and event-time logic have behaviors that are hard to infer from the happy path.

**Idempotency.** Running the same transformation twice may or may not be safe.

**Schema evolution.** A locally legitimate change can break remote downstream consumers.

**Cost.** A semantically correct generated query can still scan an unacceptable amount of data.

**Observability.** Pipeline success says little about whether its output distribution, completeness, or business meaning remains correct.

These are not merely theoretical concerns. Current data platforms explicitly expose special mechanisms because data state is consequential. BigQuery supports query dry runs to inspect processing estimates before actual execution and supports controls such as maximum bytes billed. [13](https://docs.cloud.google.com/bigquery/docs/running-queries) Delta Lake records table versions and exposes table history/time travel and restoration mechanisms; Databricks also warns that schema updates can conflict with concurrent writes and documents special handling for streaming/schema changes. [14](https://docs.databricks.com/aws/en/tables/history) Those mechanisms are valuable regardless of whether a change was written by a human or AI.

### Correctness has multiple thresholds

AI-generated code should be classified at least as:

<table header-row="true">
	<tr>
		<td>Threshold</td>
		<td>Question</td>
		<td>What it does **not** establish</td>
	</tr>
	<tr>
		<td>Syntactically valid</td>
		<td>Does it parse/compile?</td>
		<td>That it executes</td>
	</tr>
	<tr>
		<td>Executable</td>
		<td>Does it run?</td>
		<td>That the result is correct</td>
	</tr>
	<tr>
		<td>Test-passing</td>
		<td>Does it satisfy available tests?</td>
		<td>That tests capture the specification</td>
	</tr>
	<tr>
		<td>Semantically correct</td>
		<td>Does output mean what the requirement intended?</td>
		<td>That operational impact is acceptable</td>
	</tr>
	<tr>
		<td>Production-safe</td>
		<td>Is semantic correctness accompanied by bounded cost, compatibility, reversibility, governance, and acceptable blast radius?</td>
		<td>Nothing stronger should be inferred automatically</td>
	</tr>

</table>

VET and PV-SQL provide positive evidence for execution-grounded verification: VET executes intermediate reasoning against the real database, while PV-SQL probes actual database values and applies an executable rule-based checklist; both improve results versus less grounded approaches. [2](https://aclanthology.org/2026.findings-acl.1544/) But VET's 37.04% execution accuracy on Spider 2.0-lite is simultaneously evidence that **verification helps** and evidence that **verification does not make the problem solved**. [2](https://aclanthology.org/2026.findings-acl.1544/)

### Context requirements

For real data-engineering work, the useful context envelope includes at least:

<table header-row="true">
	<tr>
		<td>Context</td>
		<td>What it resolves</td>
		<td>Failure if absent or stale</td>
	</tr>
	<tr>
		<td>Repository code</td>
		<td>local conventions and implementation</td>
		<td>incompatible/generated duplicate logic</td>
	</tr>
	<tr>
		<td>SQL/dbt models</td>
		<td>transformation definitions</td>
		<td>wrong dependency/logic</td>
	</tr>
	<tr>
		<td>Physical schemas</td>
		<td>names/types</td>
		<td>hallucinated or mistyped field</td>
	</tr>
	<tr>
		<td>Samples/profiles</td>
		<td>real values, nullability, shape</td>
		<td>incorrect assumptions about domains</td>
	</tr>
	<tr>
		<td>dbt manifest / DAG</td>
		<td>dependencies</td>
		<td>incomplete impact analysis</td>
	</tr>
	<tr>
		<td>Runtime lineage</td>
		<td>actual cross-system flows</td>
		<td>missed consumer</td>
	</tr>
	<tr>
		<td>Semantic definitions</td>
		<td>metric/business meaning</td>
		<td>semantically valid but wrong calculation</td>
	</tr>
	<tr>
		<td>Data contracts</td>
		<td>producer/consumer promises</td>
		<td>breaking interface</td>
	</tr>
	<tr>
		<td>Tests</td>
		<td>known invariants</td>
		<td>regressions</td>
	</tr>
	<tr>
		<td>Orchestration</td>
		<td>scheduling/retry/order</td>
		<td>temporal bugs</td>
	</tr>
	<tr>
		<td>Historical incidents</td>
		<td>recurring edge cases</td>
		<td>repeated known failure</td>
	</tr>
	<tr>
		<td>Warehouse metadata</td>
		<td>partitions/statistics</td>
		<td>bad physical query</td>
	</tr>
	<tr>
		<td>Query history</td>
		<td>workload/cost behavior</td>
		<td>cost/performance regression</td>
	</tr>
	<tr>
		<td>Ownership</td>
		<td>escalation/review authority</td>
		<td>wrong approval path</td>
	</tr>
	<tr>
		<td>Environment/deployment config</td>
		<td>execution semantics</td>
		<td>dev/prod mismatch</td>
	</tr>
	<tr>
		<td>Business definitions</td>
		<td>intended meaning</td>
		<td>confident invention</td>
	</tr>

</table>

Machine-readable metadata can already provide substantial parts of this envelope. dbt Catalog, for example, exposes project metadata, lineage, tests, and applied-state information; model contracts can constrain output shape. [15](https://docs.getdbt.com/guides/mesh-qs) Google says its BigQuery Data Engineering Agent can retrieve BigQuery and catalog metadata, sample rows, and profiling information as context. [8](https://docs.cloud.google.com/gemini/data-agents/data-engineering-agent/agent-overview?utm_source=chatgpt.com) These are documented capabilities, not evidence that context eliminates model errors.

**More context does not consistently produce better outcomes.** A 2026 data-analytics preprint reported a large improvement when compact hand-authored business semantics were added to schema context across three frontier models, indicating that authoritative semantic context can be highly valuable. [16](https://arxiv.org/abs/2604.25149) In the opposite direction, SWE-PRBench found monotonically degrading code-review performance as additional repository context was supplied to all eight tested models. [7](https://arxiv.org/abs/2603.26130)

The best-supported interpretation is therefore:

> **Context quality, authority, relevance, freshness, and retrieval strategy matter more than raw context quantity.**

A practical authority order—not established as a universal empirical ranking, but supported by the observed failure modes—is:

**executable facts and machine metadata → explicit contracts/specifications → curated business semantics → repository/runtime evidence → historical prose → inferred model explanations.**

Stale context is particularly dangerous because it creates the appearance of grounding without actual alignment to current system state. Contradictory context introduces another problem: unless the system knows which source is authoritative, retrieval merely moves uncertainty from the model's parametric memory into its prompt.

## Evidence by task class: generation, evaluation, documentation, change, and operations

### Code generation and autonomous iteration

There is strong evidence that code generation can reduce the cost of producing candidate artifacts, but weak evidence that generation alone predicts production correctness.

Older or simplified SQL benchmarks can give a misleading impression. Spider 2.0 was explicitly designed because previous text-to-SQL benchmarks underrepresented enterprise schema scale, dialect variation, repository context, and multistep workflows. Its 632 tasks include data transformation and project-level work; the benchmark paper reports dramatically lower agentic performance than on Spider 1.0 or BIRD. [3](https://arxiv.org/abs/2411.07763)

Recent techniques make real progress by introducing **external verification loops rather than trusting a single generation**. VET uses executable intermediate steps against the actual database and achieves 70.93% on BIRD but only 37.04% on Spider 2.0-lite under its reported matched setting. [2](https://aclanthology.org/2026.findings-acl.1544/) PV-SQL's probing and rule-based verification improved execution accuracy by five percentage points over the baseline used in that paper while also increasing valid efficiency. [17](https://aclanthology.org/2026.findings-acl.1286/)

The implication is important: **agentic iteration is most defensible when the feedback is more reliable than the generator**. Compilation errors are an excellent agent feedback signal. A failing deterministic unit test is useful. A verified schema violation is useful. An LLM's own confidence that the data “looks right” is not equivalent.

Generic coding-agent productivity evidence is also highly task-sensitive. A GitHub-sponsored controlled experiment reported that Copilot users completed a bounded JavaScript HTTP-server task 55% faster. [18](https://github.blog/news-insights/research/research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/?utm_source=chatgpt.com) In contrast, METR's 2025 randomized study of experienced open-source developers working in their own familiar repositories found AI use increased completion time by 19% for its participants and tasks. [19](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/?utm_source=chatgpt.com) METR's February 2026 follow-up said newer tools likely provide more benefit but that severe selection effects made its updated magnitude estimates unreliable. [20](https://metr.org/blog/2026-02-24-uplift-update/?utm_source=chatgpt.com)

Thus neither “AI coding makes engineers 55% faster” nor “AI coding makes engineers 19% slower” transfers generically to data engineering. Task boundedness, repository familiarity, verification cost, interaction overhead, and model generation quality materially change the outcome.

### Test generation and correlated failure

Test generation is useful primarily as **candidate coverage expansion**, not as an automatically independent oracle.

The strongest direct warning comes from Zhao, Zhou, and Cohen's July 2026 study: when LLMs were prompted using buggy code, they generated more tests that validated the erroneous behavior and fewer tests that exposed the bug. Replacing the code-under-test context with specification-oriented context reduced the effect. [6](https://arxiv.org/abs/2607.22883?utm_source=chatgpt.com)

That produces a data-engineering-specific anti-pattern:

> AI generates a transformation from ambiguous prose → AI examines its own transformation → AI generates assertions consistent with that transformation → all assertions pass.

No independently encoded information has entered the loop.

The strongest test suites for AI-generated data transformations therefore derive expected properties from **different sources**: business contracts, fixed historical golden data, producer guarantees, hand-authored invariants, prior production distributions, cross-system reconciliations, mutation/adversarial cases, or independently maintained reference implementations.

Generic software evidence reinforces this point. SWE-Proof found that conventional tests missed counterexamples in a substantial fraction of patches that otherwise passed the benchmark test suite; moreover, when models had to write their own formal specifications, only 62% passed the study's specification audit. [21](https://arxiv.org/abs/2609.21190?utm_source=chatgpt.com) This is not a data-engineering experiment, but it demonstrates that **moving from generated implementation to generated specification does not automatically create independence either**.

### Code review

AI review is best supported as an additional reviewer rather than the final approval mechanism.

SWE-PRBench evaluated eight frontier models against human-identified PR issues and found only 15–31% detection on its diff-only condition. More context did not rescue performance; all eight degraded as the context expanded. [7](https://arxiv.org/abs/2603.26130) METR's maintainer study similarly found that approximately half of AI-generated SWE-bench patches which passed the benchmark's automated grader would not have been accepted by maintainers. [5](https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/?utm_source=chatgpt.com)

Data engineering adds review classes poorly represented by generic code-review benchmarks:

<table header-row="true">
	<tr>
		<td>Review target</td>
		<td>Why AI can help</td>
		<td>Why deterministic/human review still matters</td>
	</tr>
	<tr>
		<td>SQL syntax/style</td>
		<td>Highly explicit rules</td>
		<td>Linters can do much of it deterministically</td>
	</tr>
	<tr>
		<td>Obvious join predicates</td>
		<td>Pattern recognition</td>
		<td>Correct relationship depends on grain/cardinality</td>
	</tr>
	<tr>
		<td>Missing null handling</td>
		<td>Local pattern</td>
		<td>Desired semantics may be business-specific</td>
	</tr>
	<tr>
		<td>Performance</td>
		<td>Can explain plans/suggest rewrites</td>
		<td>Actual cost/runtime must be measured</td>
	</tr>
	<tr>
		<td>Security</td>
		<td>Can flag suspicious patterns</td>
		<td>IAM/policy engines should enforce</td>
	</tr>
	<tr>
		<td>Incremental logic</td>
		<td>Can inspect code path</td>
		<td>Correctness requires temporal/replay tests</td>
	</tr>
	<tr>
		<td>Schema compatibility</td>
		<td>Can enumerate likely impacts</td>
		<td>Authoritative dependency graph/contracts needed</td>
	</tr>
	<tr>
		<td>Business logic</td>
		<td>Can challenge assumptions</td>
		<td>Model often lacks authoritative meaning</td>
	</tr>

</table>

GitHub now documents a configurable Copilot code-review feature whose approvals can, in public preview, be allowed to count toward merge requirements. [22](https://docs.github.com/en/copilot/how-tos/copilot-on-github/set-up-copilot/configure-code-review?utm_source=chatgpt.com) That is a **configurable product capability**, not empirical evidence that AI review is sufficiently reliable to replace human approval. It also illustrates why “agent can approve” and “agent should be the sole approver” must remain separate claims.

### Documentation

Documentation is one of the lowest-blast-radius applications, but it has two very different subproblems.

**Structural documentation**—“this model selects these fields, depends on these tables, has these tests, and is downstream of X”—can be grounded in machine-readable artifacts and is relatively amenable to automation. dbt's metadata/catalog infrastructure is an example of the kind of authoritative structural source from which explanations can be generated. [23](https://docs.getdbt.com/docs/explore/explore-projects)

**Semantic documentation**—“this metric represents recognized recurring revenue,” “this field excludes reactivated accounts,” or “this dimension is authoritative for regulatory reporting”—cannot safely be inferred from names and SQL alone. SQL-comment-generation research published in 2026 explicitly reported that general LLMs struggle with complex SQL semantics and can produce inaccurate comments, motivating specialized, expert-validated training data. [24](https://arxiv.org/abs/2603.18606?utm_source=chatgpt.com)

The risk is unusually persistent: generated documentation can be wrong once and then become retrieved context for future agents, thereby **converting one hallucination into institutional context**. This failure loop makes approval and provenance more important for semantic documentation than its low immediate execution risk might suggest.

### Schema evolution, migrations, incrementals, and backfills

These tasks deserve a higher risk category than isolated query generation because they alter interfaces or historical state.

Current platforms expose dedicated mechanisms for the underlying hazards. dbt contracts can enforce declared model shape, and versioning supports controlled breaking changes. [15](https://docs.getdbt.com/guides/mesh-qs) Delta Lake enforces schemas on write and keeps versioned table history; Databricks' schema-evolution documentation describes concurrency conflicts and stateful complications around schema changes. [25](https://docs.databricks.com/aws/en/tables/update-schema)

What these controls **do not** establish is that the intended schema itself is correct.

For AI-assisted schema evolution, the deterministic evidence should therefore cover:

**compatibility** — identified downstream consumers and contract checks;

**data effect** — row counts, null rates, uniqueness, distributions, reconciliation;

**state effect** — explicit partitions/time windows to be modified;

**execution scope** — bounded resources and permissions;

**reversibility** — known restore point or old table/version;

**dual behavior** — where practical, old/new implementations compared over the same source data;

**approval** — ownership-aware gate for breaking or historical changes.

Incremental processing is particularly resistant to generic code-generation evaluation because correctness is a function of **code + prior state + event ordering + replay behavior**. A candidate implementation needs tests for first run, repeated run, late-arriving rows, changed rows, duplicate input, missing intervals, replay, deletion/CDC semantics, and full-refresh equivalence where that equivalence is supposed to hold.

For backfills, a successful query is a weak success criterion. The important assertions are often: *exact intended partition set changed; unaffected partitions did not; totals reconcile; repeated execution is understood; downstream artifacts are either recomputed or explicitly not; and historical versions are recoverable.*

The present evidence base contains **far less empirical research on AI handling stateful incremental models/backfills than on text-to-SQL generation**. Therefore, claims of autonomous reliability here remain unresolved rather than merely “lower confidence.”

### Debugging and incident response

AI is more defensible as an **evidence collector and hypothesis generator** than as an autonomous incident repairer.

OurBench shows that even offline debugging of difficult enterprise ETL SQL remains challenging: its best evaluated model solved only 32.17% of semantic-error cases. [4](https://arxiv.org/abs/2601.18119) Operational studies outside data engineering are more encouraging for evidence triage. Log-Insight, evaluated using historical production incidents, reported strong top-ranked root-cause performance after aggressive log compression, although its evaluation covered only 11 historical incidents and concerned microservices rather than data pipelines. LogSage reported industrial-scale use for CI/CD failure diagnosis, but again does not establish autonomous production data repair. [26](https://arxiv.org/abs/2607.08529?utm_source=chatgpt.com)

A robust incident workflow should distinguish four epistemic states:

<table header-row="true">
	<tr>
		<td>Stage</td>
		<td>AI role</td>
		<td>Safe authority</td>
	</tr>
	<tr>
		<td>**Evidence collection**</td>
		<td>Gather logs, job history, lineage, recent changes, query plans</td>
		<td>High, when read-only</td>
	</tr>
	<tr>
		<td>**Hypothesis generation**</td>
		<td>Rank possible causes, identify missing evidence</td>
		<td>High as advisory</td>
	</tr>
	<tr>
		<td>**Root-cause confirmation**</td>
		<td>Reproduce or discriminate among hypotheses</td>
		<td>Must depend on evidence, not narrative plausibility</td>
	</tr>
	<tr>
		<td>**Repair**</td>
		<td>Generate patch/runbook action</td>
		<td>Production execution requires the strongest controls</td>
	</tr>

</table>

Datadog's current database-investigation architecture embodies that separation: its AI can gather evidence, produce an RCA-style report, and supply suggested SQL, while explicitly never connecting to the database or executing that SQL. [10](https://docs.datadoghq.com/database_monitoring/database_investigator/?utm_source=chatgpt.com)

This is significant because incident response combines **high consequence with low epistemic certainty**. Level 11 is not simply level 10 performed faster. A production migration normally begins with an intended change; an incident agent must first infer what is wrong, often from incomplete and noisy observations, and then choose a mutation. That is a qualitatively different risk class.

## Deterministic controls, evaluation, and the human role

### Controls that materially constrain AI risk

A useful control either **limits what the agent can do, supplies information independent of the agent, detects an invalid change, or makes the consequence reversible**.

<table header-row="true">
	<tr>
		<td>Control</td>
		<td>What it actually constrains</td>
		<td>Where it is strongest</td>
		<td>Important limitation</td>
	</tr>
	<tr>
		<td>Least-privilege credentials</td>
		<td>Reachable systems/actions</td>
		<td>Every agentic workflow</td>
		<td>Fine-grained permissions still permit bad authorized actions</td>
	</tr>
	<tr>
		<td>Separate development branch</td>
		<td>Persistent writes isolated from main</td>
		<td>Repository agents</td>
		<td>Says nothing about correctness</td>
	</tr>
	<tr>
		<td>Protected PR / required review</td>
		<td>Integration authority</td>
		<td>Levels 5–9</td>
		<td>Weak if reviewer/gate shares same failure mode</td>
	</tr>
	<tr>
		<td>Compilation / type checking</td>
		<td>Structural validity</td>
		<td>SQL/dbt/Python</td>
		<td>Not semantic validity</td>
	</tr>
	<tr>
		<td>Lint/static policy</td>
		<td>Known rule violations</td>
		<td>Style, risky constructs</td>
		<td>Only encoded rules</td>
	</tr>
	<tr>
		<td>Unit tests</td>
		<td>Defined local behavior</td>
		<td>Pure transformations/functions</td>
		<td>Oracle may be incomplete</td>
	</tr>
	<tr>
		<td>Data-quality tests</td>
		<td>Dataset invariants</td>
		<td>Null, uniqueness, accepted ranges</td>
		<td>Usually sparse approximation of meaning</td>
	</tr>
	<tr>
		<td>Contracts</td>
		<td>Interface shape and selected constraints</td>
		<td>Cross-team boundaries</td>
		<td>Shape ≠ business correctness</td>
	</tr>
	<tr>
		<td>Golden-data / reconciliation</td>
		<td>Output behavior against independent reference</td>
		<td>Migration/backfill/refactor</td>
		<td>Reference can itself be stale/wrong</td>
	</tr>
	<tr>
		<td>Differential execution</td>
		<td>Detects changed output</td>
		<td>Refactoring/migration</td>
		<td>Changed output may be intended</td>
	</tr>
	<tr>
		<td>Query dry run</td>
		<td>Syntax/plan/cost estimate before query execution</td>
		<td>Warehouses</td>
		<td>Does not validate business semantics [13](https://docs.cloud.google.com/bigquery/docs/running-queries)</td>
	</tr>
	<tr>
		<td>Cost/resource cap</td>
		<td>Limits financial/compute blast radius</td>
		<td>Agent probes/backfills</td>
		<td>Does not prevent wrong cheap query</td>
	</tr>
	<tr>
		<td>Staging / clone / shadow</td>
		<td>Isolates mutation</td>
		<td>Migrations/backfills</td>
		<td>Environment may differ from production</td>
	</tr>
	<tr>
		<td>Partition/row scope limit</td>
		<td>Limits state blast radius</td>
		<td>Backfill/repair</td>
		<td>Wrong scoped rows can still matter</td>
	</tr>
	<tr>
		<td>Versioning/time travel</td>
		<td>Enables recovery/inspection</td>
		<td>Mutable warehouse tables</td>
		<td>Retention and external side effects matter [14](https://docs.databricks.com/aws/en/tables/history)</td>
	</tr>
	<tr>
		<td>Canary / progressive rollout</td>
		<td>Limits deployment exposure</td>
		<td>Services/pipelines</td>
		<td>Needs meaningful quality metrics</td>
	</tr>
	<tr>
		<td>Explicit human approval</td>
		<td>Adds semantic/accountability check</td>
		<td>High-consequence changes</td>
		<td>Humans can rubber-stamp AI output</td>
	</tr>
	<tr>
		<td>Immutable audit log</td>
		<td>Supports attribution/postmortem</td>
		<td>All agents</td>
		<td>Detective rather than preventive</td>
	</tr>
	<tr>
		<td>Tested rollback</td>
		<td>Reduces irreversibility</td>
		<td>Production mutation</td>
		<td>Some side effects are not fully reversible</td>
	</tr>

</table>

GitHub's documented cloud-agent architecture combines several of these: branch-scoped write access, branch protections, required checks, restrictions on workflow execution, and human review before merge for its cloud-agent PR flow. [9](https://docs.github.com/en/copilot/concepts/security-governance-and-network-settings/risks-and-mitigations?utm_source=chatgpt.com) Google's data-engineering agent uses a different boundary: it can modify workspace files and compile/fix generated code but cannot execute the resulting pipeline. [8](https://docs.cloud.google.com/gemini/data-agents/data-engineering-agent/agent-overview?utm_source=chatgpt.com)

### Controls that can provide false assurance

Several mechanisms look stronger than they are.

**“CI is green.”** METR's maintainer study is direct evidence against equating a benchmark test pass with an acceptable patch. [5](https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/?utm_source=chatgpt.com) In data engineering, CI may not even execute against production-like distributions or state.

**“The model wrote tests too.”** This may be correlated verification rather than independent evidence; buggy implementations can steer LLM-generated tests toward the same error. [6](https://arxiv.org/abs/2607.22883?utm_source=chatgpt.com)

**“It executed without errors.”** Execution confirms syntax, names, permissions, and some runtime behavior—not grain, business meaning, completeness, or downstream compatibility. Spider 2.0 and related enterprise-SQL research show that those higher-level problems remain difficult. [3](https://arxiv.org/abs/2411.07763)

**“Another model reviewed it.”** Independent sampling is not the same as an independent oracle. SWE-PRBench indicates substantial defect-recall limitations in current model review. [7](https://arxiv.org/abs/2603.26130)

**“The generated documentation explains the model.”** Explanation grounded in code may faithfully document the implementation while still misrepresenting intended business semantics.

**“Rollback exists.”** Rollback is only effective if all relevant side effects are reversible and the restore mechanism has adequate retention/state. Platform history features improve reversibility but do not guarantee it. [14](https://docs.databricks.com/aws/en/tables/history)

### How AI-assisted data engineering should be evaluated

The unit of evaluation should be **the change episode**, not merely the generated snippet.

A serious evaluation needs multiple outcome layers:

<table header-row="true">
	<tr>
		<td>Dimension</td>
		<td>Example metric</td>
	</tr>
	<tr>
		<td>Task completion</td>
		<td>Requirement actually fulfilled</td>
	</tr>
	<tr>
		<td>Code correctness</td>
		<td>Compile/execution/test pass</td>
	</tr>
	<tr>
		<td>**Data correctness**</td>
		<td>Reference-output match, reconciliation, invariant violations</td>
	</tr>
	<tr>
		<td>Temporal correctness</td>
		<td>Replay, late-data, repeated-run and full-refresh consistency</td>
	</tr>
	<tr>
		<td>Downstream regression</td>
		<td>Affected models/dashboards/interfaces</td>
	</tr>
	<tr>
		<td>Review quality</td>
		<td>Defects caught/missed; reviewer acceptance</td>
	</tr>
	<tr>
		<td>Human effort</td>
		<td>Active engineering and review minutes</td>
	</tr>
	<tr>
		<td>Intervention rate</td>
		<td>Fraction of agent episodes requiring rescue</td>
	</tr>
	<tr>
		<td>Agent iteration</td>
		<td>Tool calls/repair loops per accepted change</td>
	</tr>
	<tr>
		<td>Change scope</td>
		<td>Unnecessary files/tables/partitions touched</td>
	</tr>
	<tr>
		<td>Cost</td>
		<td>Inference + warehouse compute + developer time</td>
	</tr>
	<tr>
		<td>Performance</td>
		<td>Query/runtime/warehouse consumption</td>
	</tr>
	<tr>
		<td>Production outcome</td>
		<td>Incident and rollback rate</td>
	</tr>
	<tr>
		<td>Maintainability</td>
		<td>Subsequent fixes/reverts; human comprehension</td>
	</tr>
	<tr>
		<td>Reversibility</td>
		<td>Time/success to restore known-good state</td>
	</tr>

</table>

No currently common software benchmark captures this full envelope. METR itself cautions that coding benchmarks often contain self-contained tasks with clear algorithmic scoring unlike real development work. [19](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/?utm_source=chatgpt.com) Its 2026 maintainer study provides an empirical example of the gap between automated benchmark acceptance and human maintainers' standards. [5](https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/?utm_source=chatgpt.com) Spider 2.0 improves realism for enterprise SQL by adding codebases, documentation, complex schemas, and database interaction, but remains a benchmark rather than longitudinal evidence of production operation. [3](https://arxiv.org/abs/2411.07763)

For stateful data engineering, an evaluation suite should deliberately include **silent failures**: fanout that doubles a metric, dropped late events, incorrect watermark advancement, partial backfill, changed null semantics, schema compatibility breakage, history rewrite, costly but correct SQL, stale semantic context, and tests that agree with the same erroneous implementation.

### Human-role implications

The evidence does not imply that humans become unnecessary as generation becomes cheaper. It shifts scarce human attention toward areas for which authoritative truth is hardest to infer.

The responsibilities likely to increase in relative importance are:

**Problem and specification definition.** SWE-Proof's difficulty with model-written specifications and the unit-test misguidance findings both make the specification bottleneck more visible. [21](https://arxiv.org/abs/2609.21190?utm_source=chatgpt.com)

**Business semantics.** Models can infer structure; authoritative meaning must usually come from people/processes that own the metric or domain.

**Architecture and boundaries.** Agents need constrained interfaces, permissions, reversible environments, and explicit scopes.

**Contracts and observability.** Cheap implementation increases the value of independently encoded expected behavior.

**Review of consequences rather than syntax.** Humans add the most value by checking intent, grain, compatibility, state transitions, and blast radius rather than formatting.

**Governance and accountability.** Higher autonomy increases the importance of explicit ownership, approval rules, auditability, and incident responsibility.

**Evaluation design.** If the model optimizes against a check, the organization must decide whether the check actually represents what it cares about.

This is an interpretation of the combined evidence, not a measured labor-market forecast.

## Failure catalogue, contradictions, and research frontier

### Failure-mode catalogue

<table header-row="true">
	<tr>
		<td>Failure</td>
		<td>AI-specific?</td>
		<td>Why it is dangerous in data engineering</td>
		<td>Primary counter-evidence/control</td>
	</tr>
	<tr>
		<td>Valid but semantically wrong SQL</td>
		<td>Shared, AI-amplified</td>
		<td>Looks successful</td>
		<td>Golden data, reconciliation, business invariants</td>
	</tr>
	<tr>
		<td>Hallucinated table/column</td>
		<td>AI-amplified</td>
		<td>May fail clearly or target wrong similar object</td>
		<td>Schema-grounded tools, compilation</td>
	</tr>
	<tr>
		<td>Wrong grain</td>
		<td>Shared</td>
		<td>Plausible rows, wrong entity semantics</td>
		<td>Grain assertions, uniqueness checks</td>
	</tr>
	<tr>
		<td>Join fanout</td>
		<td>Shared</td>
		<td>Silent metric inflation</td>
		<td>Cardinality assertions, before/after counts</td>
	</tr>
	<tr>
		<td>Wrong join path</td>
		<td>Shared, hard for LLMs</td>
		<td>Correct names but wrong relationship</td>
		<td>Explicit relationship metadata; execution tests; SchemaScope evidence [12](https://aclanthology.org/2026.surgellm-1.17/)</td>
	</tr>
	<tr>
		<td>Incorrect incremental watermark</td>
		<td>Shared</td>
		<td>Data permanently skipped/reprocessed</td>
		<td>Replay and full-refresh comparison</td>
	</tr>
	<tr>
		<td>Late-arrival mishandling</td>
		<td>Shared</td>
		<td>Slowly diverging completeness</td>
		<td>Delayed-event simulation</td>
	</tr>
	<tr>
		<td>Non-idempotent repair/backfill</td>
		<td>Shared</td>
		<td>Duplicate/cumulative corruption</td>
		<td>repeated-execution test</td>
	</tr>
	<tr>
		<td>Stale context</td>
		<td>AI-amplified</td>
		<td>Confidently grounded in obsolete information</td>
		<td>freshness/version metadata</td>
	</tr>
	<tr>
		<td>Excessive context</td>
		<td>AI-relevant</td>
		<td>Dilutes attention</td>
		<td>retrieval/ranking; SWE-PRBench evidence [7](https://arxiv.org/abs/2603.26130)</td>
	</tr>
	<tr>
		<td>Incorrect business definition</td>
		<td>Shared, AI-amplified</td>
		<td>Output appears reasonable</td>
		<td>authoritative semantic layer/owner</td>
	</tr>
	<tr>
		<td>Expensive generated query</td>
		<td>Shared, scalable with agents</td>
		<td>Compute spend amplified by loops</td>
		<td>dry run, maximum spend/bytes</td>
	</tr>
	<tr>
		<td>Incomplete migration</td>
		<td>Shared</td>
		<td>Long-tail consumer breakage</td>
		<td>contracts, dependency graph, dual run</td>
	</tr>
	<tr>
		<td>Destructive schema change</td>
		<td>Shared</td>
		<td>Immediate downstream/state loss</td>
		<td>least privilege, approval, versioning</td>
	</tr>
	<tr>
		<td>Wrong backfill scope</td>
		<td>Shared</td>
		<td>Rewrites historical truth</td>
		<td>explicit partition manifest, shadow table</td>
	</tr>
	<tr>
		<td>Tests reproduce implementation error</td>
		<td>Strong AI correlation evidence</td>
		<td>False confidence</td>
		<td>independent specification/test authoring [6](https://arxiv.org/abs/2607.22883?utm_source=chatgpt.com)</td>
	</tr>
	<tr>
		<td>Agent optimizes until incomplete checks pass</td>
		<td>Agent-specific amplification</td>
		<td>“Green” becomes optimization target</td>
		<td>holdout/adversarial checks, scope review</td>
	</tr>
	<tr>
		<td>Unnecessary files/tables changed</td>
		<td>Agent-specific tendency</td>
		<td>Expanded blast radius</td>
		<td>path/table allowlists</td>
	</tr>
	<tr>
		<td>False incident diagnosis</td>
		<td>Shared, AI plausibility risk</td>
		<td>Wrong remediation under pressure</td>
		<td>require causal/reproduction evidence</td>
	</tr>
	<tr>
		<td>Repair outside intended scope</td>
		<td>Agent authority risk</td>
		<td>Unbounded mutation</td>
		<td>tool permissions and policy</td>
	</tr>
	<tr>
		<td>Failed/incomplete rollback</td>
		<td>Shared</td>
		<td>Recovery assumption false</td>
		<td>tested restore procedure</td>
	</tr>

</table>

Most fundamental failures are **not unique to AI**. Humans also write bad joins, stale documentation, faulty tests, and destructive migrations. AI changes the risk distribution through three mechanisms: it makes artifact generation dramatically cheaper, can reproduce the same assumption across code/tests/docs, and can operate at machine speed across a larger scope. The appropriate comparison is therefore not “AI can err but humans also err”; it is **error probability × correlated verification failure × action rate × blast radius × detectability × reversibility**.

### Contradictions in the evidence

**Productivity: strong acceleration versus slowdown.** GitHub's controlled Copilot study reported a 55% completion-time improvement on a bounded JavaScript task, whereas METR found a 19% slowdown among experienced developers performing real issues in their own repositories in early 2025. [18](https://github.blog/news-insights/research/research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/?utm_source=chatgpt.com) METR's 2026 follow-up suggests the effect changed as tools improved but explicitly says selection effects prevent a confident updated estimate. [20](https://metr.org/blog/2026-02-24-uplift-update/?utm_source=chatgpt.com) These are better explained as **different task distributions, users, tool generations, and verification burdens** than as mutually exclusive claims.

**Context helps versus context hurts.** Data-specific research finds major benefits from supplying explicit business semantics, while SWE-PRBench finds deterioration from adding broader code-review context. [16](https://arxiv.org/abs/2604.25149) The apparent contradiction weakens once “context” is decomposed into **relevant authoritative signal** versus **volume of potentially distracting material**.

**Execution feedback substantially improves SQL versus enterprise SQL remains weak.** VET and PV-SQL show measurable gains from database-grounded execution and verification, but Spider 2.0 and OurBench still show poor performance on harder enterprise tasks. [2](https://aclanthology.org/2026.findings-acl.1544/) The evidence supports iterative verification as a useful architecture—not as evidence of solved reliability.

**Tests pass versus change is correct.** Automated tests are central controls, but METR's maintainer study and SWE-Proof both demonstrate important classes of test-passing patches that remain unacceptable or incorrect. [5](https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/?utm_source=chatgpt.com) The contradiction disappears if a passing test is treated as one observation about correctness rather than its definition.

**“Agent” can imply very different authority.** Google's Data Engineering Agent edits and validates pipeline code but explicitly cannot execute the pipeline. GitHub's coding-agent tooling can perform long autonomous PR-repair loops; its cloud agent is constrained by branch and review controls, while separate current GitHub settings can allow Copilot code-review approvals to count toward merge requirements. [8](https://docs.cloud.google.com/gemini/data-agents/data-engineering-agent/agent-overview?utm_source=chatgpt.com) Therefore “agentic” is not an autonomy level.

### Unresolved questions

The evidence is insufficient to resolve several central issues:

<table header-row="true">
	<tr>
		<td>Question</td>
		<td>What evidence is missing</td>
	</tr>
	<tr>
		<td>Can an agent reliably design incremental models with late data and CDC?</td>
		<td>Controlled benchmark with hidden temporal/state cases</td>
	</tr>
	<tr>
		<td>Can AI safely execute bounded backfills?</td>
		<td>Realistic studies measuring scope errors, reconciliation, recovery</td>
	</tr>
	<tr>
		<td>Does AI review detect data-specific semantic defects such as grain/fanout?</td>
		<td>SQL/dbt PR benchmark with expert annotations</td>
	</tr>
	<tr>
		<td>How should independent AI-generated tests be made?</td>
		<td>Experiments varying generator/evaluator/specification independence</td>
	</tr>
	<tr>
		<td>What is the optimal context-selection strategy for large data estates?</td>
		<td>Controlled studies varying relevance, staleness, contradictions and authority</td>
	</tr>
	<tr>
		<td>Can agents correctly assess downstream schema impact across tools/repos?</td>
		<td>Cross-platform lineage benchmark with hidden consumers</td>
	</tr>
	<tr>
		<td>How often do green AI PRs create later data incidents?</td>
		<td>Longitudinal production data</td>
	</tr>
	<tr>
		<td>Can an agent safely decide whether a backfill is reversible?</td>
		<td>Stateful environment benchmark with external side effects</td>
	</tr>
	<tr>
		<td>How does agent use affect warehouse spend?</td>
		<td>Field studies including exploratory/repair tool calls</td>
	</tr>
	<tr>
		<td>Can AI perform RCA rather than merely produce plausible RCA narratives?</td>
		<td>Incident benchmark requiring intervention/counterfactual validation</td>
	</tr>
	<tr>
		<td>How rapidly does autonomous PR iteration exploit weak tests?</td>
		<td>Adversarial evaluator studies</td>
	</tr>
	<tr>
		<td>What production-change classes can safely reach level 9 or 10?</td>
		<td>Risk-stratified prospective deployments with incident/rollback outcomes</td>
	</tr>

</table>

### Research frontier

The most valuable next research step is not another generic code-generation benchmark. It is an **end-to-end data-engineering change benchmark with hidden state and independent evaluation**.

A representative task would include a repository, warehouse metadata, lineage, semantic definitions, historical data, orchestration state, cost model, and a requested change. An agent would be allowed to inspect, edit, execute in a clone, and iterate. Evaluation would then use hidden checks for semantic correctness, grain, late arrivals, historical reconciliation, affected downstream assets, execution cost, unnecessary changes, and rollback behavior.

A second priority is **evaluator independence**: quantifying how defect detection changes when implementation and tests are produced by the same model, different models, independently written specifications, deterministic invariants, or observed production history. The misguidance findings make this a concrete research question rather than a theoretical concern. [6](https://arxiv.org/abs/2607.22883?utm_source=chatgpt.com)

A third is **risk-conditioned autonomy**. The useful variable is unlikely to be model capability alone. A more defensible function is:

[
\\text{Permissible autonomy}
\\approx f(
\\text{verification strength},
\\text{specification quality},
\\text{blast radius},
\\text{reversibility},
\\text{observability},
\\text{consequence of error}
)
]

That implies very different autonomy for two changes produced by the same model: automatically repairing a formatting error and rerunning CI may justify level 8, while rewriting historical revenue tables may remain at level 5–6 even when the generated SQL itself is excellent.

The evidence as of October 3, 2026 therefore does **not** support treating autonomous data engineering as the inevitable endpoint of AI-assisted development. It supports a more differentiated endpoint: **high autonomy where the environment supplies strong independent verification and reversible state; low autonomy where correctness depends on ambiguous semantics, hidden temporal state, broad downstream consequences, or uncertain incident diagnosis.**

## Claim-to-source dossier

The table below records the most load-bearing claims. “Supported” means the source directly supports the stated claim in its studied/documented setting; it does not imply universal generalization. **Inspected date for all sources: 2026-10-03.**

<table header-row="true">
	<tr>
		<td>Exact claim</td>
		<td>Task</td>
		<td>Evidence type</td>
		<td>System / model context</td>
		<td>Source date</td>
		<td>Direct URL / DOI</td>
		<td>Status</td>
		<td>Main limitations</td>
	</tr>
	<tr>
		<td>Spider 2.0 contains 632 enterprise-level SQL workflow tasks, often using very large schemas and project/context resources; its abstract reports only 17.0% solved by an o1-preview code-agent framework.</td>
		<td>SQL / DE agent</td>
		<td>Benchmark</td>
		<td>o1-preview agent; BigQuery, Snowflake and other DBs</td>
		<td>2024-11-12</td>
		<td>[https://arxiv.org/abs/2411.07763](https://arxiv.org/abs/2411.07763)</td>
		<td>**Supported** [3](https://arxiv.org/abs/2411.07763)</td>
		<td>Benchmark, not live production; exact results depend on experimental configuration/version</td>
	</tr>
	<tr>
		<td>Stepwise database execution feedback can materially improve text-to-SQL, but VET still reaches only 37.04% on Spider 2.0-lite.</td>
		<td>SQL verification</td>
		<td>Peer-reviewed benchmark</td>
		<td>VET; BIRD + Spider 2.0-lite</td>
		<td>2026-07</td>
		<td>[https://doi.org/10.18653/v1/2026.findings-acl.1544](https://doi.org/10.18653/v1/2026.findings-acl.1544)</td>
		<td>**Supported** [2](https://aclanthology.org/2026.findings-acl.1544/)</td>
		<td>Execution accuracy is not production-safety evaluation</td>
	</tr>
	<tr>
		<td>Database probing plus rule-based verification improved the studied baseline by 5 points execution accuracy on BIRD.</td>
		<td>SQL generation/verification</td>
		<td>Peer-reviewed benchmark</td>
		<td>PV-SQL</td>
		<td>2026-07</td>
		<td>[https://doi.org/10.18653/v1/2026.findings-acl.1286](https://doi.org/10.18653/v1/2026.findings-acl.1286)</td>
		<td>**Supported** [17](https://aclanthology.org/2026.findings-acl.1286/)</td>
		<td>BIRD less representative than full enterprise workflows</td>
	</tr>
	<tr>
		<td>Complex enterprise SQL debugging remains difficult: best model reported 36.46% syntax and 32.17% semantic debugging accuracy.</td>
		<td>Debugging</td>
		<td>Benchmark/preprint</td>
		<td>~30 LLMs; best Claude-4-Sonnet</td>
		<td>v2 2026-07-24</td>
		<td>[https://doi.org/10.48550/arXiv.2601.18119](https://doi.org/10.48550/arXiv.2601.18119)</td>
		<td>**Supported** [4](https://arxiv.org/abs/2601.18119)</td>
		<td>Synthetic bug injection; execution-free evaluation</td>
	</tr>
	<tr>
		<td>Current frontier AI code reviewers detected only 15–31% of human-flagged issues in SWE-PRBench's diff-only condition.</td>
		<td>Review</td>
		<td>Benchmark/preprint</td>
		<td>8 frontier models, 350 OSS PRs</td>
		<td>2026-03-27</td>
		<td>[https://doi.org/10.48550/arXiv.2603.26130](https://doi.org/10.48550/arXiv.2603.26130)</td>
		<td>**Supported** [7](https://arxiv.org/abs/2603.26130)</td>
		<td>Generic SE rather than data engineering; judge-based evaluation</td>
	</tr>
	<tr>
		<td>All eight SWE-PRBench models degraded as more repository context was supplied.</td>
		<td>Context/review</td>
		<td>Benchmark/preprint</td>
		<td>8 frontier models</td>
		<td>2026-03-27</td>
		<td>[https://doi.org/10.48550/arXiv.2603.26130](https://doi.org/10.48550/arXiv.2603.26130)</td>
		<td>**Supported** [7](https://arxiv.org/abs/2603.26130)</td>
		<td>One benchmark/context design; does not prove all extra context harms</td>
	</tr>
	<tr>
		<td>Buggy implementations can misguide LLM-generated unit tests toward asserting the same incorrect behavior.</td>
		<td>Test generation</td>
		<td>Controlled empirical/preprint</td>
		<td>Multiple LLM test-generation conditions</td>
		<td>2026-07-24</td>
		<td>[https://arxiv.org/abs/2607.22883](https://arxiv.org/abs/2607.22883)</td>
		<td>**Supported** [6](https://arxiv.org/abs/2607.22883?utm_source=chatgpt.com)</td>
		<td>Unit tests/generic code, not data-quality tests</td>
	</tr>
	<tr>
		<td>Formal verification found counterexamples in roughly one-quarter to one-half of test-passing patches for two tested frontier models.</td>
		<td>Verification</td>
		<td>Benchmark/preprint</td>
		<td>SWE-Proof; 2 frontier models</td>
		<td>2026-09-18</td>
		<td>[https://arxiv.org/abs/2609.21190](https://arxiv.org/abs/2609.21190)</td>
		<td>**Supported** [21](https://arxiv.org/abs/2609.21190?utm_source=chatgpt.com)</td>
		<td>Very recent preprint; software engineering, not DE</td>
	</tr>
	<tr>
		<td>Only 62% of model-generated specifications in SWE-Proof passed the paper's specification audit.</td>
		<td>Specification</td>
		<td>Benchmark/preprint</td>
		<td>SWE-Proof</td>
		<td>2026-09-18</td>
		<td>[https://arxiv.org/abs/2609.21190](https://arxiv.org/abs/2609.21190)</td>
		<td>**Supported** [21](https://arxiv.org/abs/2609.21190?utm_source=chatgpt.com)</td>
		<td>Formal-method task setting</td>
	</tr>
	<tr>
		<td>Roughly half of SWE-bench Verified AI patches that passed the automated grader would not have been merged by maintainers in METR's study.</td>
		<td>Repository changes</td>
		<td>Maintainer empirical study</td>
		<td>296 AI PRs reviewed by maintainers</td>
		<td>2026-03-10</td>
		<td>[https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/](https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/)</td>
		<td>**Supported** [5](https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/?utm_source=chatgpt.com)</td>
		<td>Small number of repos/maintainers; static review; no active agent iteration</td>
	</tr>
	<tr>
		<td>Experienced OSS developers in METR's early-2025 RCT took 19% longer with AI on the studied tasks.</td>
		<td>Productivity</td>
		<td>Randomized controlled field study</td>
		<td>Experienced developers, own repositories</td>
		<td>2025-07-10</td>
		<td>[https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/)</td>
		<td>**Supported, historical** [19](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/?utm_source=chatgpt.com)</td>
		<td>Early-2025 tools; task/developer selection</td>
	</tr>
	<tr>
		<td>METR's later experiment provides only weak evidence on 2026 productivity because adoption created substantial selection bias.</td>
		<td>Productivity</td>
		<td>Empirical-methodology update</td>
		<td>57 developers, 143 repos, 800+ tasks</td>
		<td>2026-02-24</td>
		<td>[https://metr.org/blog/2026-02-24-uplift-update/](https://metr.org/blog/2026-02-24-uplift-update/)</td>
		<td>**Supported** [20](https://metr.org/blog/2026-02-24-uplift-update/?utm_source=chatgpt.com)</td>
		<td>Authors explicitly say estimates unreliable</td>
	</tr>
	<tr>
		<td>Google's BigQuery Data Engineering Agent can generate/modify Dataform pipelines and validate/fix compilation errors, but cannot execute pipelines.</td>
		<td>DE workflow agent</td>
		<td>Official documented capability</td>
		<td>Gemini / BigQuery / Dataform</td>
		<td>Current at inspection</td>
		<td>[https://docs.cloud.google.com/gemini/data-agents/data-engineering-agent/agent-overview](https://docs.cloud.google.com/gemini/data-agents/data-engineering-agent/agent-overview)</td>
		<td>**Supported as capability** [8](https://docs.cloud.google.com/gemini/data-agents/data-engineering-agent/agent-overview?utm_source=chatgpt.com)</td>
		<td>No independent reliability/productivity evidence</td>
	</tr>
	<tr>
		<td>The same agent can use BigQuery/catalog metadata, sample rows and profiles as additional context.</td>
		<td>Context</td>
		<td>Official documented capability</td>
		<td>BigQuery Data Engineering Agent</td>
		<td>Current at inspection</td>
		<td>[https://docs.cloud.google.com/gemini/data-agents/data-engineering-agent/agent-overview](https://docs.cloud.google.com/gemini/data-agents/data-engineering-agent/agent-overview)</td>
		<td>**Supported as capability** [8](https://docs.cloud.google.com/gemini/data-agents/data-engineering-agent/agent-overview?utm_source=chatgpt.com)</td>
		<td>Context availability ≠ correct interpretation</td>
	</tr>
	<tr>
		<td>GitHub's cloud coding agent restricts writes to a branch and is subject to branch protections/checks; its documented cloud-agent PR flow requires human review before merging.</td>
		<td>Autonomous repository change</td>
		<td>Official documented capability/control</td>
		<td>GitHub Copilot cloud agent</td>
		<td>Current at inspection</td>
		<td>[https://docs.github.com/en/copilot/concepts/security-governance-and-network-settings/risks-and-mitigations](https://docs.github.com/en/copilot/concepts/security-governance-and-network-settings/risks-and-mitigations)</td>
		<td>**Supported as capability/control** [9](https://docs.github.com/en/copilot/concepts/security-governance-and-network-settings/risks-and-mitigations?utm_source=chatgpt.com)</td>
		<td>Specific GitHub agent mode/configuration</td>
	</tr>
	<tr>
		<td>GitHub `/pr auto` can autonomously loop over review feedback, conflicts and CI until green but stops short of merge.</td>
		<td>CI repair / workflow autonomy</td>
		<td>Official documented capability</td>
		<td>Copilot CLI</td>
		<td>Current at inspection</td>
		<td>[https://docs.github.com/en/copilot/how-tos/copilot-cli/use-copilot-cli/manage-pull-requests](https://docs.github.com/en/copilot/how-tos/copilot-cli/use-copilot-cli/manage-pull-requests)</td>
		<td>**Supported as capability** [11](https://docs.github.com/en/copilot/how-tos/copilot-cli/use-copilot-cli/manage-pull-requests?utm_source=chatgpt.com)</td>
		<td>Does not establish patch correctness</td>
	</tr>
	<tr>
		<td>Copilot code-review approvals can optionally count toward merge requirements in current public preview.</td>
		<td>AI review / approval</td>
		<td>Official documented capability</td>
		<td>GitHub Copilot code review</td>
		<td>Current at inspection</td>
		<td>[https://docs.github.com/en/copilot/how-tos/copilot-on-github/set-up-copilot/configure-code-review](https://docs.github.com/en/copilot/how-tos/copilot-on-github/set-up-copilot/configure-code-review)</td>
		<td>**Supported as capability** [22](https://docs.github.com/en/copilot/how-tos/copilot-on-github/set-up-copilot/configure-code-review?utm_source=chatgpt.com)</td>
		<td>Preview; policy option is not evidence of reliability</td>
	</tr>
	<tr>
		<td>Datadog's database investigation AI gathers evidence and can propose SQL but does not connect to the DB or execute SQL.</td>
		<td>Incident investigation</td>
		<td>Official documented capability</td>
		<td>Datadog Bits Chat / DB monitoring</td>
		<td>Current at inspection</td>
		<td>[https://docs.datadoghq.com/database_monitoring/database_investigator/](https://docs.datadoghq.com/database_monitoring/database_investigator/)</td>
		<td>**Supported as capability/control** [10](https://docs.datadoghq.com/database_monitoring/database_investigator/?utm_source=chatgpt.com)</td>
		<td>Vendor architecture, not comparative effectiveness evidence</td>
	</tr>
	<tr>
		<td>BigQuery supports dry runs for pre-execution query validation/processing estimates and mechanisms for bounding query cost.</td>
		<td>Deterministic cost control</td>
		<td>Official platform documentation</td>
		<td>BigQuery</td>
		<td>Current at inspection</td>
		<td>[https://docs.cloud.google.com/bigquery/docs/running-queries](https://docs.cloud.google.com/bigquery/docs/running-queries) ; [https://docs.cloud.google.com/bigquery/docs/best-practices-costs](https://docs.cloud.google.com/bigquery/docs/best-practices-costs)</td>
		<td>**Supported** [13](https://docs.cloud.google.com/bigquery/docs/running-queries)</td>
		<td>Cost control does not validate semantics</td>
	</tr>
	<tr>
		<td>Delta/Databricks exposes table history/versioning, while schema evolution has concurrency/state implications.</td>
		<td>Rollback / schema evolution</td>
		<td>Official platform documentation</td>
		<td>Delta Lake / Databricks</td>
		<td>Current at inspection</td>
		<td>[https://docs.databricks.com/aws/en/tables/history](https://docs.databricks.com/aws/en/tables/history) ; [https://docs.databricks.com/aws/en/tables/update-schema](https://docs.databricks.com/aws/en/tables/update-schema)</td>
		<td>**Supported** [14](https://docs.databricks.com/aws/en/tables/history)</td>
		<td>Platform-specific; history does not reverse external effects</td>
	</tr>
	<tr>
		<td>Adding authoritative business-semantic context substantially improved accuracy in one 2026 paired analytics benchmark.</td>
		<td>Context / semantics</td>
		<td>Benchmark/preprint</td>
		<td>Three frontier models; 100 paired questions</td>
		<td>2026-04-28</td>
		<td>[https://arxiv.org/abs/2604.25149](https://arxiv.org/abs/2604.25149)</td>
		<td>**Conditional** [16](https://arxiv.org/abs/2604.25149)</td>
		<td>Single dataset; compact hand-authored semantic context</td>
	</tr>
	<tr>
		<td>“More context consistently improves AI data-engineering reliability.”</td>
		<td>Context</td>
		<td>Cross-source synthesis</td>
		<td>Multiple</td>
		<td>Through 2026-10-03</td>
		<td>Sources above</td>
		<td>**Contradicted** [16](https://arxiv.org/abs/2604.25149)</td>
		<td>Different tasks make the boundary conditional</td>
	</tr>
	<tr>
		<td>“AI-generated code plus AI-generated tests is sufficient evidence of correctness.”</td>
		<td>Generation/testing</td>
		<td>Cross-source synthesis</td>
		<td>Multiple</td>
		<td>Through 2026-10-03</td>
		<td>Sources above</td>
		<td>**Contradicted** [6](https://arxiv.org/abs/2607.22883?utm_source=chatgpt.com)</td>
		<td>Degree of independence varies by test architecture</td>
	</tr>
	<tr>
		<td>“Passing CI establishes production readiness.”</td>
		<td>Agent workflow</td>
		<td>Cross-source synthesis</td>
		<td>SWE-bench + maintainer evidence</td>
		<td>2026</td>
		<td>Sources above</td>
		<td>**Contradicted** [5](https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/?utm_source=chatgpt.com)</td>
		<td>Generic SE study; data systems plausibly have additional hidden correctness dimensions</td>
	</tr>
	<tr>
		<td>“Current evidence establishes that autonomous agents can generally diagnose and repair production data incidents safely.”</td>
		<td>Incident autonomy</td>
		<td>Evidence-gap assessment</td>
		<td>Current literature reviewed</td>
		<td>2026-10-03</td>
		<td>—</td>
		<td>**Unresolved / unsupported**</td>
		<td>Production DE-specific intervention studies were not found in the examined evidence base</td>
	</tr>

</table>

**Overall evidence judgment:** AI is already well supported as a **candidate generator, explainer, searcher, debugger, and bounded workflow actor**. Autonomous iteration against reliable mechanical feedback can also be rational. Evidence becomes materially weaker once the evaluator itself requires semantic judgment, and weakest where an agent both **decides what is wrong and has authority to mutate durable production state**. The defensible autonomy boundary is therefore determined less by the sophistication of the model than by whether correctness is independently specifiable, mechanically observable, tightly scoped, and recoverable.