# Decision Evidence Dossier: AI-Assisted and Agentic Data Engineering

**Research snapshot: October 3, 2026**

**Decision objective:** maximize **reliable data-platform outcomes per unit of total engineering effort and risk**, not the share of work performed by AI, not lines of code generated, and not apparent coding speed.

The evidence does **not** support a blanket decision to “automate data engineering.” It supports decomposing the engineering lifecycle into task classes with very different economics and blast radii. Controlled software-engineering evidence is mixed: a 2023 experiment found developers completed a simple JavaScript task 55.8% faster with an AI pair programmer, while a 2025 randomized trial of experienced maintainers working in their own large repositories found AI made tasks 19% slower. METR subsequently concluded that its late-2025 experiment could no longer reliably estimate the current effect because AI adoption changed who and which tasks would participate; its raw later data suggested improvement but with wide confidence intervals and important selection bias. citeturn17view5turn17view2turn17view0

Data engineering has even less direct organizational evidence. Text-to-SQL capability has improved dramatically—from substantial gaps on the original BIRD benchmark to a 2026 research system reporting 93.2% execution accuracy on an expert-verified BIRD Mini-Dev set—but benchmark SQL correctness is not the same problem as changing a production transformation graph whose semantics, downstream dependencies, incremental state, cost, access controls and historical backfills matter. citeturn24view0turn19search7

The strongest practical conclusion is therefore:

> **Use AI aggressively where outputs are cheap to verify and cheap to reverse. Delegate bounded work where deterministic systems can judge success. Preserve explicit human authority where correctness is fundamentally semantic, blast radius is high, or reversal is slow or incomplete.**

This is not an autonomy maturity ladder. A sophisticated data organization may simultaneously allow autonomous documentation updates, autonomous CI repair in branches, human-approved metric SQL, and absolutely no autonomous production schema deletion. Those are rationally different policies.

## Leadership decision frame

**Deliverable 1 — Leadership decision map**

The right unit of governance is not “the AI tool”; it is **action × environment × authority × blast radius**. The same model that may safely rewrite a test fixture in an isolated branch should not therefore inherit permission to alter a production fact table.

| Engineering activity | Recommended default | Why | What would justify more autonomy |
|---|---|---|---|
| General coding assistant | **Allow with enterprise controls** | Broad usefulness; human remains actor | Local security/privacy policy and measurable user value |
| Data-specific copilot | **Allow where metadata access is governed** | Data context can reduce context-gathering work | Proven benefit versus ordinary assistant; controlled metadata access |
| AI-generated SQL | **Allow, human-owned** | SQL generation is increasingly capable, but business semantics remain under-specified in many real tasks; benchmark execution accuracy does not certify business correctness. citeturn24view0turn19search7 | Automated semantic/data checks plus evidence from your own task distribution |
| AI-generated dbt models | **Allow in branch/PR** | Code and DAG structure are testable; transformation intent is not completely testable | Contracts, unit/data tests, lineage impact, reconciliation and cost checks |
| Test generation | **Encourage, but never treat generated tests as independent proof** | AI can expand test scaffolding; correlated mistakes can exist when one system authors implementation and oracle | Human/independent specifications, mutation/adversarial cases, production reconciliation |
| Documentation generation | **Encourage** | Low implementation cost and usually reversible | Auto-merge for mechanically derived docs after consistency checks |
| AI PR review | **Advisory by default** | Useful additional reviewer, but should not become the sole assurance mechanism | Demonstrated recall on your defect corpus; deterministic checks still primary |
| AI opens PRs | **Good bounded-agent target** | Separates generation from integration and preserves normal review/deployment control | Sandboxed repo write, audit trail, ownership, CI |
| AI reacts to CI | **Good bounded-agent target** | Test feedback provides an external feedback loop; coding agents are especially tractable when outcomes are machine-verifiable. citeturn24view5 | Attempt limits, no permission escalation, protection against weakening tests |
| AI iterates until CI passes | **Conditional** | Can save human iteration but creates risk of optimizing against incomplete checks | Independent acceptance gates; prohibit deletion/weakening of safety checks without review |
| AI merges changes | **Only low-blast-radius classes initially** | Merge converts suggestion into organizational action | Branch protection, restricted change taxonomy, strong verifier, fast rollback |
| Schema changes | **Human approval for semantic/breaking changes** | Downstream compatibility and state can make failures widespread | Additive, contract-checked, staged changes could be delegated narrowly |
| Pipeline changes | **Branch autonomy; production approval retained** | Retries, incremental state and side effects complicate apparently small edits; Airflow explicitly recommends repeatable task outcomes and partition-specific reads/writes. citeturn24view6 | Idempotence, staging, replay tests, backfill simulation and rollback |
| Incident diagnosis | **High-value early use; preferably read-only** | Logs, code, lineage and runbooks create rich context without requiring write authority | Measured diagnostic accuracy and time-to-resolution gains |
| Production repair | **Human-triggered runbooks first** | Agentic systems introduce tool misuse, privilege, cascading-failure and human-trust risks. citeturn22search16turn22view2 | Objective trigger, pre/postconditions, idempotent action, narrow permissions, canary and automatic rollback |
| Autonomous production repair | **Exception, not default** | Worth considering only when a repair is safer and more objectively verifiable than waiting for a human | Long shadow history, deterministic guardrails, bounded blast radius and named human operational owner |

The first organizational boundary I would draw is therefore **not assistant versus agent**. It is:

**AI may reason broadly; its authority to act should remain narrow.**

An agent can be given extensive read access to repository structure, lineage, CI results, observability and runbooks while still having no production write authority. That captures much of the context-gathering and iteration benefit without coupling cognitive autonomy to operational privilege. Anthropic's own engineering guidance makes essentially the same architectural distinction indirectly: agents need environmental feedback, stopping conditions, sandbox testing and meaningful oversight, and increasing complexity should be justified by measured outcomes. citeturn24view5

**Deliverable 2 — Business problems AI-assisted data engineering can address**

AI is an intervention, not a diagnosis. Several problems on the leadership list are better attacked somewhere else first.

| Organizational problem | AI fit | Likely useful AI intervention | Non-AI alternative that should be compared | Decision |
|---|---|---|---|---|
| Repetitive SQL work | **High** | Draft transformations, translations, repetitive joins/CTEs, query explanation | Standard macros, semantic models, reusable transformation patterns | Use AI when variation is real; eliminate the work when it is merely duplication |
| Engineering backlog | **Conditional** | Accelerate implementation of already-well-specified work | Kill low-value backlog, simplify platform, reduce custom pipelines | Do not automate a backlog that should not exist |
| Documentation debt | **High for drafting** | Generate/update model, column, runbook and code documentation | Make documentation derivable from metadata and contracts | AI is attractive, especially if consistency can be machine-checked |
| Slow onboarding | **Moderate/high** | Repository Q&A, explanations, tracing dependencies, examples | Better architecture, docs, naming, ownership, paved roads | Use AI as interface; fix structural confusion separately |
| Large legacy estate | **Conditional** | Explain unfamiliar code, characterize dependencies, mechanical refactors | Decommission and consolidate | Beware assuming AI makes complexity free: METR's realistic RCT found no productivity gain for experienced maintainers in familiar large repositories in early 2025. citeturn17view2 |
| Migration workload | **High for mechanical portions** | Syntax/API conversions, repetitive refactoring, test generation | Standard migration tooling, compatibility layers, platform consolidation | Strong candidate where source and target behavior can be compared |
| Limited specialist capacity | **Moderate** | Amplify implementation and information retrieval | Training, hiring, architecture simplification | AI can make expertise more leveraged; it does not eliminate need for semantic ownership |
| Incident-resolution time | **High for diagnosis; conditional for action** | Search logs/code/lineage, form hypotheses, recommend/run bounded diagnostics | Better observability, runbooks, ownership and reliability engineering | Read-only AI is much easier to justify than autonomous repair |
| Inconsistent testing | **Moderate/high** | Propose fixtures, edge cases and tests | Standard test harnesses, contracts, CI requirements | AI can help create safety infrastructure, but should not self-certify it |
| Maintenance burden | **Conditional** | Bug fixes, modernization, dependency work | Delete software, standardize, buy instead of build | Prefer less software over cheaper production of more software |
| Warehouse cost | **Mixed** | Explain plans, identify repeated queries, suggest optimizations | Workload management, budgets, materialization design, consolidation | Measure actual compute; cheaper query generation can increase consumption |
| Unclear metrics/business semantics | **Low as primary intervention** | Surface inconsistencies and draft definitions | Semantic contracts, domain ownership, governance | Fix semantic authority before delegating semantic changes |

That last distinction is important. A language model can produce syntactically valid SQL for an underspecified metric faster than a human can type the wrong SQL. Nothing about generation speed resolves whether “active customer,” “net revenue,” or “conversion” means what the organization thinks it means.

## Evidence, value and economics

**Deliverable 3 — Evidence on productivity and quality**

The evidence stack should be read in descending order of relevance to the decision—not according to whichever headline reports the biggest percentage.

| Evidence | Result | Evidence class | Applicability to data engineering | Main limitation |
|---|---|---|---|---|
| Peng et al., controlled GitHub Copilot experiment | Treatment completed a standardized JavaScript HTTP-server task **55.8% faster**. citeturn17view5 | Controlled experiment | Evidence that code generation can materially accelerate bounded implementation tasks | Simple generic software task; not production data engineering |
| METR early-2025 RCT | 16 experienced developers, 246 real issues in repositories averaging >1M LOC; AI-enabled tasks took **19% longer**. Developers nevertheless believed it sped them up. citeturn17view2 | Randomized field experiment | Highly relevant warning for complex existing codebases and review/correction overhead | Experienced open-source maintainers; early-2025 tools; not data engineering |
| METR late-2025/early-2026 follow-up | Raw data suggested speedups for some cohorts, but METR says selection and measurement problems make the estimate unreliable; developers increasingly refused no-AI tasks or omitted AI-friendly tasks. citeturn17view0 | Attempted randomized field experiment | Shows why internal measurement needs to accommodate task substitution and concurrent agents | Cannot yield trustworthy aggregate uplift estimate |
| METR early-2026 technical-worker survey | 349 technical workers reported a median **1.4–2×** change in value of work attributable to AI, while METR explicitly cautions that the magnitude should be treated skeptically. citeturn23search2 | Self-report | Signals substantial perceived value and changed workflows | Not causal; perception can diverge from measured productivity |
| DORA generative-AI research | Extensive AI use was associated with perceived productivity, flow and satisfaction, yet a 25% increase in adoption was associated with **1.5% lower delivery throughput and 7.2% lower stability**; DORA attributes part of this to larger change batches. citeturn18view0 | Cross-organizational observational research/interviews | Strong evidence that individual productivity and system delivery outcomes can diverge | Software engineering broadly; observational, not data-specific causal result |
| DORA 2025 | AI behaves primarily as an **amplifier of existing organizational strengths and weaknesses**; the organizational system drives returns. citeturn18view1 | Large-scale industry research | Highly relevant to CI, testing, review and platform design decisions | Generic software delivery evidence |
| BIRD | Introduced 12,751 text-to-SQL pairs over 95 databases and 37 domains; early LLM performance remained far below proxy human performance. citeturn24view0 | Data-specific benchmark | Direct evidence on SQL generation | Text-to-SQL does not test dbt DAGs, enterprise definitions, permissions or deployment |
| ReViSQL 2026 | Reported 93.2% execution accuracy on expert-verified BIRD Mini-Dev; cleaning training data improved accuracy materially, and the researchers found errors in 61.1% of an audited subset of BIRD Train. citeturn19search7 | Data-specific benchmark/research system | Demonstrates how far SQL reasoning has progressed—and how important trustworthy specifications/data are | Benchmark success is not production lifecycle autonomy |
| SWE-bench | Original benchmark contains 2,294 real issues from 12 Python repositories; Verified is a 500-instance human-filtered subset. citeturn24view3 | Coding benchmark | Evidence that agents can operate across multi-file real issues | Passing benchmark tests says little about data correctness or organizational semantics |
| SWE-Lancer | >1,400 paid freelance tasks totaling $1M; evaluated models in the 2025 paper still failed the majority of tasks. citeturn24view4 | Realistic coding benchmark | Stronger external validity than toy coding tasks | Still generic software; capability is time-sensitive |
| Perry et al. | In a security-focused user study, participants with an AI assistant wrote significantly less secure solutions and were more likely to believe they were secure. citeturn24view1 | Controlled user study | Relevant to overconfidence and review | Old model generation; security tasks, not DE |
| Sandoval et al. | A separate C programming study found comparatively small security differences, illustrating that effects depend on task and workflow. citeturn24view2 | Controlled user study | Prevents overgeneralizing “AI code is insecure” | Student/low-level-C setting |

The important finding is not that AI is either fast or slow. It is that **treatment effects vary dramatically with task structure, user expertise, repository familiarity, validation quality and generation-era capability**. citeturn17view5turn17view2turn17view0

There is therefore no defensible universal multiplier such as “AI makes data engineers 30% faster.” The evidence needed for a Head of Data is the effect on **their portfolio of work**, including review and production consequences.

The evidence is particularly thin for several requested outcomes:

| Outcome | Evidence assessment |
|---|---|
| Code-generation speed | **Strong evidence of improvement in many bounded settings**, but heterogeneous. citeturn17view5turn17view2 |
| End-to-end engineering cycle time | **Mixed**; local coding gains need not improve delivery throughput. citeturn18view0 |
| Throughput | **Uncertain/system-dependent**; DORA found a negative association in one major study despite perceived individual productivity. citeturn18view0 |
| Code quality | **Mixed and task-dependent**; automated verification materially changes the equation. citeturn24view1turn24view2turn24view5 |
| SQL correctness | **Rapidly improving on benchmarks**, including near/proxy-human benchmark performance in recent research. citeturn19search7 |
| Business-logic correctness | **No comparable evidence found** that would justify autonomous enterprise semantic changes |
| Documentation | Mechanism is strong and task is easy to generate, but **production-level causal evidence is weaker than the rhetoric** |
| Onboarding | Plausible and supported indirectly by heterogeneous benefits for less-experienced developers in the Copilot experiment, but **not proven specifically for data teams**. citeturn17view5 |
| Incident resolution | Strong plausible mechanism; **little controlled data-engineering evidence** for autonomous action |
| Maintenance cost | **Unknown net effect** until added change volume, review, compute and future maintenance are included |
| Engineering cost | **No universal ROI**; DORA itself now frames ROI through organization-specific calculation rather than a universal uplift figure. citeturn18view2 |

**Deliverable 4 — Second-order effects of cheaper code generation**

The first-order model is:

> cheaper code → more engineering output.

The more realistic systems model is:

> cheaper generation → more candidate changes → more validation, review, deployment, data processing and maintenance → bottleneck may move rather than disappear.

DORA provides direct evidence for this possibility: higher AI adoption was associated with reduced delivery throughput and stability, and its analysis attributes part of the effect to AI enabling larger batches that take longer to review and are more destabilizing. It explicitly recommends reinforcing automated testing and fast feedback loops as generation accelerates. citeturn18view0

For data organizations the likely second-order channels are especially important:

| Cheaper activity | Possible gain | Possible induced cost |
|---|---|---|
| SQL generation | Faster exploration/model construction | More queries and warehouse compute |
| Model generation | Faster backlog throughput | More models, dependencies and semantic surface area |
| Pipeline generation | Faster integrations | More orchestration state and operational ownership |
| Test generation | Better coverage | More CI runtime and low-value/redundant tests |
| PR generation | More candidate changes | Reviewer queue becomes bottleneck |
| Refactoring | Easier modernization | Larger change surface and harder attribution |
| Documentation | More documented artifacts | Stale/generated prose if not tied to source metadata |
| Incident automation | Faster response | Faster propagation of a mistaken diagnosis |
| Optimization agents | More experiments | Token, CI and warehouse expenditure can become material; METR's 2026 work explicitly argues that agent evaluation must account for resource expenditure rather than only success/failure. citeturn17view1 |

A metric such as **AI PRs per engineer** is therefore nearly useless as an executive KPI.

Better denominators are:

**accepted production-stable changes per engineer-hour**,  
**reliable business capabilities delivered per total engineering hour**,  
**defect-adjusted cycle time**,  
**incident-free change throughput**, and  
**total lifecycle cost per useful data product**.

**Deliverable 13 — Lifecycle cost/value model**

Use the following model rather than a generic seat-license ROI calculation.

For a task class \(j\):

\[
C_{baseline,j}
=
N_j(H_{baseline,j}W)
+
C_{infra,j}
+
E[L_{incident,j}]
\]

\[
C_{AI,j}
=
N'_j
(H_{AI,j}W
+
C_{inference,j}
+
C_{CI,j}
+
C_{warehouse,j})
+
C_{license}
+
C_{integration}
+
C_{evaluation}
+
C_{security}
+
C_{governance}
+
C_{maintenance}
+
E[L'_{incident,j}]
\]

where:

- \(N_j\) is baseline task/change volume;
- \(N'_j\) is **AI-era volume**, deliberately allowed to differ from baseline;
- \(H_{baseline}\) includes implementation, review, debugging and operational effort;
- \(H_{AI}\) includes prompting, context preparation, waiting, review, correction, intervention and cleanup;
- \(W\) is loaded human labor cost;
- inference, CI and warehouse costs are metered separately;
- expected incident loss includes probability × severity × detection/recovery cost.

Then:

\[
NetValue =
(C_{baseline}-C_{AI})
+
V_{earlier\ delivery}
+
V_{backlog\ removed}
+
V_{onboarding}
+
V_{quality}
\]

Critically, **hours saved are not automatically financial value**. If an engineer saves ten hours and the organization merely produces ten hours' worth of low-value additional pipelines, the benefit is not equal to ten hours of salary. Value exists when capacity is redeployed to work that has an economic or reliability benefit.

A useful per-task break-even test is:

\[
RequiredTimeSaving
>
\frac{
\Delta inference+
\Delta CI+
\Delta warehouse+
\Delta review+
\Delta expected\ incident\ cost
}{W}
\]

This model should be run by task class rather than organization-wide because documentation generation, SQL authoring and autonomous repair have radically different economics.

DORA's 2026 ROI work similarly treats AI adoption as a calculation requiring local inputs and explicitly acknowledges an initial productivity dip rather than claiming automatic positive ROI. citeturn18view2

## Autonomy, risk and engineering controls

**Deliverable 5 — Autonomy options and trade-offs**

These are **independent operating modes, not stages every organization should climb**.

| Operating mode | Expected value | Review burden | Reliability / blast radius | Security & governance | Operational ownership |
|---|---|---|---|---|---|
| Suggestions only | Moderate | Low/moderate | Human mediates all action | Lowest new privilege | Engineer |
| Generates code; human applies | Moderate/high | Moderate | Easy to reverse before commit | Little additional machine authority | Engineer |
| Writes branch | High for repetitive implementation | Moderate | Repository impact isolated | Agent needs repo write permission | Engineer sponsoring task |
| Opens PR | High | Review can become bottleneck | Existing change-management path preserved | Strong auditability | Named PR owner |
| Reacts to CI | High where tests are meaningful | Lower iteration effort | Feedback loop improves correction | Must stop test/guardrail manipulation | PR owner |
| Iterates autonomously to passing checks | Potentially high | Lower execution / potentially harder final review | Risk of optimizing against incomplete tests | Sandbox, attempt/time budget, protected tests | Human owner |
| Merges bounded changes | Moderate incremental gain beyond PR agent | Lower explicit review; higher control burden | Blast radius enters shared system | Policy engine/branch protection essential | Service/data-product owner |
| Operates in production | Potential incident-response value | Humans shift from execution to supervision | High | Least privilege, audit, isolation and stopping conditions become mandatory | Human on-call/service owner |
| Performs selected production repairs | High only for well-defined failure classes | Low during common incidents | Safe only if action is narrow and reversible | Preapproved action catalog, pre/postconditions, automatic rollback | Human remains accountable |

Most organizations can plausibly capture **a large fraction of available benefit before automatic merge or unrestricted production writes**. That is an inference from the structure of the workflow rather than a published universal percentage: agents can already gather context, implement changes, execute tests and prepare PRs while existing deterministic delivery mechanisms retain integration authority. Coding-agent guidance emphasizes precisely these external feedback loops and notes that human review still matters for broader system requirements even when tests pass. citeturn24view5

SWE-bench and SWE-Lancer strengthen the case for bounded autonomy but not unrestricted autonomy. They demonstrate that models can attack realistic engineering tasks, while also showing that benchmarks define narrow success criteria and that realistic work remains imperfectly solved. citeturn24view3turn24view4

**Deliverable 6 — Risk / blast-radius map**

Do not classify changes by “AI-generated versus human-generated.” Classify them by **consequence and verifiability**.

| Risk | Typical data-engineering failure | Detection difficulty | Blast radius | Default authority |
|---|---|---|---|---|
| Incorrect syntax/API | Build fails | Easy | Low | Agent may self-correct |
| Local implementation bug | Incorrect output caught by tests | Moderate | Low/medium | Agent branch + CI |
| Incorrect business logic | Query faithfully computes wrong definition | Hard | Potentially enterprise-wide | Human semantic approval |
| Silent metric corruption | Dashboard/KPI remains plausible but wrong | Very hard | High | Human approval + reconciliation |
| Incremental-state bug | Duplicates, omissions, late-data mishandling | Hard | High/historical | Human approval until replay tests strong |
| Destructive migration | Columns/table/history lost | Usually obvious after damage | Very high | Explicit human control |
| Wrong backfill | Rewrites large historical range incorrectly | Hard/expensive | Very high | Human authorization + dry run |
| Excessive compute | Large scan/retry loop | Easy eventually, costly immediately | Financial/platform | Agent budget limits |
| Privacy/data exposure | Sensitive data enters model/tool/context | Potentially hard | High/regulatory | Explicit access policy |
| Permission/credential change | Agent expands own or others' authority | Hard/high-security | Very high | Human/security approval |
| Pipeline outage | Scheduler/dependency failure | Observable | Medium/high | Bounded repair only |
| Test manipulation | Agent weakens test to make CI pass | May look “green” | High | Protect acceptance tests |
| Technical-debt proliferation | Many locally reasonable artifacts | Slow | Platform-wide | Architecture/change-volume controls |
| Loss of understanding | Humans cease understanding system | Slow | Organizational | Rotation, review, ownership |
| Agent/tool compromise | Prompt/context redirects privileged tools | Hard | Depends on permissions | Least privilege/isolation |

Agent security adds genuinely new attack paths rather than merely ordinary coding errors. OWASP's 2026 agentic taxonomy includes agent-goal hijacking, tool misuse, identity/privilege abuse, supply-chain compromise, unexpected code execution, memory/context poisoning, insecure inter-agent communication, cascading failures and human-agent trust exploitation. citeturn22search16turn22view2 NIST's GenAI Risk Management Profile likewise treats generative AI through lifecycle risk management rather than assuming model output is trustworthy by default. citeturn24view7

Humans, of course, also introduce bad migrations, incorrect SQL and outages. The relevant comparison is therefore:

\[
Risk_{AI\ workflow} \quad vs. \quad Risk_{current\ human\ workflow}
\]

not AI error versus a fictional zero-error human baseline.

The distinctive concern is **speed × scale × authority**. A human might make a bad decision once. An agent that can issue hundreds of calls, modify many files and retry automatically can propagate one wrong premise faster. OWASP explicitly identifies cascading failures as an agentic risk. citeturn22search16

**Deliverable 7 — Deterministic safety-net requirements**

My strongest recommendation is to treat **deterministic verification capacity as the scarce complement to AI autonomy**.

DORA's evidence supports this direction: AI appears to amplify existing delivery systems, and its guidance specifically calls for stronger automated testing, continuous integration and fast review as code generation accelerates. citeturn18view1turn18view0 Agent-design guidance independently emphasizes ground-truth environmental feedback, stopping conditions, sandboxing and guardrails. citeturn24view5

For data engineering:

| Agent authority | Minimum safety net |
|---|---|
| Explain/suggest | Access policy, approved data handling, attribution to source context |
| Generate branch code | Reproducible environment, parser/compiler, linting, unit tests, secret scanning |
| Open PR | CI, protected main branch, named owner, complete diff/audit trail |
| Iterate on CI | Immutable independent acceptance tests, attempt/time/token limits |
| Change SQL/dbt semantics | Unit fixtures, data tests, output reconciliation, contract checks, lineage/impact analysis |
| Change incremental models | Replay/idempotency tests, late-arrival fixtures, duplicate/missing-row checks |
| Change schemas | Compatibility contract, downstream impact check, staged deployment, migration rollback |
| Execute backfills | Explicit date/range bound, row/byte/cost estimate, dry run, reconciliation and abort threshold |
| Merge automatically | Machine-verifiable low-risk taxonomy, deployment guardrails, rollback |
| Diagnose production | Read-only observability, lineage, logs, runbooks |
| Repair production | Allowlisted actions, least-privilege temporary credentials, pre/postconditions, canary, rate limit, immutable audit log and automatic rollback |

The controls are not hypothetical abstractions. dbt contracts can enforce expected column names/types before building and flag breaking contract changes; dbt unit tests can validate SQL logic on static inputs before production materialization and are explicitly recommended for complex, critical or previously buggy logic. citeturn21view0turn21view1

Likewise, Airflow's current best-practice guidance says tasks should behave like transactions, avoid incomplete results, produce the same result on retries, prefer UPSERT over retry-sensitive INSERT, and read/write explicit partitions rather than “latest” mutable data. Those characteristics are exactly what make automated recovery and agent execution safer. citeturn24view6

**Does AI create more value in organizations with strong controls, or compensate for weak ones?**

The best-supported answer is asymmetric:

**AI can help build missing controls, but higher autonomy depends on those controls already working.**

A weak-control team can use AI to draft tests, document models, expose undocumented dependencies and help create contracts. That is valuable remediation. But granting autonomous change authority because AI also generated the test suite is circular assurance. DORA's “amplifier” result points against treating AI as a substitute for delivery fundamentals. citeturn18view1turn18view0

**Deliverable 8 — Data-platform readiness**

Avoid the vague phrase “AI-ready data platform.” The following foundations matter for concrete reasons:

| Foundation | Necessary for assistants? | Necessary for bounded agents? | Why |
|---|---:|---:|---|
| Modular repository | Helpful | **Strongly helpful** | Narrows context and change scope |
| Repeatable build/test environment | Helpful | **Yes** | Agent needs reproducible feedback |
| Deterministic deployment | No | **Yes for merge/prod action** | Otherwise success cannot be reliably reproduced |
| Data/model contracts | Optional | **Yes for autonomous interface changes** | Gives machine-checkable compatibility rules; dbt explicitly verifies contracted model shapes. citeturn21view0 |
| Ownership metadata | Helpful | **Yes** | Agent needs escalation/approval target |
| Lineage/dependency graph | Helpful | **Strongly recommended for semantic/schema changes** | Needed to reason about downstream blast radius |
| Standard orchestration | No | Strongly helpful | Reduces tool variability and allows bounded actions |
| Idempotent/replayable pipelines | Good practice | **Yes for autonomous retry/repair** | Makes reversal and repeated execution predictable; Airflow guidance reinforces repeatable task behavior. citeturn24view6 |
| Staging/canary environment | Optional for suggestions | **Yes for risky autonomy** | Creates low-blast-radius validation surface |
| Observability | Helpful | **Yes for production actions** | Provides external state and postconditions |
| Rollback/recovery | Good practice | **Yes for auto-merge/prod** | Autonomy without recovery turns errors into irreversible events |
| Fine-grained permissions | Helpful | **Yes for agents with tools** | Limits tool-misuse and privilege risks highlighted by OWASP. citeturn22search16 |
| Metadata quality | Helpful | **Important** | Wrong context can produce confidently wrong changes; ReViSQL's benchmark work found data correction itself delivered large accuracy improvements. citeturn19search7 |

The genuinely necessary foundations are therefore not “an AI catalog” or “vectorize everything.” They are the same things that make human engineering dependable: **clear interfaces, reproducibility, machine-checkable invariants, controlled deployment and recovery**. The difference is that agents expose the weakness of missing controls more quickly because they can act faster.

## Review, skills and accountability

**Deliverable 9 — Human-review model**

“Review every AI-generated line” is not a sustainable governance strategy if change volume materially rises. DORA's evidence that AI can expand batch size while worsening review/delivery outcomes is exactly the warning: a process can remain nominally human-reviewed while its actual attention per change collapses. citeturn18view0

Review should move from provenance-based to **risk-based assurance**.

| Change class | Human review focus | Automated evidence expected |
|---|---|---|
| Generated docs/comments | Accuracy/sample review | Source consistency/link checks |
| Mechanical refactor | Intent and scope | Behavioral/unit equivalence |
| Test addition | Whether test proves a useful invariant | Test execution; preferably mutation/negative cases |
| Normal SQL/model | Business intent + output | Schema/data tests, reconciliation |
| Metric/business logic | Definition and downstream consequences | Fixtures, historical comparison, lineage |
| Incremental logic | State semantics | Replay, late-data, duplicate/missing checks |
| Schema migration | Compatibility and recovery | Contract diff, dependency impact, migration dry run |
| Backfill | Scope, financial/operational impact | Row counts, cost estimate, sample/full reconciliation |
| Production repair | Diagnosis and permissible action | Objective trigger/precondition/postcondition/rollback |

For every agent-authored nontrivial PR, require an **evidence bundle**, not an essay:

> intended behavioral change → affected assets → tests run → before/after data results → lineage impact → estimated compute → deployment/rollback plan → unresolved uncertainty.

That lets the human review **the claim and its proof**, not merely thousands of generated tokens.

For high-semantic-risk data changes, the reviewer should explicitly inspect results rather than syntax. A syntactically beautiful query can encode an incorrect business definition perfectly.

Sampling can be reasonable only where two conditions hold: failures are cheaply detectable after deployment, and individual changes have low blast radius. Sample review is a bad fit for access-control changes, destructive DDL or financial metric definitions.

**Deliverable 10 — Skills / staffing implications**

Current evidence is not strong enough to justify a staffing strategy based on “AI replaces junior engineers.” The 2023 controlled experiment did find heterogeneous effects suggesting potential benefits for people transitioning into software development, while METR's early-2025 study found experienced maintainers could actually become slower when using AI on familiar repositories. Those results argue against a single experience-level effect. citeturn17view5turn17view2

A more defensible skill-shift hypothesis is:

| Role | Likely less scarce | Likely more valuable |
|---|---|---|
| Junior engineer | Boilerplate syntax, first-pass implementation | Debugging, testing, reading unfamiliar systems, data reasoning, knowing when generated output is wrong |
| Analytics engineer | Repetitive SQL/YAML generation | Metric semantics, stakeholder translation, reconciliation, model design |
| Senior data engineer | Mechanical coding | System decomposition, edge cases, incremental/state reasoning, review and operational judgment |
| Data architect | Drawing individual implementation patterns | Contracts, boundaries, simplification, lifecycle governance |
| Platform engineer | Hand-building every integration | Safe execution environments, CI/evals, permissioning, observability, agent interfaces |
| Data leader | Tracking coding utilization | Portfolio selection, risk appetite, economics, ownership, evidence quality |

The potential new bottleneck is **judgment and verification capacity**. That follows from the DORA evidence that faster generation can expand batches and slow review, not from an assumption that all implementation effort disappears. citeturn18view0

There is also a genuine deskilling risk, but evidence is not yet strong enough to quantify it for data engineering. The prudent response is not to ban AI; it is to preserve deliberate human contact with failure modes. Engineers who approve incremental transformations should still be capable of explaining the state model. On-call staff should still understand the system that an agent proposes to repair.

A particularly bad staffing pattern would be removing the junior learning pipeline while simultaneously expecting future senior engineers to possess deep systems intuition that traditionally came from years of hands-on debugging. Nothing in the current controlled evidence demonstrates that organizations can safely eliminate that developmental path.

**Deliverable 11 — Accountability and ownership**

“The agent did it” should have the same governance status as “the compiler did it”: none.

| Artifact/action | Accountable human role |
|---|---|
| Generated code | Engineer accepting/sponsoring change |
| Business correctness | Data-product/domain owner plus change author |
| Generated test quality | Change owner/reviewer |
| Model/data contract | Owning team/domain |
| PR opened by agent | Named sponsoring engineer/service owner |
| Deployment | Team owning deploy policy |
| Production outcome | Service/data-product owner |
| Autonomous runbook | Owner who approved the runbook and permission envelope |
| Incident | Existing incident commander/service owner |
| Rollback | Operational owner/on-call role |
| Generated documentation | Owner of documented artifact |
| Agent configuration/evaluation | Platform/AI-enablement owner |
| Credentials and permission boundaries | Platform/security owner |

Separate **execution** from **accountability**. An agent can execute a repair without becoming accountable for it, just as an orchestrator can retry a job without owning the business result.

This also means every autonomous action needs attributable provenance:

**who authorized the capability; which identity executed it; which model/configuration was used; what inputs/context were available; which tools were called; which checks passed; and which human/service owns the result.**

That requirement aligns with NIST's lifecycle-oriented risk-management approach and OWASP's focus on agent privilege, tool use and cascading action. citeturn24view7turn22search16

## Platform choice, pilot and autonomy policy

**Deliverable 12 — Build / buy / platform factors**

Do not choose the category with “the most agentic” demo. Choose the narrowest architecture that supplies the needed context and actions with acceptable governance.

| Option | Strength | Weakness | Best fit |
|---|---|---|---|
| General coding assistant | Low adoption friction; portable across engineering work | Limited data metadata/context without integrations | Individual assistance and broad coding |
| Data-platform-native AI | Rich warehouse/catalog/runtime context | Platform coupling; context and actions may share a large trust boundary | SQL/model assistance and platform-specific operations |
| Repository coding agent | Strong repo/CI workflow; natural PR isolation | Data/runtime semantics may be incomplete | Refactors, tests, migration and issue resolution |
| Custom data-engineering agent | Exact context, tool and policy design | Highest integration/evaluation/maintenance burden | High-value recurring workflows unavailable elsewhere |
| Workflow automation around existing tools | Deterministic and governable | Less flexible than open-ended agent | Incident/runbook automation, migrations, policy-controlled repetitive jobs |

Score candidates on:

**context** — repo, catalog, contracts, lineage, CI, incidents;  
**authority** — exactly which reads/writes/actions can be granted;  
**isolation** — branch, sandbox and production separation;  
**identity** — distinct service identity and least privilege;  
**audit** — prompts/context/tool calls/diffs/actions retained appropriately;  
**evaluation** — can your own task suite be run continuously;  
**portability** — models/tools/context interfaces replaceable;  
**cost** — seats + inference + infrastructure + downstream compute;  
**integration ownership** — who maintains it after the demo;  
**failure mode** — what happens when model, metadata or tool invocation is wrong.

Open tool protocols can make model/tool separation easier, but protocol support is not itself a safety property. OWASP's agentic framework explicitly includes supply-chain and tool/identity risks, so every connected capability still needs a trust boundary. citeturn22search16

A practical build-versus-buy rule:

> **Buy commodity cognition; build only the organizational context, policy and control surfaces that are differentiating.**

Custom agents become attractive when the valuable part is your lineage, contracts, deployment process, incident taxonomy or domain-specific verifier—not because orchestration code is intrinsically strategic.

**Deliverable 14 — Bounded pilot design**

A useful pilot should answer a business question:

> “For which task classes does this workflow improve end-to-end engineering outcomes without increasing severity-adjusted risk or total lifecycle cost?”

Do **not** evaluate only whether the model can produce plausible code.

Use a crossover or matched-task design so engineer and task difficulty do not dominate the comparison. The need for careful local experimentation is reinforced by the starkly different effects observed between the Copilot experiment and METR's realistic repository study. citeturn17view5turn17view2

Test at least these task families:

| Task | Normal case | Hard/negative case |
|---|---|---|
| New SQL model | Clear specification | Ambiguous business term |
| Refactor | Well-tested model | Hidden downstream dependency |
| Test generation | Known transformation | Existing code and documentation share same wrong assumption |
| Documentation | Normal model | Stale or contradictory metadata |
| Bug fix | Reproducible bug | Misleading initial diagnosis |
| Incremental model | Simple key/time window | Late arrivals, duplicate events, replay |
| Pipeline failure | Known failure pattern | Symptoms caused by upstream source issue |
| Migration | Mechanical syntax/platform change | Behavior differs subtly across engines |
| Schema change | Additive column | Breaking type/removal |
| Backfill | Small bounded interval | Very large interval with cost/late-data constraints |
| Security/context | Benign repository | Malicious instruction embedded in retrieved context |

Use at least three treatment modes where feasible:

**human baseline**,  
**AI-assisted human**,  
**bounded agent operating in branch/PR**.

Do not initially mix production write autonomy into the productivity experiment; it changes the risk question too radically.

Measure the whole task:

| Metric | Definition |
|---|---|
| Task success | Acceptance criteria met independently |
| Semantic correctness | Business/output correctness, not just executable code |
| Human active time | Prompting + implementation + review + correction + incident effort |
| Wall-clock cycle time | Request to accepted production-ready change |
| Review time | Reviewer active minutes |
| Rework | Human edits/interventions after AI output |
| Defects | Pre-merge and post-merge, severity weighted |
| Change size | Files/models/SQL affected |
| CI expenditure | Runs × compute |
| Warehouse expenditure | Development/test/backfill query cost |
| Agent expenditure | Tokens/inference/tool runtime |
| Maintainability | Independent reviewer assessment and subsequent edits |
| Operational outcome | Failed deployment, rollback or incident |
| Human understanding | Owner can explain design/failure/rollback |
| Intervention rate | Fraction of agent attempts requiring rescue |

Run evaluators against explicit acceptance tests where possible, but include semantic human evaluation because benchmark research already demonstrates how much results depend on the quality of the verifier/reference data. ReViSQL's finding that correcting benchmark data alone materially improved performance is an unusually relevant reminder: **an evaluation is only as trustworthy as its oracle.** citeturn19search7

Record abandoned attempts and failures. Do not report only successful AI interactions.

Also track task substitution. METR's follow-up found that AI adoption changed which tasks developers were willing to submit and how they worked, undermining a naïve randomized comparison. Your pilot needs to capture not only “same task faster” but whether AI causes engineers to select materially different work. citeturn17view0

**Deliverable 15 — Autonomy-expansion criteria**

Do not promote autonomy because a new model scores better on a public coding benchmark. SWE-bench itself covers defined repository issues; those results do not certify your semantic environment. citeturn24view3

Move from **assistant → code generator** when:

- local task-success quality is at least acceptable relative to human work;
- saved implementation time is not consumed by correction;
- security/privacy controls are established;
- engineers remain accountable for output.

Move from **generator → PR agent** when:

- repository actions are sandboxed;
- every change has a named owner;
- CI is sufficiently representative;
- agent-created PRs show positive net end-to-end time after review;
- no material increase in defect severity occurs.

Move from **PR agent → bounded autonomous engineer** only for explicitly enumerated task classes when:

- a machine-readable success condition exists;
- failure is reversible;
- permission scope matches task scope;
- the system has demonstrated performance on representative hard and negative cases;
- the agent cannot silently weaken the verifier;
- cost/action/iteration limits exist;
- human intervention and override rates are acceptably low.

Move to **selected production action** only when:

- the action is pre-enumerated, not arbitrary shell/database authority;
- trigger, precondition and postcondition are objective;
- action is idempotent or otherwise demonstrably retry-safe;
- blast radius is bounded;
- rollback is fast and tested;
- shadow/canary history demonstrates that the automated decision is at least as safe as the current human process;
- operational ownership and escalation remain unambiguous.

No universal “95% success” threshold is defensible. A 99.9% system that can delete the warehouse may be unacceptable, while a 90% documentation drafter may be entirely useful. Required confidence must rise with expected loss and irreversibility.

**Evidence that should reduce autonomy:**

rising human correction time; rising escaped defects; degraded delivery stability; growing review queues; repeated test weakening; higher-than-planned warehouse/CI spend; unexpected tool calls; permission escalation attempts; model/version changes that have not been re-evaluated; materially changed repository/platform architecture; loss of rollback reliability; incidents attributable to incorrect agent reasoning; or evidence that human owners no longer understand what they approve.

Capability policies should therefore be **evaluation-bound, not model-brand-bound**. METR's changing results and measurement difficulties between early 2025 and 2026 illustrate how rapidly workflow effects can move. citeturn17view2turn17view0

**Deliverable 16 — Stop / redesign conditions**

Stop or redesign the deployment when any of these occurs:

| Signal | Interpretation | Response |
|---|---|---|
| Code/PR volume rises but lead time does not | Bottleneck moved to review/validation | Constrain generation; improve feedback; smaller changes |
| Review minutes per accepted change rise | AI is externalizing effort to reviewers | Narrow tasks/context; improve evidence bundle |
| More tests, same escaped defects | Test quantity is not assurance | Improve oracle/specification; use negative/mutation/reconciliation tests |
| Agent repeatedly edits tests to pass CI | Objective is being gamed | Protect verifier; separate implementation and acceptance authority |
| Warehouse spend grows faster than useful throughput | Cheap SQL created expensive execution | Query budgets/cost gates; consolidate artifacts |
| Model count/pipeline count expands without product value | AI lowered architectural friction too far | Introduce deletion/consolidation gate |
| Human approvers routinely skim large AI diffs | Human-in-the-loop exists only on paper | Reduce batch size or autonomy |
| Humans cannot explain critical pipelines | Organizational understanding deteriorating | Rotate hands-on ownership and require explainability |
| Incidents become faster but more severe | Automation increased blast radius | Return production actions to approval boundary |
| Agent needs broad permanent production credentials | Architecture cannot bound authority | Redesign tool interface before delegation |
| Evaluation success diverges from production | Benchmark/oracle mismatch | Replace evaluation suite with representative cases |
| Integration/maintenance burden consumes saved effort | Custom agent economics failed | Simplify or buy commodity capability |
| AI savings rely on producing more low-value work | Output metric is wrong | Re-anchor on platform/business outcomes |

**Deliverable 19 — Decision reversals**

Additional automation becomes counterproductive when **marginal verification, operational, compute or risk cost exceeds marginal implementation savings**.

Formally, reverse an autonomy increase when:

\[
\Delta C_{review}
+\Delta C_{compute}
+\Delta C_{maintenance}
+\Delta E[L_{incidents}]
>
\Delta C_{implementation}
+\Delta V_{business}
\]

The most common reversal patterns are likely to be:

**generation outruns review; autonomy outruns observability; action authority outruns reversibility; repository access outruns security boundaries; change volume outruns architectural discipline; and agent-generated verification becomes correlated with agent-generated implementation.**

The last failure is subtle. The easiest way for an autonomous coding loop to become apparently excellent is to let the same loop define both answer and exam.

## Executive questions and evidence ledger

**Deliverable 17 — Questions a Head of Data / CTO should answer before allowing AI to own more of the lifecycle**

| Question | Why it matters |
|---|---|
| What concrete bottleneck are we removing? | Prevents automation theater |
| Would deleting/simplifying the work create more value than accelerating it? | Architecture may dominate AI ROI |
| What task class—not job title—is being delegated? | Risk differs by activity |
| What is the human baseline for time, quality and incidents? | Without baseline there is no measured benefit |
| Can correctness be determined independently of the agent? | Central prerequisite for bounded autonomy |
| Which aspects are syntactic, data-semantic and business-semantic? | These have different verifiability |
| What happens if the change is wrong but plausible? | Captures silent data corruption |
| What is the maximum blast radius of one action? | Determines acceptable autonomy |
| Is the action reversible, and has rollback actually been tested? | “Rollback available” is not the same as recoverable |
| Can retries safely repeat the action? | Essential for autonomous recovery |
| Can the agent alter its own tests, guardrails or permissions? | Prevents self-certification |
| What repository/data/production credentials does it really need? | Least privilege |
| What sensitive data can enter model context? | Privacy/security boundary |
| Who is accountable when the agent opens or merges a PR? | Prevents accountability vacuum |
| What evidence will the reviewer receive besides a diff? | Makes scaled review feasible |
| Are we measuring review and correction time, not only generation time? | Avoids local-productivity illusion |
| Are CI and warehouse costs captured? | Prevents inference-only ROI accounting |
| What happens to change volume after generation gets cheaper? | Captures induced demand |
| What observable evidence will earn more autonomy? | Avoids autonomy by enthusiasm |
| What evidence automatically reduces autonomy? | Creates a real control loop |
| What happens after a model, tool or platform upgrade? | Capability is nonstationary |
| Does the human owner still understand the production system? | Protects long-run operational capacity |
| Would we permit the same action automatically if the implementation were written by a human? | Separates provenance anxiety from genuine risk |
| Conversely, are we accepting AI risk merely because a human clicked “approve”? | Detects ceremonial oversight |

The final executive test is:

> **If the AI were ten times faster tomorrow, would our platform become ten times more valuable—or would our review queue, warehouse bill and number of things to maintain simply become larger?**

If the latter is plausible, generation is not the current constraint.

**Deliverable 18 — Claim-to-primary-source table**

| Decision-relevant claim | Primary source | Evidence class | What it supports | What it does **not** support |
|---|---|---|---|---|
| AI can materially accelerate a bounded coding task | Peng et al./Microsoft Research: treatment completed a JS HTTP-server task 55.8% faster. citeturn17view5 | Controlled experiment | AI coding can have large positive local effects | Universal 55.8% engineering uplift |
| AI can slow expert real-world development | METR RCT: experienced maintainers took 19% longer with early-2025 AI. citeturn17view2 | Randomized field experiment | Benefits are task/context dependent; perception can be misleading | AI is generally unproductive |
| Current productivity measurement is getting harder | METR follow-up reports strong task/participant selection effects and unreliable current central estimate. citeturn17view0 | Field experiment methodological follow-up | Local pilots must measure task selection and concurrent-agent work | Exact 2026 productivity effect |
| Self-reported AI value can be very high | METR survey of 349 technical workers reports median 1.4–2× self-reported value change, with explicit caution. citeturn23search2 | Self-report | Adoption/perceived value is real and significant | Causal productivity or ROI |
| Individual gains can coexist with weaker system delivery | DORA found positive perceived individual outcomes but negative associations with throughput and stability as adoption rose. citeturn18view0 | Observational organizational research | Optimize the delivery system, not individual typing speed | Causal proof that AI always harms delivery |
| AI amplifies organizational conditions | DORA 2025 characterizes AI primarily as an amplifier of strengths and weaknesses. citeturn18view1 | Cross-organizational research | Strong engineering foundations complement AI | Exact causal size for data teams |
| SQL generation has advanced dramatically | ReViSQL reports 93.2% execution accuracy on expert-verified BIRD Mini-Dev. citeturn19search7 | Data-specific benchmark | AI can solve sophisticated text-to-SQL workloads | Autonomous dbt/pipeline/business logic |
| Data/specification quality strongly affects SQL AI | ReViSQL found errors in 61.1% of an audited BIRD Train subset and 8.2–13.9-point accuracy gains from improved data quality under its setup. citeturn19search7 | Data-specific research | Context/oracle quality is a first-class variable | Same gains in enterprise metadata |
| Early text-to-SQL remained far below human proxy performance | Original BIRD benchmark exposed substantial model-human gap on large, dirty, domain-rich databases. citeturn24view0 | Benchmark | SQL correctness is a substantive reasoning problem | Current frontier performance |
| Coding benchmarks represent bounded slices of engineering | SWE-bench is 2,294 GitHub issues from 12 Python repos; Verified contains 500 human-filtered cases. citeturn24view3 | Benchmark definition | Useful capability test | Enterprise production readiness |
| Realistic software tasks remained challenging to 2025 models | SWE-Lancer's >1,400 paid tasks totaled $1M; evaluated frontier models solved fewer than a majority. citeturn24view4 | Peer-reviewed benchmark | Real engineering remains harder than toy coding | Current 2026 model ceiling |
| Agents benefit from external feedback and bounded stopping | Agent-engineering guidance recommends environment feedback, checkpoints, stopping conditions, sandbox testing and guardrails. citeturn24view5 | Vendor engineering experience | Architecture of safer bounded agents | Independent causal evidence |
| Automated tests do not remove need for broader review | Coding-agent guidance notes tests can verify functionality while human review remains necessary for broader system requirements. citeturn24view5 | Engineering practitioner evidence | Tests + human semantic/system review | Every line needs manual review |
| Data contracts provide machine-checkable boundaries | dbt contracts verify returned model shape and can detect breaking changes. citeturn21view0 | Platform documentation | Contracts improve autonomous-change verifiability | Contracts guarantee business correctness |
| SQL unit tests can catch logic before production | dbt supports static-input unit tests and recommends them for complex/high-criticality logic and before major refactors. citeturn21view1 | Platform documentation | Deterministic pre-production validation | Unit tests prove real-data correctness |
| Pipelines need repeatable retry behavior | Airflow advises transaction-like tasks, repeatable retries, UPSERT and fixed partitions. citeturn24view6 | Orchestrator documentation | Idempotence/replayability are important foundations | Airflow practices alone make autonomy safe |
| Agent autonomy introduces distinct security risks | OWASP 2026 identifies goal hijack, tool misuse, privilege abuse, supply chain, code execution, cascading failure and related risks. citeturn22search16turn22view2 | Expert security framework | Least privilege and bounded actions are essential | Probability of those failures in a specific deployment |
| GenAI requires lifecycle risk management | NIST GenAI Profile extends AI RMF to design, development, use and evaluation of generative-AI systems. citeturn24view7 | Government risk framework | Continuous evaluation/governance | A prescriptive data-engineering autonomy threshold |
| AI-assisted coding can create overconfidence/security problems | Perry et al. found AI-assisted participants produced less secure solutions and were more likely to believe them secure. citeturn24view1 | Controlled user study | Human approval is not automatically independent assurance | Modern AI always makes code less secure |
| Security effects are context-dependent | Sandoval et al. found much smaller differences in another coding-assistant study. citeturn24view2 | Controlled user study | Avoid blanket claims about AI-generated quality | Security is not a concern |

### Bottom-line decision position

Taken together, the evidence supports a **wide deployment of assistance but a narrow initial deployment of authority**.

For a typical reasonably engineered data organization, I would permit AI to **explain systems, generate SQL/dbt/code, generate tests and documentation, perform advisory review, modify branches, open PRs, run diagnostics and iteratively respond to CI**—with humans remaining accountable for accepted changes. This captures activities where generation and iteration are expensive enough to matter but existing software-engineering boundaries still provide control. The heterogeneous experimental evidence strongly favors measuring these benefits locally rather than assuming a standard productivity multiplier. citeturn17view5turn17view2turn17view0

I would introduce **bounded autonomous merging only by change class**, beginning with changes whose correctness is externally machine-verifiable, blast radius is small and rollback is straightforward. Contracts, CI, tests and replayable pipelines make this substantially safer; DORA's evidence suggests that these existing organizational capabilities become more—not less—important as AI accelerates generation. citeturn18view1turn18view0turn21view0turn24view6

I would keep **business-semantic changes, material metric definitions, destructive migrations, broad backfills, privacy/permission changes and unbounded production writes under explicit human authority**. Selected autonomous production repair can be rational, but only when the repair itself is more deterministic than the diagnosis: a narrow action, objective preconditions and postconditions, restricted credentials, known blast radius and tested rollback. OWASP's current agentic-security work gives good reason not to collapse reasoning authority and production privilege into the same trust boundary. citeturn22search16turn22view2

The strategic mistake to avoid is equating cheaper implementation with cheaper engineering. DORA already provides evidence that accelerated generation can create larger batches, more review friction and lower system stability. Data platforms add another denominator—warehouse execution—so excess output can also produce direct compute cost and a larger long-term surface to own. citeturn18view0

The durable operating principle is therefore:

> **Automate generation liberally. Automate verification wherever possible. Delegate execution selectively. Automate authority only when the organization can independently detect failure, constrain its consequences and recover without relying on the same AI that caused it.**

That policy captures the upside of rapidly improving AI capability without turning percentage-of-work-done-by-AI into a goal—and keeps the optimization target where it belongs: **reliable, maintainable data-platform outcomes per unit of total engineering effort, cost and risk.**