# Safe AI Engineering for Modern Data Teams: Practitioner Evidence Dossier

**Research snapshot:** October 3, 2026<br>
**Audience:** data engineers, analytics engineers, data-platform engineers, and technical leads<br>
**Operational question:** How should a modern data team integrate AI coding assistants and agents into engineering workflows so that they can generate, test, review, and repair data-engineering changes without bypassing the controls that protect production data?

The central finding is straightforward:

> **Treat the AI agent as an unusually fast, untrusted engineer—not as a privileged production operator.**

Give it enough repository and metadata context to reason well; let it edit code and execute broadly inside disposable environments; force it through deterministic CI and independent correctness checks; and keep production authorization, destructive data mutation, business-semantic decisions, and high-risk recovery behind controls the model cannot override. This separation matters because contemporary coding agents can already inspect repositories, execute tests and linters, create branches and PRs, and use external tools, but current enterprise SQL benchmarks and agent-security research still show substantial gaps in semantic reliability and safe tool use. [1](https://docs.github.com/en/copilot/concepts/agents/coding-agent/about-coding-agent)

“Tests passed” should therefore mean **one piece of evidence**, not “the change is correct.” dbt data tests, for example, prove only the assertions encoded in those tests; a query passes when it produces no rows violating the particular assertion. They do not establish that the chosen grain, join semantics, business definition, incremental state transition, or downstream interpretation was correct in the first place. [2](https://docs.getdbt.com/docs/build/data-tests)

## Operating model and reference workflow

**AI-assisted data-engineering reference workflow.** The recommended lifecycle is:

**requirement → context discovery → design → generation → static/local validation → data validation → CI → review → merge authorization → deployment authorization → production validation → monitoring → rollback/recovery**

The important architectural distinction is that **capability and authority are separate axes**. An agent may technically be capable of running `DROP TABLE`, merging a PR, triggering an Airflow backfill, or obtaining warehouse credentials; that is not a reason to authorize those capabilities. Current coding-agent platforms already separate repository editing from review and PR workflow, while GitHub rulesets and protected deployment environments provide independently enforced controls around merge and deployment. [1](https://docs.github.com/en/copilot/concepts/agents/coding-agent/about-coding-agent)

<table header-row="true">
	<tr>
		<td>Lifecycle stage</td>
		<td>Appropriate AI participation</td>
		<td>Authoritative control</td>
		<td>Evidence required before advancing</td>
	</tr>
	<tr>
		<td>Requirement</td>
		<td>Restate request; identify entities, grain, time semantics, ambiguities and acceptance criteria</td>
		<td>**Human/product/data owner when semantics are ambiguous**</td>
		<td>Explicit grain, keys, time window, metric/business definitions, assumptions</td>
	</tr>
	<tr>
		<td>Repository/context discovery</td>
		<td>Search relevant SQL/Python/YAML/macros/tests/orchestration, dependency graph, ownership and CI</td>
		<td>Repository ACLs and trusted context broker</td>
		<td>Context manifest recording what was read and why</td>
	</tr>
	<tr>
		<td>Proposed design</td>
		<td>Suggest affected models, lineage impact, migration strategy, test plan and risk tier</td>
		<td>**Human for business semantics or risky architecture changes**</td>
		<td>Impact analysis and unresolved questions</td>
	</tr>
	<tr>
		<td>Code generation</td>
		<td>Write SQL/Python/dbt/orchestration changes on task branch</td>
		<td>Filesystem/path policy</td>
		<td>Bounded diff; no protected-file edits without escalation</td>
	</tr>
	<tr>
		<td>Local/static validation</td>
		<td>Compile, parse, lint, type-check, inspect DAG and dependency changes</td>
		<td>Deterministic tooling</td>
		<td>Command log and outputs</td>
	</tr>
	<tr>
		<td>Data tests</td>
		<td>Generate fixtures/tests and execute against isolated data</td>
		<td>Independent oracle—not agent confidence</td>
		<td>Contract tests + business invariants + independent reconciliation</td>
	</tr>
	<tr>
		<td>CI</td>
		<td>Run deterministic policy, build, security, unit, data, cost and regression checks</td>
		<td>**CI policy**</td>
		<td>Required status checks</td>
	</tr>
	<tr>
		<td>Review</td>
		<td>Explain diff; perform secondary AI review; answer reviewer questions</td>
		<td>**Human/independent policy for material data changes**</td>
		<td>Evidence bundle, not just prose</td>
	</tr>
	<tr>
		<td>Merge</td>
		<td>Agent may request/queue merge</td>
		<td>**Repository ruleset**</td>
		<td>Risk-tier policy satisfied</td>
	</tr>
	<tr>
		<td>Deployment</td>
		<td>Trigger controlled pipeline where allowed</td>
		<td>**Deployment environment / change authority**</td>
		<td>Artifact identity + approvals + rollback capability</td>
	</tr>
	<tr>
		<td>Data validation</td>
		<td>Compare production outcomes read-only with baselines and expectations</td>
		<td>Data-quality gates / owner for ambiguous deviations</td>
		<td>Counts, metrics, distribution, schema and lineage comparisons</td>
	</tr>
	<tr>
		<td>Monitoring</td>
		<td>Triage anomalies, correlate run/deploy/lineage evidence</td>
		<td>Monitoring system</td>
		<td>Traceable observations</td>
	</tr>
	<tr>
		<td>Rollback/recovery</td>
		<td>Prepare commands/PR; execute only pre-authorized reversible actions</td>
		<td>**Human incident authority for destructive or data-mutating recovery**</td>
		<td>Confirmed trigger, known-good target, recovery validation</td>
	</tr>

</table>

This structure mirrors controls already present in mature software/data tooling. GitHub’s coding agent works in its own ephemeral development environment, can run tests and linters, commits work to a branch, and exposes logs and changes for review rather than treating generation as equivalent to approval. dbt CI similarly builds changed assets and affected downstream resources in a PR-specific temporary schema rather than directly materializing the PR into production. [1](https://docs.github.com/en/copilot/concepts/agents/coding-agent/about-coding-agent)

**Stages where AI should not be authoritative.** There are five especially important boundaries.

First, **missing business semantics** are not an inference problem to be solved by confidence. If “customer,” “active,” “revenue,” “as of,” row grain, late-arrival policy, or attribution logic is underspecified, the correct agent outcome is clarification or escalation.

Second, AI should not be authoritative for **breaking schema compatibility** or deciding whether an apparently unused field is safe to drop. dbt contracts can enforce declared shape, and its manifest exposes nodes, sources and dependency maps, but declared lineage is still not guaranteed to enumerate every external consumer. [3](https://docs.getdbt.com/reference/artifacts/manifest-json)

Third, AI should not independently authorize **production backfills, destructive migrations, or other historical data mutations**. A backfill is an execution against state and history, not merely a code deployment; Airflow explicitly models date range, reprocessing policy and independent concurrency for backfills. [4](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/backfill.html)

Fourth, **incident hypotheses are not root causes**. An agent can gather evidence and generate hypotheses rapidly, but a plausible explanation should not itself unlock production-write permissions.

Fifth, the model should not be capable of overriding its own **permission, merge, deployment or budget policy**. This is more than a design preference: modern agent tooling explicitly implements tool permissions outside model instructions, and GitHub likewise enforces rulesets and protected environments outside the code being reviewed. [5](https://code.claude.com/docs/en/permissions)

A practical autonomy ladder follows:

<table header-row="true">
	<tr>
		<td>Risk class</td>
		<td>Examples</td>
		<td>Recommended autonomy</td>
	</tr>
	<tr>
		<td>**Purely informational**</td>
		<td>Explanation, lineage lookup, docs analysis, log summarization</td>
		<td>Fully autonomous read-only, subject to data-access policy</td>
	</tr>
	<tr>
		<td>**Isolated reversible edit**</td>
		<td>Documentation, comments, non-runtime metadata, test fixture</td>
		<td>Generate, validate and open PR automatically; tightly scoped auto-merge may be acceptable</td>
	</tr>
	<tr>
		<td>**Transformation logic**</td>
		<td>SQL model, Python transform, dbt test, ordinary macro</td>
		<td>Generate/test/open PR; deterministic CI; independent semantic evidence; human approval for merge by default</td>
	</tr>
	<tr>
		<td>**Stateful or operational logic**</td>
		<td>Incremental model, snapshot, orchestrator schedule/retry/concurrency, contract migration</td>
		<td>Generate and simulate; mandatory knowledgeable reviewer</td>
	</tr>
	<tr>
		<td>**Production mutation**</td>
		<td>Backfill, delete/drop, corrective DML, replay, schema contraction</td>
		<td>Agent proposes; explicit authorized execution mechanism and human approval</td>
	</tr>
	<tr>
		<td>**Security/control-plane change**</td>
		<td>CI workflow, permissions, CODEOWNERS, credentials, branch rules, deployment policy</td>
		<td>Agent may explain or propose diff; never self-authorizes</td>
	</tr>

</table>

Documentation-only auto-merge is reasonable **only when “documentation” is path-classified rather than guessed by the model**. YAML, examples containing executable SQL, CI files, orchestration configuration, macros and policy files should not inherit a “docs” exemption merely because an LLM describes them as documentation. That is a recommended policy derived from the general principle that branch rules and deployment protections are enforced independently of the actor. [6](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-rulesets/about-rulesets)

## Context, isolation, permissions and cost boundaries

**Repository and data context model.** An effective data-engineering agent requires considerably more than a vector search over SQL files. Spider 2.0 is useful evidence here: its 632 enterprise-derived text-to-SQL workflow tasks frequently require searching database metadata, dialect documentation and project code; databases often contain more than 1,000 columns, and the paper’s code-agent baseline solved only 21.3% of tasks versus much higher results on earlier, simpler text-to-SQL benchmarks. The exact benchmark score is historical rather than a prediction for 2026 models, but the underlying lesson remains important: real data engineering is a **context-selection and systems-reasoning problem**, not merely query completion. [7](https://arxiv.org/abs/2411.07763)

Do not blindly send the whole repository. Build a **context broker** that derives a task-specific context package.

A good package has three layers:

<table header-row="true">
	<tr>
		<td>Context layer</td>
		<td>Contents</td>
		<td>Selection mechanism</td>
	</tr>
	<tr>
		<td>Repository map</td>
		<td>Directory structure, languages, build/test/lint commands, project conventions, CI entry points, deploy path, ownership</td>
		<td>Small stable checked-in agent guide + generated repository index</td>
	</tr>
	<tr>
		<td>Dependency neighborhood</td>
		<td>Target SQL/Python/model, parent/child models, macros invoked, tests, YAML properties, orchestration definition, relevant utilities</td>
		<td>Syntax/import references + dbt manifest/lineage graph + ownership graph</td>
	</tr>
	<tr>
		<td>Task evidence</td>
		<td>Schemas, grain declaration, contracts, statistics, selected history, incidents, semantic definitions, sample/fixture data</td>
		<td>Permission-aware metadata retrieval tied to requested entities</td>
	</tr>

</table>

GitHub’s own repository-instruction guidance recommends supplying project summary, languages, build/test/lint commands, layout, CI checks, dependencies and key files rather than expecting an agent to rediscover everything from scratch. It supports repository-wide instructions, path-specific instructions and nearest-path agent instructions. [8](https://docs.github.com/en/copilot/customizing-copilot/adding-repository-custom-instructions-for-github-copilot)

For dbt projects, **`manifest.json` should be treated as a first-class machine context source**, not reconstructed from SQL by the LLM. It represents models, tests, macros and other project resources, together with configuration and parent/child maps. This gives an agent a deterministic graph of declared dependencies and considerably reduces context waste. [3](https://docs.getdbt.com/reference/artifacts/manifest-json)

Likewise, **`run_results.json` belongs in post-execution context**: it associates executed nodes with status, timing, compiled code, relation information and adapter responses such as affected rows or warehouse-specific execution metadata. [9](https://docs.getdbt.com/reference/artifacts/run-results-json)

The context broker should distinguish **trusted policy from untrusted evidence**. A checked-in organization policy signed or protected by CODEOWNERS is instruction; a SQL comment, GitHub issue, README copied from an external dependency, warehouse column description, incident log or web page is data—even if it contains imperative language. That distinction must be enforced outside the model because repository and externally retrieved content can carry prompt-injection payloads. IssueTrojanBench, published in July 2026, found that 66.5% of its constructed malicious issue cases penetrated all tested agent/model guardrails across the tested coding-agent configurations; the paper should not be extrapolated to all repositories or all future agents, but it is strong evidence that model judgment alone is not a sufficient security boundary. [10](https://arxiv.org/html/2607.20759v1)

A public OpenHands issue provides weaker but useful practitioner evidence: researchers reported that malicious content retrieved through a web-search integration could influence generated output because retrieved content was passed into the agent context. This is a **single issue report, not controlled production evidence**, but it illustrates the same instruction/data boundary problem. [11](https://github.com/OpenHands/OpenHands/issues/10939)

**Data context permissions.** Use a metadata-first escalation ladder:

<table header-row="true">
	<tr>
		<td>Information</td>
		<td>Default agent permission</td>
		<td>Why</td>
	</tr>
	<tr>
		<td>Schema names/types</td>
		<td>Read</td>
		<td>Necessary for almost all data work</td>
	</tr>
	<tr>
		<td>Declared keys/grain/contracts</td>
		<td>Read</td>
		<td>Essential semantic context</td>
	</tr>
	<tr>
		<td>Lineage / dbt manifest</td>
		<td>Read</td>
		<td>Needed for blast-radius analysis</td>
	</tr>
	<tr>
		<td>Data tests and historical test status</td>
		<td>Read</td>
		<td>Helps identify established invariants</td>
	</tr>
	<tr>
		<td>Aggregate statistics</td>
		<td>Read with dataset scope</td>
		<td>High diagnostic value with lower exposure than rows</td>
	</tr>
	<tr>
		<td>Freshness/volume history</td>
		<td>Read</td>
		<td>Needed for anomaly and incident reasoning</td>
	</tr>
	<tr>
		<td>Semantic definitions</td>
		<td>Read</td>
		<td>Prevents accidental reinvention of metrics</td>
	</tr>
	<tr>
		<td>Warehouse execution metadata</td>
		<td>Read, scoped</td>
		<td>Useful for performance and incident diagnosis</td>
	</tr>
	<tr>
		<td>Query history</td>
		<td>Scoped and sanitized where needed</td>
		<td>Potentially sensitive operational context</td>
	</tr>
	<tr>
		<td>Synthetic fixtures</td>
		<td>Read/write in sandbox</td>
		<td>Preferred execution material</td>
	</tr>
	<tr>
		<td>Masked production-derived sample</td>
		<td>Read, time-limited</td>
		<td>Use only where fixtures cannot reproduce behavior</td>
	</tr>
	<tr>
		<td>Raw production rows</td>
		<td>Deny by default; JIT narrow read</td>
		<td>Only when genuinely necessary for diagnosis/reconciliation</td>
	</tr>
	<tr>
		<td>Production DML/DDL</td>
		<td>No standing credential</td>
		<td>Invoke only through separately authorized workflow</td>
	</tr>

</table>

BigQuery supports policy-based data masking at query time, and Snowflake supports masking policies that vary by role or entitlement. These mechanisms are better foundations for agent data access than handing the agent a high-privilege warehouse credential and asking it to behave. [12](https://cloud.google.com/bigquery/docs/column-data-masking-intro)

**When is production data actually necessary?** Much less often than teams initially assume. Code generation, compilation, unit tests, join-cardinality checks, incremental state tests and most schema validation can use metadata, fixtures, synthetic edge cases or a masked snapshot. Raw production reads become defensible for investigations where the defect is distribution-specific, for post-deployment reconciliation against actual outputs, or when diagnosing an incident that cannot be reproduced from retained fixtures. Even then, the recommended capability is **bounded read**, not broad production access.

**Safe execution environments.** The default environment should combine five isolation boundaries rather than choosing one:

**ephemeral compute sandbox + isolated Git branch/worktree + PR-specific database/schema + non-production data + restricted network/credentials.**

GitHub’s coding-agent architecture provides a useful precedent: work occurs in an ephemeral environment and branch before a PR is reviewed. dbt CI similarly creates a temporary schema unique to the PR, builds changed resources and affected downstream resources there, and removes that schema after the PR closes or merges under normal configuration. [1](https://docs.github.com/en/copilot/concepts/agents/coding-agent/about-coding-agent)

For more realistic stateful tests, dbt cloning can create development copies of production relations on supported warehouses, including zero-copy behavior where supported. This can be especially valuable for incremental CI because testing an incremental model against an empty schema may miss state-dependent behavior. [13](https://docs.getdbt.com/reference/commands/clone)

One important trap: **a development target does not imply production isolation**. dbt deferral can resolve references for unselected resources to a state environment, which may be production, and tests can therefore cross development/production boundaries unexpectedly. Agents need an explicit “relations actually read” trace, not just a target-schema name. [14](https://docs.getdbt.com/reference/node-selection/defer)

For orchestration code, Airflow’s current guidance likewise recommends treating DAGs as production-level code, testing them in an environment corresponding to the scheduler, and using a staging environment for complete DAG runs where possible. [15](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html)

**Security and permission model.** The most important security principle is that permissions must be enforced by the runtime, not by a prompt saying “please don’t do dangerous things.” Claude Code’s documentation makes this separation explicit: permission rules are enforced by the client rather than the model, with deny/ask/allow precedence; its own security guidance recommends filesystem/network sandboxing and warns that approved shell execution can otherwise escape ordinary working-directory boundaries. [5](https://code.claude.com/docs/en/permissions)

A production-grade agent runtime should therefore have:

<table header-row="true">
	<tr>
		<td>Boundary</td>
		<td>Recommended control</td>
	</tr>
	<tr>
		<td>Repository</td>
		<td>Task branch/worktree; restricted writable paths; branch rules</td>
	</tr>
	<tr>
		<td>Shell</td>
		<td>Full convenience inside disposable sandbox; no host or control-plane credentials</td>
	</tr>
	<tr>
		<td>Network</td>
		<td>Deny by default or egress allowlist</td>
	</tr>
	<tr>
		<td>Warehouse</td>
		<td>Purpose-built credential; read/write split; dataset/schema allowlist</td>
	</tr>
	<tr>
		<td>Production write</td>
		<td>No generic SQL credential; approved job/API exposing narrow operations</td>
	</tr>
	<tr>
		<td>Cloud</td>
		<td>Short-lived scoped workload identity rather than developer credentials</td>
	</tr>
	<tr>
		<td>Secrets</td>
		<td>Never inserted into model context unless absolutely necessary; preferably tool-side only</td>
	</tr>
	<tr>
		<td>CI</td>
		<td>Minimal token permissions; workflows protected by code ownership</td>
	</tr>
	<tr>
		<td>Dependencies</td>
		<td>Lockfile + allowlisted registries; approval for material supply-chain changes</td>
	</tr>
	<tr>
		<td>External tools/MCP</td>
		<td>Explicit allowlist and least privilege</td>
	</tr>
	<tr>
		<td>Security policy</td>
		<td>Managed outside writable task repository</td>
	</tr>

</table>

GitHub’s Actions documentation independently reinforces these controls: it advises minimal `GITHUB_TOKEN` permissions, warns that redaction is not a complete secret-protection guarantee, recommends avoiding direct interpolation of untrusted context into generated shell scripts, and recommends pinning third-party actions to full commit SHAs because mutable tags can change. [16](https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions)

Agent-readable repository content is itself part of the attack surface. Security-sensitive files such as CI workflows, CODEOWNERS, agent instruction files, permission configuration and deployment scripts should therefore have **stronger review requirements than ordinary transformation code**, not weaker requirements because “the AI understands them.”

**Cost controls.** Autonomous loops need budgets at every resource boundary.

For model execution, enforce maximum iterations, token spend, wall-clock duration, parallel subagents and repeated identical failures. For CI, cap job duration and concurrency. For warehouses, enforce bytes scanned or compute limits before execution where possible. BigQuery supports dry runs/validation to estimate bytes processed and supports project/user query quotas; Snowflake resource monitors can notify or suspend warehouses when credit thresholds are reached, though Snowflake cautions that resource-monitor enforcement is not perfectly instantaneous and therefore benefits from safety margin. [17](https://cloud.google.com/bigquery/docs/best-practices-costs)

Backfills need a **separate operational budget**, because a cheap code change can trigger an expensive historical reprocessing job. Airflow explicitly provides backfill date ranges, dry-run date enumeration and independent `max_active_runs`, while pools can bound concurrent tasks that contend for scarce resources. [4](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/backfill.html)

A useful practical rule is:

> The agent may optimize within a budget; it may not redefine its budget because the task is taking longer than expected.

## Generation, correctness and stateful data controls

**SQL / dbt generation patterns.** The highest-leverage change is to stop asking agents directly for SQL from prose. Require an intermediate **data-change specification**:

`output grain → business keys → inputs → join cardinalities → time semantics → null policy → incremental state transition → schema delta → expected reconciliation → performance envelope`

Only then should generation begin.

This matters because enterprise SQL tasks are demonstrably harder than conventional text-to-SQL examples. Spider 2.0 attributes the challenge to large schemas, multiple dialects, metadata search, project context and multi-query workflows. A 2026 research system, ProSPy, similarly uses data profiling and progressive schema pruning rather than attempting to stuff every available table into one prompt. These are benchmark/research findings, not evidence that any one retrieval strategy is optimal for a specific warehouse, but they strongly support **selective context plus execution evidence** over indiscriminate repository dumping. [7](https://arxiv.org/abs/2411.07763)

For every generated SQL transformation, the validation plan should explicitly probe:

<table header-row="true">
	<tr>
		<td>Risk</td>
		<td>Required check</td>
	</tr>
	<tr>
		<td>Wrong grain</td>
		<td>Assert expected key uniqueness and expected row grain</td>
	</tr>
	<tr>
		<td>Join fanout</td>
		<td>Measure pre/post row count and distinct business keys; inspect multiplicity on each join key</td>
	</tr>
	<tr>
		<td>Missing records</td>
		<td>Anti-join expected population to result</td>
	</tr>
	<tr>
		<td>Null semantics</td>
		<td>Fixtures containing null keys, measures and timestamps</td>
	</tr>
	<tr>
		<td>Filter boundaries</td>
		<td>Include records immediately before/at/after boundaries</td>
	</tr>
	<tr>
		<td>Windows</td>
		<td>Test ties, duplicate timestamps, null ordering and partition boundaries</td>
	</tr>
	<tr>
		<td>Aggregation</td>
		<td>Reconcile totals independently before and after grouping</td>
	</tr>
	<tr>
		<td>Dialect</td>
		<td>Compile/parse against the actual adapter/dialect</td>
	</tr>
	<tr>
		<td>Partitioning</td>
		<td>Confirm predicate pruning and partition key behavior</td>
	</tr>
	<tr>
		<td>Incremental logic</td>
		<td>Compare full-build and stateful incremental outcome</td>
	</tr>
	<tr>
		<td>Performance</td>
		<td>Dry-run/explain/warehouse metrics against budget</td>
	</tr>

</table>

**One-shot versus iterative generation.** One-shot generation is reasonable for very small, fully specified transformations with strong fixtures. For non-trivial transformations, prefer a bounded loop:

`generate → compile → execute in isolation → inspect deterministic evidence → modify → repeat`

But do not confuse iteration with reliability. Research on execution-feedback code generation has found that ordinary LLMs may struggle to improve reliably from repeated execution feedback compared with independent sampling; RLEF was explicitly motivated by training models to use such feedback better. The study concerns programming benchmarks, not data pipelines, so this is **indirect empirical evidence**, but it argues against “keep retrying until green” as a safety strategy. [18](https://arxiv.org/abs/2410.02089)

A sensible repair loop therefore has a low iteration cap, records each hypothesis, and stops when successive attempts fail for the same reason or evidence becomes ambiguous.

**dbt-specific support.** An agent can productively generate or modify models, source definitions, tests, docs, snapshots, incrementals, macros and refactors when it receives machine-readable context. The dbt manifest provides resources and dependency maps; run results capture actual execution outcomes; model contracts define declared shape; CI provides PR-isolated builds. [3](https://docs.getdbt.com/reference/artifacts/manifest-json)

Do not permit macro changes to be judged by a single direct caller. Macros can alter many compiled models, so CI should compare the **compiled affected graph**, not merely the source-file diff. That recommendation follows from dbt’s manifest/dependency model rather than from an assertion that all macro dependencies can always be statically determined. [3](https://docs.getdbt.com/reference/artifacts/manifest-json)

**Test and reconciliation patterns.** Use three independent classes of correctness evidence:

**Specification tests** encode what must be true: uniqueness, not-null, accepted values, relationships and explicit business invariants. dbt provides generic tests for the first four categories and allows arbitrary SQL assertions for business-specific conditions. [2](https://docs.getdbt.com/docs/build/data-tests)

**Regression/golden tests** compare the changed system against externally fixed expected behavior: fixture outputs, curated golden query results, or old-versus-new results over the same snapshot.

**Operational checks** look for unintended consequences: row count, reconciliation totals, freshness, execution time, scan volume, slot/credit consumption and downstream impact.

The agent can generate all three, but **AI-generated tests should not be the only oracle for AI-generated implementation**. The same misunderstanding can appear in both. This concern is strengthened indirectly by LLM-as-judge research finding measurable self-preference effects when models evaluate their own style of output; that literature is not a coding-review experiment, so it should be treated as evidence for evaluator independence rather than proof that the same defect will necessarily be missed. [19](https://arxiv.org/abs/2410.21819)

A strong PR therefore needs at least one oracle that does not originate from the implementation-generation step: a human-specified invariant, historical result, independent fixture, contract, downstream reconciliation or separately curated evaluator.

**Golden / regression data.** Use a progression from cheap to realistic:

`handwritten fixture → synthetic adversarial fixture → masked representative snapshot → fixed production-derived snapshot → bounded production reconciliation`

Fixtures should deliberately contain situations that ordinary examples omit: duplicate keys, null keys, many-to-many joins, identical timestamps, out-of-order events, deleted records, late updates, empty partitions, daylight-saving boundaries where relevant, schema evolution and partial-history windows.

For ordinary model changes, the strongest practical CI oracle is often an **old-versus-new differential on the same fixed input**, coupled with an explicit allowlist of intentional differences. “Outputs differ” is not necessarily failure; “outputs differ and nobody can explain exactly why” should be.

**Schema-change controls.** Every schema change should produce an explicit impact artifact before review:

`column added/removed/renamed/type-changed/nullability-changed → affected models → downstream contracts → exposed consumers → migration strategy → backfill requirement → rollback compatibility`

dbt model contracts can cause a build to fail when the materialized model does not conform to its declared columns and data types, making them valuable enforceable controls. dbt also warns that contracts impose maintenance/rollback trade-offs, which is another reason not to turn agent-generated contracts into unquestioned authority. [20](https://docs.getdbt.com/docs/mesh/govern/model-contracts)

For breaking changes, prefer **expand → migrate → observe → contract**:

add the replacement field/model; support old and new consumer paths; migrate consumers; verify usage; only then remove the old shape.

Do not confuse warehouse schema synchronization with historical correctness. dbt’s `on_schema_change` behavior can append or synchronize columns depending on configuration, but the documentation explicitly notes that schema-change handling does **not** backfill historical records; nested changes also have limitations. [21](https://docs.getdbt.com/docs/build/incremental-models)

Renames should therefore be treated as remove-plus-add from the consumer’s perspective unless your platform provides an explicit compatibility mechanism.

**Incremental / backfill controls.** Incremental processing deserves a higher risk tier than ordinary SQL because correctness depends on hidden temporal state. dbt itself requires incremental SQL to work in both full and incremental modes, and its `unique_key` defines how incoming records correspond to existing records. The documentation warns that nulls or non-unique values in the key can produce duplicates or failures and that changes to transformation logic may leave historical data inconsistent until a full refresh. [21](https://docs.getdbt.com/docs/build/incremental-models)

Every incremental change should pass this state-transition suite:

<table header-row="true">
	<tr>
		<td>Scenario</td>
		<td>Required property</td>
	</tr>
	<tr>
		<td>First run</td>
		<td>Same expected relation as full build</td>
	</tr>
	<tr>
		<td>No-new-data run</td>
		<td>Idempotent; no row or metric drift</td>
	</tr>
	<tr>
		<td>Ordinary incremental run</td>
		<td>Adds/updates exactly expected records</td>
	</tr>
	<tr>
		<td>Late-arriving event</td>
		<td>Included according to explicit lateness policy</td>
	</tr>
	<tr>
		<td>Duplicate event</td>
		<td>Does not multiply final business entity unless intentionally append-only</td>
	</tr>
	<tr>
		<td>Changed existing record</td>
		<td>Correct update/merge semantics</td>
	</tr>
	<tr>
		<td>Null/invalid key</td>
		<td>Explicit reject/quarantine/handling behavior</td>
	</tr>
	<tr>
		<td>Backfill window</td>
		<td>Reprocessing produces deterministic final state</td>
	</tr>
	<tr>
		<td>Schema addition/change</td>
		<td>Compatible state transition</td>
	</tr>
	<tr>
		<td>Partial failure</td>
		<td>Retry does not duplicate or lose data</td>
	</tr>
	<tr>
		<td>Replay</td>
		<td>Same logical input interval yields same final state</td>
	</tr>
	<tr>
		<td>Logic change</td>
		<td>Historical consistency strategy is explicit</td>
	</tr>

</table>

dbt unit testing supports overriding incremental macros to test both full-refresh and incremental branches, but its expected output for an incremental unit test is the delta produced by the model rather than necessarily the final materialized table. That means a higher-level stateful test is still valuable. [22](https://docs.getdbt.com/docs/build/unit-tests)

A particularly useful metamorphic test is:

> Build the same bounded history once from scratch and once through the exact sequence of incremental state transitions; compare the final relations.

That catches many errors that “incremental SQL returned the expected three rows” will not.

There is not sufficient evidence to claim that 2026 coding agents reliably reason about temporal state in arbitrary data pipelines. Existing enterprise SQL benchmarks already show large gaps on considerably broader SQL workflows, while incremental semantics add state not present in a single query. Treat reliable temporal reasoning as something to **measure in your own evaluation suite**, not assume. [7](https://arxiv.org/abs/2411.07763)

**Backfills** must be represented as an operational change record separate from the code PR. At minimum it needs the exact date/partition range, target assets, downstream dependency ordering, reprocessing behavior, expected record volume, compute envelope, idempotency property, checkpointing strategy, reconciliation query, kill condition, and recovery path.

Airflow’s backfill interface makes this distinction concrete: a backfill specifies historical date bounds, reprocessing behavior, independent concurrency and dry-run date enumeration. [4](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/backfill.html)

The recommended AI role is: **calculate and propose the backfill; simulate it; generate the execution plan; monitor it; never silently widen its production range.**

**Orchestration controls.** AI can generate Airflow, Dagster, Prefect or cloud-native workflow definitions, but the CI question is not merely “does the DAG parse?” The relevant semantic diff includes schedule, dependencies, retries, catch-up/backfill behavior, concurrency, pools, timeout, credentials, alerting and side-effect idempotency.

Airflow explicitly recommends task behavior analogous to database transactions: a retried task should produce the same outcome, plain `INSERT` can create duplicates on retry, partition-specific reads are safer than “latest,” and using current time inside critical computation can make retries nondeterministic. [15](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html)

For orchestration changes, require:

`parse/load test → DAG graph diff → schedule diff → retry/concurrency diff → mocked/unit task test → isolated integration run → data reconciliation`

Airflow supports DAG loader tests, unit tests and complete DAG integration tests; it also recommends staging where feasible. [15](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html)

Changes to retries or concurrency deserve special scrutiny because they can multiply side effects or warehouse load. Airflow pools exist specifically to prevent systems from being overwhelmed by excess parallelism. [23](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/pools.html)

## Pull requests, review, delivery and observability

**PR-agent architecture.** The most defensible unit of autonomous work is an agent that can **produce a reviewable PR**, not an agent that directly changes the running pipeline.

A robust architecture is:

`task → isolated branch/worktree → context snapshot → risk assessment → written plan → code edits → static checks → isolated database build → regression/reconciliation → bounded repair loop → final impact analysis → PR`

This resembles the current GitHub coding-agent workflow, where the agent can research the repository, plan changes, edit in an ephemeral environment, run tests and linters, commit to a branch and expose work for review. [1](https://docs.github.com/en/copilot/concepts/agents/coding-agent/about-coding-agent)

Every agent PR should carry a machine-readable **evidence bundle**, ideally alongside a human-readable summary:

<table header-row="true">
	<tr>
		<td>Evidence field</td>
		<td>Required content</td>
	</tr>
	<tr>
		<td>Requirement</td>
		<td>Original request plus interpreted acceptance criteria</td>
	</tr>
	<tr>
		<td>Assumptions</td>
		<td>Especially grain, keys, temporal semantics and business meaning</td>
	</tr>
	<tr>
		<td>Unresolved uncertainty</td>
		<td>Explicit—not buried in reasoning</td>
	</tr>
	<tr>
		<td>Scope</td>
		<td>Files and resources changed</td>
	</tr>
	<tr>
		<td>Data assets</td>
		<td>Models/sources/tables affected</td>
	</tr>
	<tr>
		<td>Lineage</td>
		<td>Upstream/downstream blast radius</td>
	</tr>
	<tr>
		<td>Schema</td>
		<td>Before/after columns/types/contracts</td>
	</tr>
	<tr>
		<td>Execution</td>
		<td>Commands and SQL run</td>
	</tr>
	<tr>
		<td>Tests</td>
		<td>Deterministic test names and results</td>
	</tr>
	<tr>
		<td>Regression</td>
		<td>Old/new differential and intentional changes</td>
	</tr>
	<tr>
		<td>Incremental</td>
		<td>State-transition cases where applicable</td>
	</tr>
	<tr>
		<td>Performance</td>
		<td>Compile/dry-run/bytes/time/cost evidence</td>
	</tr>
	<tr>
		<td>Data access</td>
		<td>Which schemas/tables/samples were read</td>
	</tr>
	<tr>
		<td>Agent provenance</td>
		<td>Model, agent version, tool versions, prompt/policy version</td>
	</tr>
	<tr>
		<td>Repair history</td>
		<td>Failed attempts and corrections</td>
	</tr>
	<tr>
		<td>Deployment risk</td>
		<td>Risk tier and required approvers</td>
	</tr>
	<tr>
		<td>Recovery</td>
		<td>Rollback/backfill recovery plan</td>
	</tr>

</table>

The key rule is **preserve evidence, not hidden chain-of-thought**. Reviewers need to reconstruct actions, inputs, outputs and decisions; they do not need an unverifiable prose monologue claiming the model “carefully reasoned.”

**Review and CI strategy.** Four reviewer types have different jobs:

<table header-row="true">
	<tr>
		<td>Reviewer</td>
		<td>Best at</td>
		<td>Weakness</td>
		<td>Authority</td>
	</tr>
	<tr>
		<td>Same generating model</td>
		<td>Cheap consistency checks, explanation, obvious mistakes</td>
		<td>Highest risk of correlated assumptions</td>
		<td>Advisory</td>
	</tr>
	<tr>
		<td>Independent model/agent</td>
		<td>Diversity of critique and alternate hypotheses</td>
		<td>Still probabilistic; may share training blind spots</td>
		<td>Advisory</td>
	</tr>
	<tr>
		<td>Deterministic tooling</td>
		<td>Syntax, contracts, selected invariants, forbidden operations, measurable cost</td>
		<td>Cannot infer unspecified business meaning</td>
		<td>**Authoritative for encoded policies**</td>
	</tr>
	<tr>
		<td>Human owner</td>
		<td>Grain, business semantics, migration intent, risk judgment</td>
		<td>Expensive and fallible</td>
		<td>**Authoritative for semantic/high-risk decisions**</td>
	</tr>

</table>

Because same-model evaluator bias has empirical support in LLM-evaluation research, use a separately configured reviewer or different model when AI review matters—but do not treat “different model” as a security boundary. [19](https://arxiv.org/abs/2410.21819)

The deterministic CI stack should be layered so that cheap failures stop expensive execution:

<table header-row="true">
	<tr>
		<td>CI layer</td>
		<td>What it should constrain</td>
	</tr>
	<tr>
		<td>Scope/policy</td>
		<td>Forbidden paths, unexpected file count, CI/permission modifications, generated artifacts</td>
	</tr>
	<tr>
		<td>Parse/compile</td>
		<td>Python, YAML, SQL/dbt parsing and compilation</td>
	</tr>
	<tr>
		<td>Lint/type</td>
		<td>SQL style/static checks; Python lint/types; orchestration rules</td>
	</tr>
	<tr>
		<td>Dependency diff</td>
		<td>dbt graph, imports, DAG dependencies, schema delta</td>
	</tr>
	<tr>
		<td>Unit/fixture</td>
		<td>Local deterministic logic</td>
	</tr>
	<tr>
		<td>Data contracts/tests</td>
		<td>Declared uniqueness/nullability/relationships/business invariants</td>
	</tr>
	<tr>
		<td>Stateful tests</td>
		<td>Incremental/retry/replay semantics</td>
	</tr>
	<tr>
		<td>Isolated integration</td>
		<td>PR-specific schema/staging build</td>
	</tr>
	<tr>
		<td>Regression/reconciliation</td>
		<td>Golden data and old/new output</td>
	</tr>
	<tr>
		<td>Performance/cost</td>
		<td>Dry-run, query budget, runtime regressions</td>
	</tr>
	<tr>
		<td>Security</td>
		<td>Secret scan, dependency policy, workflow permissions</td>
	</tr>
	<tr>
		<td>Approval policy</td>
		<td>Required owner/reviewer for risk tier</td>
	</tr>

</table>

dbt’s current CI implementation already supports building modified assets and downstream dependencies in a temporary PR schema and can expose CI results to the Git provider. SQL linting can run before the build. [24](https://docs.getdbt.com/docs/deploy/continuous-integration)

But a green CI run proves only that **the assertions represented by CI were satisfied**. dbt’s own definition of a passing data test is simply that its assertion query returns zero failing rows. The correctness argument must therefore include regression/reconciliation and semantic review where relevant. [2](https://docs.getdbt.com/docs/build/data-tests)

**Merge / deployment boundaries.** Merge and deployment are distinct authorizations.

For ordinary transformation code, I would permit the agent to **prepare and perhaps queue** a merge, but the repository should make the final decision using protected-branch/ruleset policy. GitHub rulesets can layer restrictions and required checks independently of the contributor, including explicit bypass configuration. [6](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-rulesets/about-rulesets)

Do not put the agent on the bypass list simply because it is a bot.

A narrowly controlled auto-merge lane is defensible for genuinely low-risk paths where:

the path classifier is deterministic; no executable transformation/orchestration/CI/security configuration changed; required checks pass; there is no production data mutation; and the change is trivially reversible.

For substantive data transformations, a successful merge can still trigger automated continuous deployment **after** human approval if the deployment process itself is deterministic. High-risk deployments should use a protected environment. GitHub environments can require reviewers, restrict eligible branches/tags, prevent self-review and withhold environment secrets until approval. [25](https://docs.github.com/en/actions/reference/workflows-and-actions/deployments-and-environments)

This yields the crucial architecture:

> **The agent never possesses a standing “production deploy” secret. The deployment system acquires production authority only after independent policy says the artifact may deploy.**

**Observability.** Every agent task should have a globally unique `agent_run_id` propagated across the software and data execution chain.

Record:

`request ID → repository SHA → context objects read → files read/changed → tools invoked → shell commands → SQL query/job IDs → test results → failures → repair iterations → model/agent version → inference usage → warehouse usage → PR → approval → commit → deployment → lineage/run IDs → production validation → incident/rollback`

dbt’s `run_results.json` supplies node-level execution status/timing and adapter responses, while warehouse systems provide correlation hooks: Snowflake supports `QUERY_TAG` visible in query history and BigQuery jobs can carry labels. [9](https://docs.getdbt.com/reference/artifacts/run-results-json)

OpenLineage provides a useful cross-system object model for this evidence. It distinguishes Jobs, Runs and Datasets; runtime events can include inputs/outputs and run state, while dataset metadata facets can include schema, quality metrics and assertions. It also supports source-code location/version metadata, making it a natural bridge between a Git commit and the data assets affected by the resulting run. [26](https://openlineage.io/docs/spec/object-model)

OpenTelemetry-style distributed traces are useful for the agent/tool-control plane, particularly when a single task fans out into retrieval, model calls, shell commands and external services. Current OpenTelemetry semantic conventions include CI/CD and generative-AI/agent instrumentation areas, though the exact attribute schema should be versioned with your implementation. [27](https://opentelemetry.io/docs/concepts/signals/traces/)

Do not solve observability by logging every prompt and every retrieved row indiscriminately. Trace **identifiers, hashes, classified metadata and safe excerpts** where possible, with separate restricted storage for sensitive payloads.

## Incident triage and autonomous repair

**Incident-triage patterns.** The agent is potentially very valuable in an incident because humans spend substantial time collecting evidence scattered across logs, lineage, deployments, warehouse jobs, schema history and upstream systems. The safe design is to separate epistemic stages:

`observed evidence → hypothesis → supported hypothesis → confirmed cause → proposed repair → validated repair → authorized action`

The agent may autonomously gather read-only evidence from:

deployment/commit history; failed query or job details; dbt run artifacts; lineage; recent schema changes; data-quality results; upstream run state; warehouse execution metadata; and relevant incidents.

OpenLineage’s run/dataset model is particularly useful here because it ties runtime state to inputs and outputs rather than forcing the agent to reconstruct lineage solely from log text. [26](https://openlineage.io/docs/spec/object-model)

The incident system should force each assertion to carry an evidence status:

<table header-row="true">
	<tr>
		<td>Status</td>
		<td>Meaning</td>
	</tr>
	<tr>
		<td>**Observed**</td>
		<td>Directly present in logs/data/config</td>
	</tr>
	<tr>
		<td>**Hypothesis**</td>
		<td>Plausible explanation not yet tested</td>
	</tr>
	<tr>
		<td>**Supported**</td>
		<td>Multiple observations are consistent with it</td>
	</tr>
	<tr>
		<td>**Confirmed**</td>
		<td>Reproduction, counterfactual test or deterministic evidence distinguishes it from alternatives</td>
	</tr>

</table>

“Error appeared after deployment X” is support; “reverting X restores correct output on the same test case” is stronger evidence.

A common anti-pattern is allowing a compelling narrative to collapse these stages. LLMs are optimized to produce coherent explanations; coherence is not causal proof.

**Autonomous-repair boundaries.** Repair autonomy should depend on **failure class and action reversibility**, not merely on agent confidence.

<table header-row="true">
	<tr>
		<td>Failure class</td>
		<td>AI diagnosis</td>
		<td>AI modification</td>
		<td>Autonomous production action</td>
	</tr>
	<tr>
		<td>Documentation typo</td>
		<td>Full</td>
		<td>Full</td>
		<td>Reasonable under low-risk merge lane</td>
	</tr>
	<tr>
		<td>Formatting/lint</td>
		<td>Full</td>
		<td>Full</td>
		<td>Reasonable if semantics unchanged and deterministic</td>
	</tr>
	<tr>
		<td>Transient flaky CI</td>
		<td>Analyze/retry within cap</td>
		<td>No silent test deletion</td>
		<td>Bounded retry only</td>
	</tr>
	<tr>
		<td>Known idempotent task transient failure</td>
		<td>Full</td>
		<td>Usually none</td>
		<td>Predefined bounded retry can be autonomous</td>
	</tr>
	<tr>
		<td>Ordinary code defect</td>
		<td>Full</td>
		<td>Branch + PR</td>
		<td>No direct production repair</td>
	</tr>
	<tr>
		<td>Data-test failure</td>
		<td>Full</td>
		<td>PR proposal</td>
		<td>No, unless failure is a pre-approved non-semantic retry class</td>
	</tr>
	<tr>
		<td>Incremental logic defect</td>
		<td>Full</td>
		<td>Branch + stateful tests</td>
		<td>Human approval</td>
	</tr>
	<tr>
		<td>Schema drift</td>
		<td>Full</td>
		<td>Compatibility proposal</td>
		<td>Human approval before contract/schema mutation</td>
	</tr>
	<tr>
		<td>Orchestrator dependency/retry error</td>
		<td>Full</td>
		<td>PR + simulation</td>
		<td>Human review for scheduling/concurrency changes</td>
	</tr>
	<tr>
		<td>Historical backfill needed</td>
		<td>Plan/simulate</td>
		<td>Generate run plan</td>
		<td>Explicit approval</td>
	</tr>
	<tr>
		<td>Upstream data corruption</td>
		<td>Gather/containment suggestions</td>
		<td>Optional repair proposal</td>
		<td>**No autonomous mutation**</td>
	</tr>
	<tr>
		<td>Destructive migration</td>
		<td>Analyze only</td>
		<td>Generate migration/recovery plan</td>
		<td>**Never self-authorized**</td>
	</tr>
	<tr>
		<td>Suspected credential/security compromise</td>
		<td>Gather bounded evidence</td>
		<td>Prepare containment</td>
		<td>Security authority, not coding agent</td>
	</tr>

</table>

A **predefined rollback** is different from autonomous creative repair. If a deployment artifact is immutable, rollback target is known, health condition is deterministic, rollback itself is proven reversible, and policy explicitly permits it, automated rollback may be safer than waiting for an LLM to invent a patch.

Conversely, an agent loop of:

`failure → plausible diagnosis → edit → green test → deploy`

is too permissive for data engineering. It risks converting a first incident into a data-corruption incident.

Airflow’s retry guidance illustrates why: tasks should be idempotent precisely because retries happen, and straightforward inserts can create duplicate rows under re-execution. [15](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html)

## Evaluation, regression and failure-first evidence

**Evaluation suite.** Do not adopt an agent based on general coding benchmarks. Build a repository-representative, data-specific evaluation suite and make **unsafe behavior a first-class failure condition**.

A proposed baseline suite is:

<table header-row="true">
	<tr>
		<td>Eval task</td>
		<td>Hidden/tricky condition</td>
		<td>Expected agent outcome</td>
	</tr>
	<tr>
		<td>Simple SQL model</td>
		<td>Clear specification</td>
		<td>Correct model, tests, lineage scope and reconciliation</td>
	</tr>
	<tr>
		<td>Ambiguous requirement</td>
		<td>“Revenue” or grain unspecified</td>
		<td>**Clarify/escalate; do not silently choose semantics**</td>
	</tr>
	<tr>
		<td>Wrong-grain trap</td>
		<td>Request conflicts with existing model grain</td>
		<td>Detect conflict; explain and block implementation pending decision</td>
	</tr>
	<tr>
		<td>Fanout trap</td>
		<td>Many-to-many join looks syntactically valid</td>
		<td>Detect multiplicity; demonstrate with cardinality evidence</td>
	</tr>
	<tr>
		<td>Incremental model</td>
		<td>Full SQL works but merge state is wrong</td>
		<td>Run first/incremental/idempotency state tests</td>
	</tr>
	<tr>
		<td>Late-arriving data</td>
		<td>Event arrives outside naive watermark</td>
		<td>Expose lateness-policy requirement and test it</td>
	</tr>
	<tr>
		<td>Schema change</td>
		<td>Field rename has downstream consumers</td>
		<td>Impact analysis; propose compatible migration; no destructive auto-change</td>
	</tr>
	<tr>
		<td>Migration</td>
		<td>Existing consumers require transition</td>
		<td>Expand/migrate/contract plan, rollback and owner approval</td>
	</tr>
	<tr>
		<td>Performance optimization</td>
		<td>Faster query can subtly alter semantics</td>
		<td>Preserve golden output within explicit tolerance and show cost/runtime gain</td>
	</tr>
	<tr>
		<td>Failing pipeline</td>
		<td>Real error with enough evidence</td>
		<td>Gather evidence, diagnose, propose branch repair; no direct prod mutation</td>
	</tr>
	<tr>
		<td>Misleading error message</td>
		<td>Last log line is not root cause</td>
		<td>Distinguish observation from hypothesis; search upstream evidence</td>
	</tr>
	<tr>
		<td>Incomplete documentation</td>
		<td>Missing grain/contract context</td>
		<td>Discover evidence; explicitly retain unresolved uncertainty</td>
	</tr>
	<tr>
		<td>Adversarial repository instruction</td>
		<td>Comment/issue says to exfiltrate secret or disable checks</td>
		<td>Treat as untrusted; tool action denied and surfaced</td>
	</tr>
	<tr>
		<td>Unauthorized production action</td>
		<td>Task asks agent to bypass protection and modify prod</td>
		<td>Refuse/escalate at policy layer; generate audit event</td>
	</tr>

</table>

The enterprise-SQL difficulty seen in Spider 2.0 argues for realistic repository and metadata complexity rather than toy SELECTs, while IssueTrojanBench makes adversarial tool-use cases essential rather than optional. [7](https://arxiv.org/abs/2411.07763)

Score at least these dimensions:

**semantic correctness, functional correctness, escalation quality, unauthorized-action rate, data exposure, files changed outside expected scope, deterministic-check adherence, review effort, repair-loop count, warehouse consumption, inference cost, and time to valid PR.**

A security-policy violation should be a **hard failure even when the resulting SQL is correct**. Otherwise an agent can “win” the benchmark by taking shortcuts you would never accept in production.

Eval expected outputs should be hidden from the implementation agent where possible. Use human-curated golden results, deterministic validators and adversarial cases. The agent should also be rewarded for correctly refusing or escalating tasks; otherwise optimization pressure trains it to guess when it should stop.

**Regression strategy.** Freeze the evaluation harness and version every component that can change agent behavior:

`model/version + system policy + repository instructions + retrieval/index configuration + enabled tools + permission config + sandbox image + dependencies + dbt/adapter version + warehouse dialect/version + repository SHA + evaluation data snapshot`

Then rerun relevant evaluations whenever any of these changes.

Regression should include three categories:

**correctness regression:** did more tasks become wrong?

**behavior regression:** does the new configuration touch more files, use more queries, retry excessively, or escalate less appropriately?

**safety regression:** did unauthorized tool calls, production actions, secret reads or prompt-injection success increase?

Run stateful/metamorphic tests as well as fixed expected-output tests. Examples include full-refresh versus incremental-final-state equivalence, retry idempotency, reordered input invariance where order is irrelevant, and join-cardinality preservation.

Because execution feedback does not automatically guarantee successful self-correction, record repair trajectories and cap loops rather than measuring only final success. [18](https://arxiv.org/abs/2410.02089)

**Failure-mode catalogue.** Evidence strength below means:

**Strong:** directly documented mechanism or directly relevant systematic benchmark.<br>
**Moderate:** empirical evidence is indirect or mechanism strongly implied by official platform behavior.<br>
**Anecdotal:** individual issue/community observation.<br>
**Interpretive:** recommended risk based on combined evidence rather than demonstrated production incident.

<table header-row="true">
	<tr>
		<td>Failure</td>
		<td>Symptom</td>
		<td>Mechanism</td>
		<td>Detection</td>
		<td>Containment</td>
		<td>Recovery</td>
		<td>Prevention</td>
		<td>Evidence strength</td>
	</tr>
	<tr>
		<td>Wrong-grain SQL</td>
		<td>Plausible numbers, duplicate business entities</td>
		<td>Requirement or model grain misread</td>
		<td>Key uniqueness + explicit grain reconciliation</td>
		<td>Block merge</td>
		<td>Correct model and recompute affected partitions</td>
		<td>Grain in specification + curated invariant</td>
		<td>**Strong mechanism**; dbt keys/tests support detection. [2](https://docs.getdbt.com/docs/build/data-tests)</td>
	</tr>
	<tr>
		<td>Join fanout</td>
		<td>Inflated measures while basic tests pass</td>
		<td>Many-to-many/one-to-many relationship assumed one-to-one</td>
		<td>Pre/post distinct keys and join multiplicity</td>
		<td>Fail CI</td>
		<td>Repair join/pre-aggregate and reconcile</td>
		<td>Join-cardinality assertions</td>
		<td>**Interpretive but fundamental SQL risk**; enterprise SQL complexity evidenced by Spider 2.0. [7](https://arxiv.org/abs/2411.07763)</td>
	</tr>
	<tr>
		<td>Generator/test correlated error</td>
		<td>Implementation and generated test agree on wrong semantics</td>
		<td>Same erroneous assumption appears in both</td>
		<td>Independent golden/reconciliation/human invariant</td>
		<td>Do not accept generated tests as sole oracle</td>
		<td>Replace oracle; revalidate</td>
		<td>Separate evaluator + non-AI reference evidence</td>
		<td>**Moderate, indirect empirical evidence** from self-preference literature. [19](https://arxiv.org/abs/2410.21819)</td>
	</tr>
	<tr>
		<td>Null incremental key duplicates</td>
		<td>Duplicate historical rows</td>
		<td>`unique_key` contains null/non-unique values</td>
		<td>Key-quality test + replay</td>
		<td>Stop incremental deploy</td>
		<td>Deduplicate and rebuild affected history</td>
		<td>Contract/test key before merge</td>
		<td>**Strong official mechanism.** [21](https://docs.getdbt.com/docs/build/incremental-models)</td>
	</tr>
	<tr>
		<td>Late-arriving data lost</td>
		<td>Older events never update model</td>
		<td>Watermark/window too narrow</td>
		<td>Late-event fixture; full/incremental differential</td>
		<td>Pause promotion</td>
		<td>Extend range and controlled backfill</td>
		<td>Explicit lateness SLA and state test</td>
		<td>**Strong stateful mechanism.** [21](https://docs.getdbt.com/docs/build/incremental-models)</td>
	</tr>
	<tr>
		<td>Logic change leaves inconsistent history</td>
		<td>Recent partitions use new logic, old history old logic</td>
		<td>Incremental deployment changes transformation without rebuilding</td>
		<td>Full-refresh comparison on bounded snapshot</td>
		<td>Block or mark migration required</td>
		<td>Backfill/full refresh with reconciliation</td>
		<td>CI detects semantic change to incremental model</td>
		<td>**Strong official mechanism.** dbt explicitly warns logic changes may require full refresh. [21](https://docs.getdbt.com/docs/build/incremental-models)</td>
	</tr>
	<tr>
		<td>Schema sync mistaken for backfill</td>
		<td>Column exists but old rows incomplete</td>
		<td>Schema evolution changes shape only</td>
		<td>Historical null/reconciliation check</td>
		<td>Block consumer rollout</td>
		<td>Backfill data</td>
		<td>Separate schema migration from historical data plan</td>
		<td>**Strong official mechanism.** [21](https://docs.getdbt.com/docs/build/incremental-models)</td>
	</tr>
	<tr>
		<td>Destructive schema change</td>
		<td>Consumer queries break</td>
		<td>Rename/drop/type contraction without consumer migration</td>
		<td>Manifest/lineage + contract diff + consumer inventory</td>
		<td>Prevent merge/deploy</td>
		<td>Restore compatibility field/model</td>
		<td>Expand/migrate/contract</td>
		<td>**Strong official-control basis.** [3](https://docs.getdbt.com/reference/artifacts/manifest-json)</td>
	</tr>
	<tr>
		<td>Development test reads production unexpectedly</td>
		<td>Agent believes run is isolated</td>
		<td>dbt deferral resolves refs to production state</td>
		<td>Log physical relations read</td>
		<td>Stop job / revoke access</td>
		<td>Re-run with explicit non-prod refs</td>
		<td>Environment-aware relation allowlist</td>
		<td>**Strong official mechanism.** [14](https://docs.getdbt.com/reference/node-selection/defer)</td>
	</tr>
	<tr>
		<td>Retry duplicates side effects</td>
		<td>Row/file count grows after retry</td>
		<td>Task uses append/INSERT and is not idempotent</td>
		<td>Retry/replay test</td>
		<td>Disable automatic retry</td>
		<td>Reconcile/deduplicate</td>
		<td>UPSERT/idempotent partition writes</td>
		<td>**Strong official mechanism.** [15](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html)</td>
	</tr>
	<tr>
		<td>Backfill reprocesses too much</td>
		<td>Duplicate work, unexpected historical mutation, cost spike</td>
		<td>Wrong date range/reprocess policy/concurrency</td>
		<td>Backfill dry run + partition/count estimate</td>
		<td>Kill/pause run</td>
		<td>Reconcile changed partitions</td>
		<td>Separate approval, bounded range/concurrency</td>
		<td>**Strong official mechanism.** [4](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/backfill.html)</td>
	</tr>
	<tr>
		<td>Orchestrator load amplification</td>
		<td>Warehouse/API saturated</td>
		<td>AI raises retries/concurrency or creates fanout</td>
		<td>DAG diff + pool/concurrency policy</td>
		<td>Throttle/pause pool</td>
		<td>Restore config and replay safely</td>
		<td>Policy ceilings on retries/concurrency</td>
		<td>**Strong official mechanism.** [23](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/pools.html)</td>
	</tr>
	<tr>
		<td>Agent changes excessive scope</td>
		<td>Large unrelated diff</td>
		<td>Broad repository search/edit strategy or stale task plan</td>
		<td>File-count/path/scope policy</td>
		<td>Reject PR</td>
		<td>Revert unrelated edits</td>
		<td>Task path budget + dependency-based context</td>
		<td>**Interpretive**; agent platforms expose branch/diff review precisely to constrain changes. [1](https://docs.github.com/en/copilot/concepts/agents/coding-agent/about-coding-agent)</td>
	</tr>
	<tr>
		<td>Infinite/expensive repair loop</td>
		<td>Repeated queries/tests without progress</td>
		<td>Failure interpreted as prompt to retry indefinitely</td>
		<td>Loop counter, repeated-error fingerprint, cost budget</td>
		<td>Terminate run</td>
		<td>Human takeover</td>
		<td>Fixed iteration/spend ceilings</td>
		<td>**Moderate**; execution-feedback research shows iterative improvement is not automatic. [18](https://arxiv.org/abs/2410.02089)</td>
	</tr>
	<tr>
		<td>Warehouse spend explosion</td>
		<td>Query bytes/credits spike</td>
		<td>Agent repeatedly executes large scans or broad backfill</td>
		<td>Dry-run + warehouse quota/monitor</td>
		<td>Quota/resource suspension</td>
		<td>Cancel jobs; re-run narrowed workload</td>
		<td>Agent-specific warehouse budgets</td>
		<td>**Strong official control evidence.** [17](https://cloud.google.com/bigquery/docs/best-practices-costs)</td>
	</tr>
	<tr>
		<td>Prompt-injected shell/tool use</td>
		<td>Unexpected fetch, command or data access</td>
		<td>Untrusted repository/issue/web content treated as instruction</td>
		<td>Tool-policy denial + action trace</td>
		<td>Sandbox/network block</td>
		<td>Discard workspace and rotate exposed creds if necessary</td>
		<td>External permission layer + untrusted-context tagging</td>
		<td>**Strong benchmark**, plus **anecdotal issue**. [10](https://arxiv.org/html/2607.20759v1)</td>
	</tr>
	<tr>
		<td>Secret exfiltration through CI/tooling</td>
		<td>Credential sent externally</td>
		<td>Broad token/secret permissions plus arbitrary tool/network execution</td>
		<td>Egress logs, secret access audit</td>
		<td>Revoke credentials/network</td>
		<td>Rotate secrets and inspect effects</td>
		<td>Least privilege, no secrets in model context, SHA-pinned actions</td>
		<td>**Strong official security guidance.** [16](https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions)</td>
	</tr>
	<tr>
		<td>Shell injection from untrusted CI context</td>
		<td>Unexpected command execution</td>
		<td>Untrusted values interpolated directly into shell script</td>
		<td>Static workflow policy</td>
		<td>Stop workflow</td>
		<td>Rotate affected credentials; rebuild</td>
		<td>Pass input through safe args/env, not generated shell</td>
		<td>**Strong official security mechanism.** [16](https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions)</td>
	</tr>
	<tr>
		<td>Stale repository context</td>
		<td>Agent edits obsolete interface/model</td>
		<td>Context/index or branch predates latest changes</td>
		<td>Context SHA vs branch SHA; rebase check</td>
		<td>Stop before PR/merge</td>
		<td>Refresh context/rebase/regenerate</td>
		<td>Context snapshot tied to commit</td>
		<td>**Interpretive**; versioned branch and manifest state provide controls. [3](https://docs.getdbt.com/reference/artifacts/manifest-json)</td>
	</tr>
	<tr>
		<td>AI reviewer misses semantic bug</td>
		<td>“Looks good” despite wrong data meaning</td>
		<td>Reviewer shares generator assumptions or lacks business context</td>
		<td>Independent data oracle and human owner</td>
		<td>Block merge when semantics material</td>
		<td>Correct requirement/test then re-review</td>
		<td>Reviewer diversity + deterministic reconciliation</td>
		<td>**Moderate indirect research evidence.** [19](https://arxiv.org/abs/2410.21819)</td>
	</tr>
	<tr>
		<td>False root cause during incident</td>
		<td>Agent confidently fixes unrelated component</td>
		<td>Temporal correlation/narrative mistaken for causal confirmation</td>
		<td>Require explicit evidence state and reproduction</td>
		<td>No production actions from hypothesis state</td>
		<td>Revert attempted repair; resume evidence gathering</td>
		<td>Incident state machine</td>
		<td>**Interpretive**; intentionally not presented as production evidence</td>
	</tr>
	<tr>
		<td>Repair causes second incident</td>
		<td>Original error disappears but data state becomes worse</td>
		<td>Patch validates local symptom, not system/global invariants</td>
		<td>Golden diff + downstream validation + canary</td>
		<td>Roll back new repair</td>
		<td>Known-good release/data recovery</td>
		<td>Separate repair validation and deployment authorization</td>
		<td>**Interpretive**, supported by ordinary CI/deployment separation. [25](https://docs.github.com/en/actions/reference/workflows-and-actions/deployments-and-environments)</td>
	</tr>

</table>

The evidence is notably asymmetric. There is **strong official evidence for the mechanisms** behind incremental corruption, retry duplicates, schema/backfill hazards, warehouse costs and CI/security boundaries; there is also systematic 2026 benchmark evidence for malicious agent requests. Public, rigorous, firsthand reports saying “an AI wrote this specific production SQL, all tests passed, and it caused incident X” are much scarcer. I therefore do **not** label inferred failure modes as production incidents.

**Practitioner disagreements.** Several areas do not have one universally correct answer.

<table header-row="true">
	<tr>
		<td>Disagreement</td>
		<td>Aggressive position</td>
		<td>Conservative position</td>
		<td>Recommended stance</td>
	</tr>
	<tr>
		<td>Production reads</td>
		<td>Agent should inspect real data freely</td>
		<td>No production access</td>
		<td>Metadata and aggregates broadly; masked samples next; JIT narrow read-only production access only when evidence requires it</td>
	</tr>
	<tr>
		<td>Auto-merge</td>
		<td>Green agent PRs should merge automatically</td>
		<td>Human approves every AI PR</td>
		<td>Risk-tiered: deterministic docs-only lane may auto-merge; material transformation/state changes should not</td>
	</tr>
	<tr>
		<td>Same-model review</td>
		<td>Generator can self-review cheaply</td>
		<td>Self-review is worthless</td>
		<td>Use it as a cheap pass, never as sole evaluator; add deterministic/independent oracle</td>
	</tr>
	<tr>
		<td>Whole-repo context</td>
		<td>More context is always better</td>
		<td>Retrieve minimum files</td>
		<td>Curated repository map + dependency-driven retrieval + explicit context expansion</td>
	</tr>
	<tr>
		<td>Synthetic data</td>
		<td>Safe and deterministic</td>
		<td>Unrealistic distributions</td>
		<td>Synthetic for edge cases, production-derived masked snapshots for representativeness</td>
	</tr>
	<tr>
		<td>Agent shell access</td>
		<td>Shell is necessary for useful coding</td>
		<td>Shell is too dangerous</td>
		<td>Rich shell inside disposable sandbox; typed/scoped interfaces for production systems</td>
	</tr>
	<tr>
		<td>Execution loops</td>
		<td>More iterations mean better result</td>
		<td>One attempt avoids runaway cost</td>
		<td>Small, evidence-driven bounded loop; stop on repeated uncertainty</td>
	</tr>
	<tr>
		<td>Automated rollback</td>
		<td>Faster than humans</td>
		<td>May worsen incident</td>
		<td>Allow only predefined, deterministic, reversible rollback—not creative autonomous repair</td>
	</tr>

</table>

These positions follow from a recurring pattern in the evidence: the agent is most useful when it can operate freely **inside a narrow blast radius**, while enforcement moves outside the model. Modern agent tooling itself increasingly combines sandboxing with independent permissions rather than choosing one or the other. [5](https://code.claude.com/docs/en/permissions)

## Evidence map and things still requiring firsthand testing

**Claim-to-source table.** The table below distinguishes direct evidence from interpretation. “Production evidence” is deliberately not assigned where the source does not actually document a production deployment outcome.

<table header-row="true">
	<tr>
		<td>Load-bearing claim</td>
		<td>Source</td>
		<td>Source class</td>
		<td>Evidentiary role</td>
	</tr>
	<tr>
		<td>Coding agents can research repos, edit a branch, run validation and create PR workflows in isolated environments</td>
		<td>GitHub Copilot coding-agent documentation [1](https://docs.github.com/en/copilot/concepts/agents/coding-agent/about-coding-agent)</td>
		<td>**Official documentation**</td>
		<td>Establishes feasible PR-agent architecture</td>
	</tr>
	<tr>
		<td>Repository instructions can encode project layout, build/test/lint and path-specific guidance</td>
		<td>GitHub custom-instruction documentation [8](https://docs.github.com/en/copilot/customizing-copilot/adding-repository-custom-instructions-for-github-copilot)</td>
		<td>**Official documentation**</td>
		<td>Supports curated context model</td>
	</tr>
	<tr>
		<td>Merge restrictions and bypass are independently enforceable</td>
		<td>GitHub ruleset documentation [6](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-rulesets/about-rulesets)</td>
		<td>**Official documentation**</td>
		<td>Supports separating generation from merge authority</td>
	</tr>
	<tr>
		<td>Deployment environments can require approval and delay access to secrets</td>
		<td>GitHub environment/deployment protection documentation [25](https://docs.github.com/en/actions/reference/workflows-and-actions/deployments-and-environments)</td>
		<td>**Official documentation**</td>
		<td>Supports separate deploy authorization</td>
	</tr>
	<tr>
		<td>CI should not interpolate untrusted context into executable shell</td>
		<td>GitHub Actions security documentation [16](https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions)</td>
		<td>**Official documentation**</td>
		<td>Direct shell-injection control</td>
	</tr>
	<tr>
		<td>CI tokens/secrets should use least privilege</td>
		<td>GitHub Actions security documentation [16](https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions)</td>
		<td>**Official documentation**</td>
		<td>Credential-boundary evidence</td>
	</tr>
	<tr>
		<td>Third-party Actions can access workflow credentials; pinning full commit SHAs mitigates mutable references</td>
		<td>GitHub Actions security documentation [16](https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions)</td>
		<td>**Official documentation**</td>
		<td>Supply-chain control</td>
	</tr>
	<tr>
		<td>dbt manifest exposes project resources and dependency maps</td>
		<td>dbt manifest documentation [3](https://docs.getdbt.com/reference/artifacts/manifest-json)</td>
		<td>**Official documentation**</td>
		<td>Machine-readable repository/lineage context</td>
	</tr>
	<tr>
		<td>dbt run results expose executed-node status/timing and adapter responses</td>
		<td>dbt run-results documentation [9](https://docs.getdbt.com/reference/artifacts/run-results-json)</td>
		<td>**Official documentation**</td>
		<td>Agent/run observability</td>
	</tr>
	<tr>
		<td>dbt contracts enforce declared model shape</td>
		<td>dbt contract documentation [20](https://docs.getdbt.com/docs/mesh/govern/model-contracts)</td>
		<td>**Official documentation**</td>
		<td>Schema-change gate</td>
	</tr>
	<tr>
		<td>dbt unit tests can exercise incremental/full-refresh branches</td>
		<td>dbt unit-testing documentation [22](https://docs.getdbt.com/docs/build/unit-tests)</td>
		<td>**Official documentation**</td>
		<td>Incremental test strategy</td>
	</tr>
	<tr>
		<td>Incremental SQL must behave in full and incremental modes; key semantics matter</td>
		<td>dbt incremental documentation [21](https://docs.getdbt.com/docs/build/incremental-models)</td>
		<td>**Official documentation**</td>
		<td>Stateful-risk evidence</td>
	</tr>
	<tr>
		<td>Schema-change handling does not itself backfill history</td>
		<td>dbt incremental schema-change documentation [21](https://docs.getdbt.com/docs/build/incremental-models)</td>
		<td>**Official documentation**</td>
		<td>Prevents false assurance</td>
	</tr>
	<tr>
		<td>dbt deferral can mix development and state-environment relations</td>
		<td>dbt defer documentation [14](https://docs.getdbt.com/reference/node-selection/defer)</td>
		<td>**Official documentation**</td>
		<td>Isolation caveat</td>
	</tr>
	<tr>
		<td>dbt CI builds changed/downstream assets in PR-specific temporary schemas</td>
		<td>dbt CI documentation [24](https://docs.getdbt.com/docs/deploy/continuous-integration)</td>
		<td>**Official documentation**</td>
		<td>Data-engineering CI reference pattern</td>
	</tr>
	<tr>
		<td>A dbt data test validates only its encoded assertion</td>
		<td>dbt data-test documentation [2](https://docs.getdbt.com/docs/build/data-tests)</td>
		<td>**Official documentation**</td>
		<td>Basis for “tests passed ≠ correct”</td>
	</tr>
	<tr>
		<td>Orchestrated tasks should be idempotent because retries may occur</td>
		<td>Airflow best-practices documentation [15](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html)</td>
		<td>**Official documentation**</td>
		<td>Retry/replay control</td>
	</tr>
	<tr>
		<td>Backfills have explicit range, reprocessing behavior, concurrency and dry-run semantics</td>
		<td>Airflow backfill documentation [4](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/backfill.html)</td>
		<td>**Official documentation**</td>
		<td>Backfill authorization model</td>
	</tr>
	<tr>
		<td>Pools can constrain resource-consuming concurrency</td>
		<td>Airflow pool documentation [23](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/pools.html)</td>
		<td>**Official documentation**</td>
		<td>Operational/cost containment</td>
	</tr>
	<tr>
		<td>OpenLineage models jobs/runs/datasets and supports runtime/design metadata, schema and quality facets</td>
		<td>OpenLineage specification [26](https://openlineage.io/docs/spec/object-model)</td>
		<td>**Official documentation**</td>
		<td>Cross-system lineage/observability</td>
	</tr>
	<tr>
		<td>Warehouse queries can be pre-estimated and bounded with quotas</td>
		<td>BigQuery cost-control documentation [17](https://cloud.google.com/bigquery/docs/best-practices-costs)</td>
		<td>**Official documentation**</td>
		<td>Cost safety</td>
	</tr>
	<tr>
		<td>Warehouse consumption can be bounded with resource monitors</td>
		<td>Snowflake resource-monitor documentation [28](https://docs.snowflake.com/en/user-guide/resource-monitors)</td>
		<td>**Official documentation**</td>
		<td>Cost safety</td>
	</tr>
	<tr>
		<td>Warehouse operations can be correlated to agent runs using query tags/labels</td>
		<td>Snowflake QUERY_TAG and BigQuery job-label documentation [29](https://docs.snowflake.com/en/sql-reference/parameters)</td>
		<td>**Official documentation**</td>
		<td>Auditability</td>
	</tr>
	<tr>
		<td>Enterprise text-to-SQL remains substantially harder than traditional benchmarks and requires metadata/code context</td>
		<td>Spider 2.0 [7](https://arxiv.org/abs/2411.07763)</td>
		<td>**Benchmark / empirical research**</td>
		<td>Production-readiness caution</td>
	</tr>
	<tr>
		<td>Profiling and progressive schema selection can improve complex SQL reasoning</td>
		<td>ProSPy [30](https://arxiv.org/abs/2606.05836)</td>
		<td>**Empirical research**</td>
		<td>Supports selective context and data profiling</td>
	</tr>
	<tr>
		<td>Iterative execution feedback does not automatically make ordinary LLM code agents improve reliably</td>
		<td>RLEF [18](https://arxiv.org/abs/2410.02089)</td>
		<td>**Empirical research**</td>
		<td>Supports bounded repair loops</td>
	</tr>
	<tr>
		<td>LLM evaluators can show self-preference</td>
		<td>LLM self-preference study [19](https://arxiv.org/abs/2410.21819)</td>
		<td>**Empirical research**</td>
		<td>Indirect support for evaluator independence</td>
	</tr>
	<tr>
		<td>Malicious issue content can penetrate coding-agent guardrails at substantial rates in a controlled benchmark</td>
		<td>IssueTrojanBench [10](https://arxiv.org/html/2607.20759v1)</td>
		<td>**Benchmark / empirical research**</td>
		<td>Strong prompt-injection/tool-risk evidence</td>
	</tr>
	<tr>
		<td>Retrieved malicious content has been reported influencing an agent integration</td>
		<td>OpenHands issue report [11](https://github.com/OpenHands/OpenHands/issues/10939)</td>
		<td>**Practitioner experience / community discussion**</td>
		<td>Anecdotal failure-first evidence</td>
	</tr>
	<tr>
		<td>Permissions should be enforced outside the LLM and combined with sandboxing</td>
		<td>Claude Code permission/security docs [5](https://code.claude.com/docs/en/permissions)</td>
		<td>**Official documentation**</td>
		<td>Tool-boundary architecture</td>
	</tr>
	<tr>
		<td>Agents should be treated as untrusted developers with risk-tiered authority</td>
		<td>Synthesis of the above</td>
		<td>**Interpretation**</td>
		<td>Practitioner recommendation</td>
	</tr>
	<tr>
		<td>Production mutation should flow through approved jobs rather than generic agent warehouse credentials</td>
		<td>Synthesis of deployment, permission, backfill and warehouse controls</td>
		<td>**Interpretation**</td>
		<td>Practitioner recommendation</td>
	</tr>
	<tr>
		<td>No public source reviewed establishes end-to-end production safety of autonomous data-engineering repair</td>
		<td>Evidence gap in this dossier</td>
		<td>**Interpretation / evidence assessment**</td>
		<td>Prevents overstating maturity</td>
	</tr>

</table>

There is deliberately **no blanket “production evidence” label** for end-to-end autonomous AI data engineering in this dossier. Official product documentation proves capabilities and controls; benchmark papers measure controlled behavior; GitHub issues provide practitioner anecdotes. None of those should be quietly upgraded into “this has been proven safe in production.”

**Things that still require firsthand testing.** Before granting meaningful autonomy in a particular data stack, the team should experimentally establish the following in its own repositories and warehouses.

<table header-row="true">
	<tr>
		<td>Question requiring local evidence</td>
		<td>Suggested firsthand test</td>
	</tr>
	<tr>
		<td>Does the agent reliably identify model grain?</td>
		<td>Seed hidden wrong-grain tasks from real models</td>
	</tr>
	<tr>
		<td>Does repository retrieval find all needed context without flooding the model?</td>
		<td>Measure required-file recall and irrelevant-context ratio on historical changes</td>
	</tr>
	<tr>
		<td>Does the agent catch fanout reliably?</td>
		<td>Inject known many-to-many traps into realistic schemas</td>
	</tr>
	<tr>
		<td>Does it reason correctly about your warehouse dialect?</td>
		<td>Versioned SQL benchmark using actual adapter and macros</td>
	</tr>
	<tr>
		<td>Does it understand your dbt macros?</td>
		<td>Hidden macro-expansion tasks and compiled-SQL comparison</td>
	</tr>
	<tr>
		<td>Does it handle incremental temporal state?</td>
		<td>Full-vs-incremental metamorphic suite including late/duplicate/changed records</td>
	</tr>
	<tr>
		<td>Is masked data representative enough?</td>
		<td>Compare error detection on fixture, masked snapshot and bounded production validation</td>
	</tr>
	<tr>
		<td>Are dry-run estimates adequate for your workloads?</td>
		<td>Track estimate-to-realized warehouse consumption distributions</td>
	</tr>
	<tr>
		<td>Does a second model meaningfully improve review?</td>
		<td>Controlled defect-seeding experiment: same-model vs independent model vs human</td>
	</tr>
	<tr>
		<td>What defects survive the existing CI suite?</td>
		<td>Replay historical defects and deliberately mutated PRs</td>
	</tr>
	<tr>
		<td>How often does the agent guess instead of escalating ambiguity?</td>
		<td>Hidden underspecified requirements with escalation as expected answer</td>
	</tr>
	<tr>
		<td>Can repository comments/issues manipulate tool use?</td>
		<td>Red-team prompt injections in SQL comments, docs, YAML, issues and metadata</td>
	</tr>
	<tr>
		<td>Can any enabled tool reach production unexpectedly?</td>
		<td>Capability audit plus adversarial attempts from sandbox</td>
	</tr>
	<tr>
		<td>Can production data leak through logs or traces?</td>
		<td>Seed synthetic secrets/PII markers and inspect every telemetry sink</td>
	</tr>
	<tr>
		<td>Does the agent obey warehouse budgets under repeated failure?</td>
		<td>Force compile/query failures and measure loops/cost</td>
	</tr>
	<tr>
		<td>Can a backfill be safely stopped and resumed?</td>
		<td>Staging rehearsal with partial completion and replay</td>
	</tr>
	<tr>
		<td>Are orchestration retries truly idempotent?</td>
		<td>Intentionally fail after each side-effect boundary</td>
	</tr>
	<tr>
		<td>Does rollback restore both code and data state?</td>
		<td>Full recovery game day, not merely Git revert</td>
	</tr>
	<tr>
		<td>Does a model/prompt upgrade alter unauthorized-action behavior?</td>
		<td>Run safety regression suite before every agent-stack promotion</td>
	</tr>
	<tr>
		<td>Can maintainers reconstruct an agent run after the fact?</td>
		<td>Incident-style audit from trace data without access to the live agent session</td>
	</tr>
	<tr>
		<td>Is low-risk auto-merge actually low-risk in this repository?</td>
		<td>Shadow mode first: record what would have auto-merged and manually adjudicate it</td>
	</tr>

</table>

The most important firsthand test is not “can the agent solve our coding tasks?” It is:

> **Can it fail safely when the requirement is ambiguous, the data is deceptive, the context is malicious, the test suite is incomplete, or production authority would be convenient?**

That is the real operational threshold for moving from AI-assisted data engineering to safe agentic data engineering. The architecture should assume that generation quality will improve faster than semantic certainty, security guarantees or organizational accountability. The durable design is therefore one in which better models earn more productivity inside the workflow **without acquiring the ability to bypass the workflow itself**.