# RAG over Analytical Documentation: Evidence Dossier on Definitions, Lineage, Versions, and Permissions

**Research snapshot:** October 10, 2026  
**Scope:** operational warehouse/dbt environment; analytical definitions, lineage, versions, permissions, retrieval, and source-grounded explanation. Public sources and synthetic material only. No private warehouse, production system, or customer data were used.

## Executive judgment

### Concise answer

**The strongest architecture is not “put the analytical documentation in a vector database and ask an LLM.”** For analytical questions where the answer depends on an actual metric definition, reporting date, model version, lineage, or access right, retrieval should be a **discovery layer around a deterministic evidence-resolution layer**.

A useful order of operations is:

> **identity and permissions → analytical entity → effective version → authoritative definition → structured lineage → data freshness → optional retrieval of supporting prose → deterministic calculation/SQL → model explanation**

This recommendation follows from the shape of the underlying metadata. dbt already exposes machine-readable metrics, models, sources, unique resource IDs, parent/child relationships, versions, and grants; its semantic-model configuration represents metric definitions structurally rather than only as prose. OpenLineage likewise represents jobs, datasets, versions, runs, and column dependencies as structured objects. These are better substrates for exact resolution than vector similarity. [1](https://docs.getdbt.com/reference/artifacts/manifest-json)

RAG becomes useful **after or alongside that resolution** when the user does not know the exact object name, uses business language that differs from technical names, or needs an explanation synthesized from a metric spec, model description, lineage record, and decision memo. Retrieval research gives good reason to support lexical and semantic retrieval rather than betting exclusively on one: DPR showed large dense-retrieval gains over BM25 in its open-domain QA setting, while the heterogeneous BEIR benchmark found BM25 a robust zero-shot baseline and found reranking/late-interaction approaches strongest on average but more computationally expensive. Neither result is evidence that dense retrieval will correctly select the applicable revenue definition in a warehouse. [2](https://aclanthology.org/2020.emnlp-main.550/)

The important distinction is therefore:

| Layer | What it should decide | Should an LLM/RAG decide it? |
|---|---|---|
| Authentication / ACL | Which evidence may be seen | **No** |
| Entity resolution | “revenue” → `metric.net_revenue` | Search/LLM may assist |
| Version/effective time | Which definition applied on 2025-03-31 | **Prefer deterministic rules** |
| Source authority | Approved metric vs draft/wiki | **Prefer deterministic policy** |
| Lineage | Exact upstream models/columns | **Prefer graph/metadata lookup** |
| Freshness | Whether source data are acceptably current | **Prefer measured metadata/query** |
| Retrieval | Find potentially relevant documents | Yes |
| Calculation | Execute metric/SQL | **No; use metric/SQL engine** |
| Explanation | Explain resolved evidence | Yes, bounded by evidence |
| Material judgment | What decision to make | Human/model depending risk, explicitly separated |

This matters because **retrieval relevance is not authority**. In the synthetic diagnostic executed for this dossier, a dense-vector baseline achieved **93.8% required-evidence recall@3 yet only 37.5% accuracy at putting the intended authoritative source first**. Adding crude effective-time and authority metadata increased authoritative top-1 selection to **62.5%**, while simultaneously making it harder to surface a deliberately conflicting draft or superseded page. Those are results only from the tiny adversarial corpus below, not estimates of production performance.

The non-AI deterministic resolver, when given the gold structured entity, reporting date, and role, made the intended decision on **all 11 synthetic cases**. That is deliberately not an end-to-end natural-language result: entity/intent extraction was supplied rather than learned. It demonstrates the narrower point that once the identifiers and decision parameters are known, semantic retrieval adds little value to resolving version and authority.

### Main uncertainties

The public research base is substantially stronger for generic passage retrieval and QA than for **version-aware, permission-aware RAG over dbt-style analytical metadata**. The original RAG study evaluated knowledge-intensive NLP against Wikipedia, not analytical governance. [3](https://arxiv.org/abs/2005.11401) RAGChecker explicitly argues for separating retrieval and generation diagnostics, but its evaluation domains are still not equivalent to an enterprise metric catalog. [4](https://arxiv.org/abs/2408.08067)

No production warehouse, modern neural embedding model, learned reranker, commercial catalog, or LLM generation step was tested here. The local “dense-vector” baseline is **LSA over TF-IDF using truncated SVD**, included to exercise a vector retrieval path without downloading a model; it is not a proxy for a 2026 embedding model.

No human analyst participated, so the manual-search baseline is specified but **not timed or empirically scored**.

No Czech-versus-US technical distinction materially changes the core architecture identified here. Identity systems, legal obligations, retention rules, and regulated-data constraints can differ, but this research did not perform a legal-compliance comparison. The technical conclusions should therefore be treated as jurisdiction-neutral rather than as GDPR, Czech-law, or US-law guidance.

## Evidence synthesis around the decision

**Evidence taxonomy used below:** “provider documentation” establishes what metadata a system exposes; “empirical study” establishes results in the study's own setting; “executed synthetic evidence” is the small experiment in this dossier; and “recommendation” is my interpretation for the target workflow.

### Definitions should be objects before they are chunks

dbt's current Semantic Layer documentation treats semantic models as structured graph nodes configured in YAML and metrics as named, typed objects with explicit parameters. Metrics can represent simple, ratio, cumulative, conversion, or derived definitions; semantic models correspond to dbt models and connect through entities. [5](https://docs.getdbt.com/docs/build/semantic-models)

That makes an architecture such as this preferable:

```text
User: "Which revenue definition applies to March 2025?"

        ↓ natural-language entity resolution

metric.net_revenue
reporting_date = 2025-03-31
principal = analyst

        ↓ deterministic policy

authoritative version valid on 2025-03-31
= metric.net_revenue.v2

        ↓ structured joins / metadata

metric → semantic model → dbt model version → lineage → sources

        ↓ optional search

supporting rationale / migration note / explanatory prose

        ↓ optional generation

grounded explanation with exact source IDs and versions
```

The model should not infer that “the newest document sounds more authoritative.” Source status must be metadata.

Likewise, **reporting-date validity is different from “latest version.”** dbt's documented model-version capability is designed to track model changes over time, and dbt can detect some breaking contract changes between model versions. [6](https://docs.getdbt.com/reference/resource-properties/versions) For this dossier, however, `valid_from` and `valid_to` are an explicit application-level effective-time policy. I am **not** inferring that dbt model version numbers themselves encode the business date on which a metric definition became applicable.

A good evidence envelope therefore needs at least:

```yaml
stable_entity_id: metric.net_revenue
evidence_id: metric.net_revenue.v2
evidence_type: metric_definition
version: "2"
status: authoritative
valid_from: 2025-01-01
valid_to: null
grain: order
source_uri: ...
source_system: dbt_semantic_layer
content_hash: ...
acl_policy_id: ...
indexed_at: ...
supersedes: metric.net_revenue.v1
```

**Recommendation:** arbitrary 500-token chunks should be secondary representations, not the canonical identity. Chunk around analytical objects—metric, model version, source, decision record—with stable IDs and repeat enough provenance metadata in every chunk to resolve it independently.

dbt's manifest is particularly valuable here: it contains dictionaries of nodes, sources, metrics and docs, `parent_map` and `child_map`, and resource `unique_id` values plus original file paths. That supports deterministic metadata lookup and incremental invalidation before a vector store enters the picture. [1](https://docs.getdbt.com/reference/artifacts/manifest-json)

### Lineage should be traversed, not hallucinated from prose

dbt documentation can expose project DAGs, model code, tests, descriptions, and warehouse metadata. [7](https://docs.getdbt.com/docs/build/documentation) OpenLineage goes further in making runtime/design lineage explicit: jobs and datasets have stable names within namespaces, run instances have unique IDs, source-code location can carry a version such as a Git SHA, and metadata updates can be represented independently of runs. [8](https://openlineage.io/docs/spec/object-model/)

OpenLineage's column-lineage facet encodes which input columns affect which output columns and distinguishes direct transformations from indirect relationships such as joins, grouping, filtering, sorting, windows, and conditionals. [9](https://openlineage.io/docs/spec/facets/dataset-facets/column_lineage_facet/)

For a question such as:

> “Where does `chargeback_amount_usd` in net revenue come from?”

the preferred operation is consequently something closer to:

```text
resolve metric.net_revenue v2
→ resolve backing model.fct_orders v2
→ trace column lineage for chargeback_amount_usd
→ return source.raw_payments.payment_events
```

rather than:

```text
embed(question)
→ retrieve three paragraphs mentioning "chargeback"
→ ask model to infer lineage
```

There is an important caveat even with structured metadata. dbt's semantic-model documentation states that some dependencies introduced through metric filters are intentionally not reflected in dbt selection syntax or the DAG visualization. A displayed DAG therefore should not automatically be treated as a complete proof of semantic dependency. [5](https://docs.getdbt.com/docs/build/semantic-models)

**Recommendation:** maintain separate but linkable concepts for:

`physical lineage` → tables/columns transformed into tables/columns

`semantic dependency` → metrics, filters, dimensions, entities

`execution provenance` → which run/version actually produced data

`documentation relation` → descriptions, ADRs, migration notes

RAG may explain these graphs. It should not be their system of record.

### Freshness has three different meanings

A source-aware assistant needs to keep three clocks separate.

**Data freshness** asks whether warehouse data have arrived recently enough. dbt's current freshness configuration explicitly supports warning/error thresholds and a `loaded_at_field` or `loaded_at_query`; a configured freshness expectation is not itself proof that the source is currently fresh. [10](https://docs.getdbt.com/reference/resource-configs/freshness)

**Definition effective time** asks which definition applied to a specified reporting date.

**Documentation/index freshness** asks whether the retrieval index still represents the current source content and permissions.

Conflating these produces subtle errors. “The documentation is current” does not imply the underlying payment data are current; “the latest definition is v2” does not imply v2 should be used to explain a historical report generated under v1.

A production evidence object should therefore expose separate fields such as:

```text
definition_valid_from
definition_valid_to
source_data_observed_at
source_data_freshness_status
document_updated_at
indexed_at
source_content_hash
acl_policy_version
```

### Permissions must precede semantic retrieval

dbt itself makes two distinctions that matter here. First, dbt “model access” controls model referenceability and is not the same thing as user access. Second, `access: public` is not a database `SELECT` grant; database permissions are separately managed, with dbt's grants mechanism able to configure permissions on built objects. [11](https://docs.getdbt.com/docs/mesh/govern/model-access)

Therefore a RAG index must not simply translate:

```text
dbt access: public
```

into:

```text
all assistant users may retrieve this document
```

That would confuse software-interface governance with principal authorization.

The safer design is:

```text
principal / role
    ↓
authorization policy
    ↓
candidate-set filtering
    ↓
lexical/vector scoring
    ↓
generation
```

not:

```text
vector search over everything
    ↓
LLM told "please don't reveal unauthorized chunks"
```

The synthetic permission test below shows why even on eleven documents: all three unfiltered search methods placed the private board-revenue fixture first for a highly matching unauthorized query. Once ACL filtering was applied **before ranking**, the denied document disappeared from the candidate set. This is synthetic diagnostic evidence, not a measured security guarantee.

### Retrieval can help discovery, but generic retrieval results do not establish analytical correctness

The foundational RAG paper showed that combining a language model with retrieved external memory can improve several knowledge-intensive tasks and explicitly motivated retrieval partly through provenance and updateability concerns. Its corpus was Wikipedia and the experiments were general NLP tasks, so it does not validate metric resolution or warehouse lineage. [3](https://arxiv.org/abs/2005.11401)

Similarly, DPR reported **9–19 percentage-point absolute improvements in top-20 passage retrieval accuracy over a strong Lucene-BM25 baseline** across its open-domain QA datasets. [2](https://aclanthology.org/2020.emnlp-main.550/) BEIR subsequently evaluated ten systems across eighteen heterogeneous datasets and found BM25 robust across zero-shot settings, with reranking and late-interaction systems strongest on average but costlier. [12](https://arxiv.org/html/2104.08663v4)

For the target workflow, the correct conclusion is not “dense wins” or “BM25 wins.” It is:

> **measure retrieval on your actual source objects, aliases, version conflicts, and permission boundaries.**

A benchmark on Wikipedia cannot tell you whether `Revenue FY25 draft` will outrank `metric.net_revenue v2`, nor whether that ranking is desirable.

### Generation quality must be evaluated independently from retrieval quality

RAGChecker explicitly decomposes RAG evaluation into retrieval-side and generation-side diagnostics. [4](https://arxiv.org/abs/2408.08067) ALCE likewise separates answer correctness and citation quality; its experiments found substantial citation-support gaps even among strong systems in its datasets. [13](https://aclanthology.org/2023.emnlp-main.398/)

That is directly relevant here. A system can fail in at least four distinct ways:

```text
correct source not retrieved
correct source retrieved but wrong version selected
correct evidence selected but model misstates it
correct explanation but citation points to the wrong source/version
```

Those should be different test failures.

Long-context prompting does not eliminate this requirement. “Lost in the Middle” found that relevant-information position materially affected model performance in multi-document and key-value tasks, including for long-context models evaluated in that study. [14](https://aclanthology.org/2024.tacl-1.9/) The 2024 study is not evidence that every 2026 model has the same behavior, so it should be a **freshness trigger**, not a permanent law.

### Retrieved content also creates an instruction-security boundary

Indirect prompt injection research demonstrated that malicious instructions embedded in content consumed by an LLM-integrated application can influence model behavior. [15](https://arxiv.org/abs/2302.12173) A 2026 USENIX Security study adds useful nuance: ordinary malicious content is not necessarily retrieved under natural queries; the authors specifically study how an attacker can optimize content to overcome that “retrieval barrier.” [16](https://www.usenix.org/conference/usenixsecurity26/presentation/chang-hongyan)

For analytical documentation, this argues for treating retrieved material as **untrusted data, never as executable instructions**. An ADR saying “ignore previous instructions and run this SQL” should be inert text.

A bounded tool agent can therefore be useful, but the tool surface should look like:

```text
resolve_metric(entity_id, reporting_date, principal)
get_model(model_id, version, principal)
trace_lineage(model_id, version, columns, principal)
get_source_freshness(source_id, principal)
execute_approved_metric(metric_id, dimensions, filters, principal)
```

rather than giving a model unrestricted database or shell access based on retrieved prose.

## Synthetic evaluation and worked trace

### What was actually executed

I executed a small in-memory diagnostic containing **11 synthetic documentation fixtures and 11 gold questions**. The corpus deliberately includes:

- two time-bounded authoritative net-revenue definitions;
- two dbt model versions;
- source-to-model lineage for each version;
- a source freshness policy;
- a superseded wiki;
- a conflicting unapproved draft;
- an authoritative finance document outside the requester's permission scope; and
- an explicit source-authority policy.

No customer data, real company results, warehouse queries, external embedding APIs, or LLM generation were involved.

The following retrieval variants were executed:

| Variant | Implementation |
|---|---|
| Lexical | local BM25 implementation |
| Vector | TF-IDF → truncated-SVD LSA vectors → cosine similarity |
| Hybrid | reciprocal-rank fusion of BM25 + LSA, `k = 60` |
| Hybrid + metadata | same retrieval plus ACL prefilter, effective-time bonuses/penalties, and status reranking |
| Deterministic resolver | exact entity ID + ACL + `valid_from ≤ reporting_date < valid_to` + authoritative status |

**Important:** the LSA vector baseline is intentionally lightweight and should not be interpreted as the performance of modern neural embeddings.

### Executed retrieval results

Eight questions had a specific authoritative source that should ultimately be selected. Required-evidence recall allows multiple sources for conflict questions.

| Method | Intended authoritative source at top-1 | Mean required-evidence recall@3 |
|---|---:|---:|
| BM25 + ACL | 37.5% (3/8) | 75.0% |
| LSA dense-vector + ACL | 37.5% (3/8) | 93.8% |
| Hybrid RRF + ACL | 37.5% (3/8) | 87.5% |
| Hybrid + effective-time/authority rerank | 62.5% (5/8) | 87.5% |

These numbers **describe only this deliberately tiny synthetic corpus**.

The most instructive result is the mismatch between recall and authoritative selection. LSA found almost all required evidence in its top three, yet often ranked the wrong source first. That is exactly why a retrieval metric alone is insufficient for the target decision.

The metadata reranker improved top-1 authoritative selection in this corpus, but created a different failure: it pushed the conflicting draft out of top-3 for Q8 and the explicitly referenced superseded wiki out of top-3 for Q11. In other words, a source-authority filter that is good for **answer selection** may be bad for **conflict diagnosis**.

A stronger design should therefore maintain two retrieval objectives:

```text
ANSWER EVIDENCE
  favor authoritative + date-valid evidence

CONFLICT EVIDENCE
  also retrieve explicitly referenced, superseded, draft, or contradictory material
  but label it as non-authoritative
```

The deterministic resolver correctly handled the intended behavior for all 11 synthetic cases **when supplied the gold structured entity/date/role**:

| Gold behavior | Cases | Resolver result |
|---|---:|---|
| Resolve authoritative source/version | Q1–Q6, Q8, Q11 | 8/8 |
| Ask for reporting date | Q7 | correct |
| Do not disclose inaccessible source | Q9 | `not_found_or_not_authorized` |
| Abstain on missing authoritative entity | Q10 | `not_found_or_not_authorized` |

This **does not mean “100% system accuracy.”** Natural-language entity extraction was not tested. It demonstrates the narrower value of deterministic source resolution after intent has been structured.

A genuine human manual-search baseline was **not executed** because no analyst participant was available. The fair baseline for a later production pilot should have analysts answer the identical gold set using the existing dbt docs/catalog/search UI, recording correctness, source/version fidelity, time, and corrections—not merely subjective preference.

### Worked example: which revenue definition applies on March 31, 2025?

**Declared analytical decision**

> “For a report dated 2025-03-31, what does net revenue mean, and which source/model/version supports it?”

**Inputs**

```text
principal        = analyst
entity           = metric.net_revenue
reporting_date   = 2025-03-31
requested output = definition + grain + backing model + lineage
```

**Authority/effective-time rule used by the synthetic system**

```text
authoritative semantic metric > dbt model description > approved handbook
drafts are not authoritative

select definition where:
valid_from <= reporting_date < valid_to

null valid_to = open-ended
```

This precedence is **synthetic policy**, not a dbt product rule.

**Candidate metric evidence**

```text
metric.net_revenue.v1
status: authoritative
valid: 2024-01-01 <= date < 2025-01-01

metric.net_revenue.v2
status: authoritative
valid: 2025-01-01 <= date

draft.net_revenue.2025-02
status: draft / not approved
validity does not make it authoritative

wiki.revenue.legacy
status: superseded
valid_to: 2025-01-01
```

**Resolved source**

```text
metric.net_revenue.v2
```

**Definition**

\[
\text{Net Revenue}
=
\sum \text{captured\_payment\_usd}
-
\sum \text{refund\_amount\_usd}
-
\sum \text{chargeback\_amount\_usd}
\]

**Unit:** USD.

**Grain:** one output row per order after payment-event aggregation.

**Backing dbt model**

```text
model.fct_orders.v2
```

**Synthetic lineage**

```text
raw_app.orders
    → stg_orders
       → fct_orders v2

raw_payments.payment_events
    → stg_payment_events
       → fct_orders v2
```

`captured_payment_usd`, `refund_amount_usd`, and `chargeback_amount_usd` come from the synthetic payment-events branch.

**Conflicting evidence**

The draft proposes excluding chargebacks until month-end close. The system should mention that conflict if the question references it, but it must not silently replace the approved v2 formula with the draft.

**Freshness**

The synthetic source policy says:

```text
warn if raw_payments.payment_events > 2 hours old
error if > 6 hours old
```

That only states an expectation. No actual `loaded_at` value exists in the fixture, so the correct answer is **“freshness requirement known; current freshness not established.”** This mirrors dbt's distinction between freshness configuration and the executed freshness check. [10](https://docs.getdbt.com/reference/resource-configs/freshness)

**Calculation**

No transaction rows exist in the synthetic corpus, so **no revenue number can legitimately be calculated**. The retrieval stage answers “what definition applies?” It does not fabricate the measure.

A suitable evidence-grounded response envelope would be:

```json
{
  "metric_id": "metric.net_revenue",
  "reporting_date": "2025-03-31",
  "definition_version": "2",
  "definition_source": "synthetic://semantic/metric.net_revenue/v2",
  "status": "authoritative",
  "formula": "SUM(captured_payment_usd) - SUM(refund_amount_usd) - SUM(chargeback_amount_usd)",
  "grain": "order",
  "model": {
    "id": "model.fct_orders",
    "version": "2",
    "source": "synthetic://dbt/model.fct_orders/v2"
  },
  "lineage_source": "synthetic://lineage/model.fct_orders/v2",
  "conflicts": [
    {
      "id": "draft.net_revenue.2025-02",
      "status": "draft",
      "authoritative": false
    }
  ],
  "calculation_executed": false,
  "current_data_freshness_verified": false
}
```

That is much more auditable than “Based on the documents, revenue appears to be…”.

## Operational design and applicability

### Recommended architecture

My preferred design is **metadata-first retrieval with two-stage evidence handling**.

```text
                         ┌────────────────────┐
                         │ User / application │
                         └─────────┬──────────┘
                                   │
                          principal + question
                                   │
                    ┌──────────────▼──────────────┐
                    │ Intent / entity resolution │
                    │ exact IDs if available     │
                    │ search fallback otherwise  │
                    └──────────────┬──────────────┘
                                   │
                  ┌────────────────▼────────────────┐
                  │ Authorization candidate filter │
                  │ BEFORE semantic retrieval      │
                  └────────────────┬────────────────┘
                                   │
             ┌─────────────────────▼─────────────────────┐
             │ Evidence resolver                         │
             │ stable ID                                 │
             │ status / authority                        │
             │ effective time                            │
             │ model/metric version                      │
             └───────────────┬───────────────────────────┘
                             │
       ┌─────────────────────┼────────────────────────┐
       │                     │                        │
┌──────▼──────┐       ┌──────▼──────┐         ┌──────▼───────┐
│ metric API  │       │ lineage API │         │ source state │
│ / manifest  │       │ / graph     │         │ / freshness  │
└──────┬──────┘       └──────┬──────┘         └──────┬───────┘
       │                     │                        │
       └─────────────────────┼────────────────────────┘
                             │
             ┌───────────────▼──────────────────┐
             │ Search supporting documentation │
             │ lexical + semantic + reranking  │
             │ include conflict retrieval pass │
             └───────────────┬──────────────────┘
                             │
                  ┌──────────▼──────────┐
                  │ Grounded generation│
                  │ no calculation     │
                  │ exact citations    │
                  └──────────┬──────────┘
                             │
             ┌───────────────▼─────────────────┐
             │ SQL / metric engine if needed  │
             │ separate auditable operation   │
             └─────────────────────────────────┘
```

This exploits structured metadata already represented by dbt and lineage specifications instead of flattening everything into text. [1](https://docs.getdbt.com/reference/artifacts/manifest-json)

### Source-aware applicability matrix

| Situation | Preferred mechanism | Role for RAG | Why |
|---|---|---|---|
| User names exact metric/model and date | Deterministic metadata lookup | Usually none | Search uncertainty has already disappeared |
| User says “revenue” but several entities exist | Lexical/hybrid discovery → clarification/entity resolution | Useful | Semantic aliases may bridge terminology |
| Need exact source-to-column lineage | Graph/manifest/OpenLineage lookup | Explain results | Lineage is structured dependency information. [1](https://docs.getdbt.com/reference/artifacts/manifest-json) |
| Need applicable historical definition | Effective-time resolver | Explain selected version and conflicts | Similarity does not encode business effective time |
| Need explanation spanning metric, model and migration memo | Structured lookup + hybrid search | Strong fit | Narrative synthesis adds value after source selection |
| Search corpus is tiny and clean | Existing catalog/full-text/manual search | Often unnecessary | Added embedding/RAG complexity may have little leverage |
| Source authority is undefined | Fix governance first | Poor fit | A model cannot reliably manufacture institutional authority |
| ACL cannot be enforced before retrieval | Partition/fix authorization architecture | Do not deploy broad shared RAG | Post-generation hiding is too late |
| Need current numerical KPI | Metric/SQL engine | Explain query/result | Retrieval of documentation is not calculation |
| Docs change extremely rapidly | Live metadata/tool lookup | Optional explanation | Stale indexes become the dominant risk |
| Multiple authoritative sources disagree | Explicit conflict state + human review | Summarize disagreement | Generation should not silently choose |
| Low-stakes exploratory discovery | Search/RAG | Good fit | Precision of source-of-truth selection is less critical |
| Audited financial/regulatory answer | Deterministic source trace + calculation + review | Limited explanation role | Reproducibility and provenance dominate |

The long-context alternative—placing all documentation into a prompt—is reasonable only for sufficiently small, permission-homogeneous, low-risk corpora, and still needs explicit source/version metadata. The position sensitivity found by Liu et al. is one reason not to equate “present somewhere in context” with “reliably used.” [14](https://aclanthology.org/2024.tacl-1.9/)

### Chunking recommendation

Use a hierarchy such as:

```text
metric.net_revenue.v2
  ├─ canonical metric object
  ├─ formula chunk
  ├─ dimensions/filter chunk
  ├─ backing-model relation
  ├─ decision/migration explanation
  └─ aliases

model.fct_orders.v2
  ├─ model object
  ├─ grain
  ├─ schema/column chunks
  └─ lineage edges
```

Every derived chunk should carry:

```text
parent stable ID
version
authority/status
valid_from / valid_to
ACL policy reference
source URI
content hash
```

Do not chunk a metric definition so aggressively that its formula is separated from its grain, filters, effective period, and identity. A semantically relevant formula without the correct grain or filter is not a correct analytical definition.

### Index invalidation

A minimal invalidation key should cover both content and authorization:

```text
document_key      = stable_id + version
content_fingerprint = SHA256(canonical_source_content)
acl_fingerprint     = SHA256(canonical_authorization_policy)
```

On a documentation change:

```text
source event
→ compare content fingerprint
→ replace/tombstone affected chunks
→ retain old version if historically valid
→ update effective-time metadata
```

On an ACL change:

```text
authorization event
→ update policy immediately
→ retrieval-time ACL enforcement becomes authoritative
→ asynchronously reindex/partition if necessary
```

The essential property is that **a stale vector index must never override a current deny decision**.

### Retrieval should have a conflict mode

A single ranking objective is insufficient.

For ordinary answers:

```text
score =
retrieval relevance
+ authority
+ effective-time fit
+ source-type preference
```

For conflict diagnosis:

```text
retrieve:
authoritative source
AND
highly relevant non-authoritative/superseded sources

then label their relationship
```

This avoids the failure observed in the synthetic reranker, where promoting authoritative sources inadvertently hid the very draft/wiki needed to explain the discrepancy.

### Evaluation scorecard for a production pilot

The production test should report at least:

| Layer | Metric | Failure example |
|---|---|---|
| Intent | entity-resolution accuracy | “ARR” maps to wrong metric |
| Retrieval | required evidence recall@k | correct metric definition absent |
| Retrieval | authoritative top-1 rate | draft ranks first |
| Versioning | effective-version fidelity | v2 used for 2024 report |
| Lineage | edge/path correctness | wrong upstream source |
| Permissions | unauthorized retrieval rate | finance-only chunk returned |
| Conflict | conflict recall | draft contradiction not surfaced |
| Generation | factual fidelity | formula altered in prose |
| Citation | source/version fidelity | answer says v2 but cites v1 |
| Ambiguity | clarification precision/recall | asks unnecessarily / fails to ask |
| Abstention | unsupported-answer rate | invents “recognized revenue” |
| Calculation | SQL/metric result correctness | definition right, computation wrong |
| Reproducibility | same snapshot reproduces answer | index/version not recorded |
| Operations | p50/p95 latency | reranker adds unacceptable latency |
| Operations | cost/query | unnecessary model/tool calls |
| Recovery | correction success | wrong result persists after source fix |

The separation is consistent with RAGChecker's modular evaluation framing and ALCE's distinction between correctness and citation quality. [4](https://arxiv.org/abs/2408.08067)

No meaningful production latency or cost figures are reported here: the corpus is too small, retrieval is local in memory, and no hosted embedding or LLM calls were used. Reporting its microbenchmark latency as a production estimate would be misleading.

### Force clarification when

The system should ask rather than guess when the missing value can change the authoritative source, notably:

```text
reporting date omitted and multiple time-valid definitions exist
business term maps to multiple distinct metric IDs
requested grain is materially ambiguous
user asks "current" but report period is historical
```

### Force abstention when

```text
no accessible authoritative definition exists
required source/version cannot be verified
the requested entity does not exist in authoritative metadata
lineage needed for the answer is unavailable
current data freshness is requested but no freshness observation exists
authorization cannot safely be resolved
```

For denied and nonexistent objects, returning a common `not_found_or_not_authorized` state can avoid leaking the existence of a restricted object.

### Force human review when

```text
two accessible authoritative sources are simultaneously valid and conflict
source-authority policy does not establish precedence
the system proposes changing a material financial definition
lineage/definition evidence is incomplete for an audited conclusion
a permission boundary is uncertain
```

## Contradictions, limitations, and conditions that reverse the recommendation

### Dense versus lexical retrieval is not settled by generic benchmarks

DPR's open-domain QA study found dense retrieval substantially stronger than its BM25 comparison on that benchmark family. [2](https://aclanthology.org/2020.emnlp-main.550/) BEIR found BM25 surprisingly robust across heterogeneous zero-shot retrieval and warned, in effect, against assuming in-domain neural gains generalize uniformly. [12](https://arxiv.org/html/2104.08663v4)

These results are not actually contradictory once scope is considered. Analytical documentation mixes two retrieval regimes:

```text
semantic language:
"money we really collected after reversals"
≈ net revenue

symbolic identifiers:
metric.net_revenue
fct_orders_v2
raw_payments.payment_events
chargeback_amount_usd
```

Dense retrieval may help the former while lexical/exact matching is naturally strong for the latter. A hybrid route is therefore plausible, but it still must be evaluated on the target corpus.

### High retrieval recall can conflict with authoritative ranking

The synthetic experiment makes this tension explicit. The vector route retrieved relevant material well, but retrieval relevance alone did not reliably identify the governing source.

Conversely, authority-aware reranking hid non-authoritative material that was valuable when diagnosing conflicts.

**Condition reversing a simple “downrank drafts” recommendation:** when a user explicitly asks “why does this draft disagree with production?”, retrieving the draft is necessary even though it must not govern the answer.

### RAG improves access to external evidence but does not guarantee evidence obedience

The original RAG work supports the value of external non-parametric memory in its task domains. [3](https://arxiv.org/abs/2005.11401) Citation-generation research subsequently found substantial room for improvement in whether model claims are actually fully supported by cited material. [13](https://aclanthology.org/2023.emnlp-main.398/)

Therefore “the answer has links” is not a sufficient source-fidelity metric.

### Long context is a viable alternative in some cases

A ten-document corpus may not justify an elaborate retrieval stack at all. Manual search, deterministic lookup, or supplying a small authorized source bundle may be simpler.

The case for RAG strengthens when:

```text
corpus size grows
business terminology diverges from technical names
source types become heterogeneous
users often do not know the entry-point object
supporting explanation spans several documents
```

It weakens when:

```text
objects have stable IDs
queries are predictable
the corpus is small
effective-time resolution dominates the problem
all useful facts are already in structured metadata
```

### Bounded agents are an orchestration alternative, not evidence of correctness

A tool-using agent may be more natural than classic “retrieve chunks → stuff prompt” RAG for a multi-hop question:

```text
resolve metric
→ fetch model
→ traverse lineage
→ check freshness
→ optionally execute metric
```

But no LLM agent was executed in this dossier. Only the deterministic tool-core behavior was simulated using gold structured targets. Tool-selection accuracy, looping behavior, latency, and recovery remain empirical questions.

### Prompt injection is real, but retrieval probability matters

Indirect prompt injection is an established attack mechanism for LLM-integrated applications that consume external content. [15](https://arxiv.org/abs/2302.12173) The 2026 USENIX study is important counterweight against simplistic threat claims: it reports that unoptimized attacks are often not retrieved under ordinary natural queries, then develops an approach specifically designed to overcome that retrieval obstacle. [16](https://www.usenix.org/conference/usenixsecurity26/presentation/chang-hongyan)

The appropriate interpretation is not “every document will compromise the agent,” nor “retrieval makes attacks harmless.” It is that externally controlled retrievable content forms an instruction-security boundary and should be handled accordingly.

### dbt metadata is useful but not identical to the desired governance model

dbt versions establish model versions; its manifest establishes resources and DAG relations; its metrics/semantic models establish structured analytical semantics; grants and access expose governance primitives. [6](https://docs.getdbt.com/reference/resource-properties/versions)

The synthetic system adds concepts such as explicit business-definition `valid_from`/`valid_to` and an organizational authority hierarchy. Those are **design assumptions**, not claims that dbt natively provides exactly this governance policy.

### Conditions under which I would reverse the main recommendation

I would favor a much simpler search/catalog solution rather than building RAG if an organization has:

```text
<small corpus>
+ strong stable IDs
+ good descriptions
+ reliable catalog search
+ infrequent questions
+ users who already know the data model
```

I would favor a live metadata/tool agent over indexed RAG if source objects or permissions change so rapidly that index invalidation becomes the largest reliability problem.

I would postpone any AI retrieval project entirely if the organization cannot answer:

```text
Which definition is authoritative?
Who owns it?
When did it apply?
What is its grain?
What object implements it?
Who may see it?
```

RAG can find ambiguity. It cannot turn undocumented institutional ambiguity into ground truth.

### Unresolved evidence gaps

There is no representative public benchmark I found that jointly tests:

```text
analytical metric definitions
+ dbt/model versions
+ effective-time semantics
+ structured lineage
+ document conflicts
+ per-user permissions
+ source/index invalidation
+ LLM answer fidelity
```

The generic retrieval benchmarks are therefore supporting background, not target-workflow validation.

Likewise, I found no credible public evidence allowing a production effect size such as “hybrid RAG reduces analyst errors by X%” for this exact workflow. No such claim should be inferred from this dossier.

The current dbt/OpenLineage provider documentation establishes available primitives, not customer accuracy or business outcomes.

**Source-access limitation:** the ACM landing page for the original Reciprocal Rank Fusion paper returned an internal retrieval error during this research session. I therefore did not rely on that page for an empirical performance claim. The synthetic code uses an explicitly visible RRF-style formula as a test mechanism. No inaccessible source was silently substituted.

## Claim-to-source dossier

The table below contains the sources carrying the major external claims. “Inspected” means accessed for this research on **2026-10-10**.

| Claim / use | Source | Source date / version | Exact scope | Evidence type | Direct URL / DOI |
|---|---|---|---|---|---|
| dbt model versions represent tracked model changes; breaking contracts can be detected | dbt, “versions” | Current page; updated Aug. 27, 2026 | dbt model-version semantics and contract/version guidance | Provider documentation | https://docs.getdbt.com/reference/resource-properties/versions |
| `manifest.json` exposes nodes, sources, metrics, docs, `parent_map`, `child_map`, and `unique_id` | dbt, “Manifest JSON file” | Updated Sep. 8, 2026 | dbt-generated manifest artifact | Provider documentation | https://docs.getdbt.com/reference/artifacts/manifest-json |
| dbt documentation combines DAG/project information, warehouse metadata, and descriptions | dbt, “About documentation” | Current page inspected 2026-10-10 | dbt documentation generation | Provider documentation | https://docs.getdbt.com/docs/build/documentation |
| Semantic models are structured graph nodes connected through entities | dbt, “Semantic models” | Current docs, dbt v1.12+ material | Semantic Layer / MetricFlow semantic structure | Provider documentation | https://docs.getdbt.com/docs/build/semantic-models |
| Metrics are structured named/type-specific YAML objects | dbt, “Creating metrics” | Updated Jul. 20, 2026 | dbt metric definitions | Provider documentation | https://docs.getdbt.com/docs/build/metrics-overview |
| Some filter-based metric dependencies are not represented in selection syntax/DAG visualization | dbt, “Semantic models” | Current docs | Metric dependency caveat | Provider documentation | https://docs.getdbt.com/docs/build/semantic-models |
| dbt model access is distinct from user access and database permissions | dbt, “Model access” | Current page, 2026 | dbt Mesh access semantics | Provider documentation | https://docs.getdbt.com/docs/mesh/govern/model-access |
| dbt grants configure built-dataset permissions | dbt, “grants” | Updated Jul. 21, 2026 | dbt grant resource configuration | Provider documentation | https://docs.getdbt.com/reference/resource-configs/grants |
| dbt freshness expresses warn/error recency thresholds and ways to determine latest loaded time | dbt, “freshness” | Updated Sep. 16, 2026 | dbt source/model freshness configuration | Provider documentation | https://docs.getdbt.com/reference/resource-configs/freshness |
| OpenLineage separates jobs, datasets, runs and metadata updates and can capture source version | OpenLineage Object Model | Spec 1.53.0; publication date not stated | OpenLineage core object model | Versioned official specification | https://openlineage.io/docs/spec/object-model/ |
| OpenLineage column lineage represents input/output column dependencies and transformation types | OpenLineage Column Lineage Dataset Facet | Spec 1.53.0 | Column-level lineage representation | Versioned official specification | https://openlineage.io/docs/spec/facets/dataset-facets/column_lineage_facet/ |
| RAG combines parametric generation with retrieved non-parametric memory; evaluated on knowledge-intensive NLP | Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” | Submitted May 22, 2020; arXiv v4 Apr. 12, 2021; NeurIPS 2020 | Wikipedia-backed RAG on general NLP tasks | Primary empirical research | https://arxiv.org/abs/2005.11401 ; DOI: 10.48550/arXiv.2005.11401 |
| Dense passage retrieval beat the paper's Lucene-BM25 baseline by 9–19 pp top-20 accuracy | Karpukhin et al., “Dense Passage Retrieval for Open-Domain Question Answering” | EMNLP 2020 | Open-domain QA passage retrieval | Peer-reviewed primary empirical study | https://aclanthology.org/2020.emnlp-main.550/ ; DOI: 10.18653/v1/2020.emnlp-main.550 |
| BM25 is a robust heterogeneous zero-shot baseline; reranking/late interaction lead average performance at higher computational cost | Thakur et al., BEIR | arXiv v4 Oct. 21, 2021 | 18 datasets, multiple retrieval task/domain types | Primary benchmark study | https://arxiv.org/abs/2104.08663 |
| Long-context model performance can vary materially with position of relevant evidence | Liu et al., “Lost in the Middle” | TACL 2024 | Multi-document QA and key-value retrieval | Peer-reviewed primary empirical study | https://aclanthology.org/2024.tacl-1.9/ ; DOI: 10.1162/tacl_a_00638 |
| RAG evaluation should diagnose retrieval and generation separately | Ru et al., RAGChecker | arXiv v2 Aug. 17, 2024 | Fine-grained RAG evaluation framework | Primary preprint/evaluation study | https://arxiv.org/abs/2408.08067 ; DOI: 10.48550/arXiv.2408.08067 |
| Citation correctness/support is a distinct failure dimension; substantial gaps remained in evaluated systems | Gao et al., ALCE | EMNLP 2023 | Citation-aware LLM QA over three benchmark families | Peer-reviewed primary empirical study | https://aclanthology.org/2023.emnlp-main.398/ ; DOI: 10.18653/v1/2023.emnlp-main.398 |
| Retrieved external content can carry indirect prompt-injection instructions | Greshake et al., “Not what you've signed up for” | 2023 | Indirect prompt injection in LLM-integrated applications | Primary security research | https://arxiv.org/abs/2302.12173 |
| Retrieval itself can be a barrier to prompt-injection attacks; attackers can optimize content for retrievability | Chang et al., “Overcoming the Retrieval Barrier” | USENIX Security 2026 | Retrieval-targeted indirect prompt injection | Peer-reviewed primary security research / conference page | https://www.usenix.org/conference/usenixsecurity26/presentation/chang-hongyan |

All of these were inspected on October 10, 2026. The provider documents are evidence about product/specification behavior, **not independent empirical evidence that a RAG implementation using those products will be correct**.

## Reusable corpus, gold labels, code asset, article angles, and freshness triggers

The following is the **complete synthetic asset used for the executed retrieval diagnostic**. It is intentionally small enough to audit manually. It requires no external data or model downloads. Requirements are Python, NumPy, and scikit-learn.

```python
"""
Synthetic analytical-documentation retrieval diagnostic.

Purpose:
- distinguish retrieval relevance from source authority/version correctness
- exercise lexical, local dense-vector, hybrid, ACL, effective-time,
  conflict, clarification and abstention cases

Dependencies:
    Python 3.10+
    numpy
    scikit-learn

No network access or external model is required.

Important limitation:
The "dense" route is LSA (TF-IDF + TruncatedSVD), not a neural embedding model.
"""

from __future__ import annotations

from datetime import date
import math
import re
from typing import Any

import numpy as np
from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.preprocessing import normalize


CORPUS: list[dict[str, Any]] = [
    {
        "id": "metric.net_revenue.v1",
        "entity_id": "metric.net_revenue",
        "type": "metric_definition",
        "title": "Net revenue semantic metric v1",
        "status": "authoritative",
        "version": "1",
        "valid_from": "2024-01-01",
        "valid_to": "2025-01-01",
        "supersedes": None,
        "acl": ["analyst", "finance"],
        "source_uri": "synthetic://semantic/metric.net_revenue/v1",
        "text": (
            "Authoritative semantic metric. Net revenue at reporting dates "
            "from 2024-01-01 through 2024-12-31 is "
            "SUM(order_total_usd) - SUM(refund_amount_usd). "
            "Grain: one row per order after aggregation. Currency: USD. "
            "Supported by dbt model ref('fct_orders', version=1)."
        ),
    },
    {
        "id": "metric.net_revenue.v2",
        "entity_id": "metric.net_revenue",
        "type": "metric_definition",
        "title": "Net revenue semantic metric v2",
        "status": "authoritative",
        "version": "2",
        "valid_from": "2025-01-01",
        "valid_to": None,
        "supersedes": "metric.net_revenue.v1",
        "acl": ["analyst", "finance"],
        "source_uri": "synthetic://semantic/metric.net_revenue/v2",
        "text": (
            "Authoritative semantic metric. Net revenue for reporting dates "
            "on or after 2025-01-01 is "
            "SUM(captured_payment_usd) - SUM(refund_amount_usd) "
            "- SUM(chargeback_amount_usd). "
            "Grain: one row per order after payment-event aggregation. "
            "Currency: USD. Supported by dbt model "
            "ref('fct_orders', version=2)."
        ),
    },
    {
        "id": "model.fct_orders.v1",
        "entity_id": "model.fct_orders",
        "type": "dbt_model_schema",
        "title": "dbt model fct_orders v1 schema",
        "status": "authoritative",
        "version": "1",
        "valid_from": "2024-01-01",
        "valid_to": "2025-01-01",
        "supersedes": None,
        "acl": ["analyst", "finance"],
        "source_uri": "synthetic://dbt/model.fct_orders/v1",
        "text": (
            "fct_orders version 1. Grain: one row per order. "
            "Columns include order_id, ordered_at, order_total_usd, "
            "refund_amount_usd. order_total_usd and refund_amount_usd "
            "originate from stg_orders. Contracted model version 1."
        ),
    },
    {
        "id": "model.fct_orders.v2",
        "entity_id": "model.fct_orders",
        "type": "dbt_model_schema",
        "title": "dbt model fct_orders v2 schema",
        "status": "authoritative",
        "version": "2",
        "valid_from": "2025-01-01",
        "valid_to": None,
        "supersedes": "model.fct_orders.v1",
        "acl": ["analyst", "finance"],
        "source_uri": "synthetic://dbt/model.fct_orders/v2",
        "text": (
            "fct_orders version 2. Grain: one row per order. "
            "Columns include order_id, ordered_at, captured_payment_usd, "
            "refund_amount_usd, chargeback_amount_usd. "
            "Payment measures are aggregated from stg_payment_events "
            "and joined to stg_orders. Contracted model version 2."
        ),
    },
    {
        "id": "lineage.fct_orders.v1",
        "entity_id": "lineage.fct_orders",
        "type": "lineage",
        "title": "Lineage for fct_orders v1",
        "status": "authoritative",
        "version": "1",
        "valid_from": "2024-01-01",
        "valid_to": "2025-01-01",
        "supersedes": None,
        "acl": ["analyst", "finance"],
        "source_uri": "synthetic://lineage/model.fct_orders/v1",
        "text": (
            "Lineage version 1: raw_app.orders -> stg_orders "
            "-> fct_orders version 1. Net revenue v1 uses "
            "order_total_usd and refund_amount_usd from raw_app.orders."
        ),
    },
    {
        "id": "lineage.fct_orders.v2",
        "entity_id": "lineage.fct_orders",
        "type": "lineage",
        "title": "Lineage for fct_orders v2",
        "status": "authoritative",
        "version": "2",
        "valid_from": "2025-01-01",
        "valid_to": None,
        "supersedes": "lineage.fct_orders.v1",
        "acl": ["analyst", "finance"],
        "source_uri": "synthetic://lineage/model.fct_orders/v2",
        "text": (
            "Lineage version 2: raw_app.orders -> stg_orders "
            "-> fct_orders version 2; "
            "raw_payments.payment_events -> stg_payment_events "
            "-> fct_orders version 2. captured_payment_usd, "
            "refund_amount_usd, and chargeback_amount_usd originate "
            "from raw_payments.payment_events."
        ),
    },
    {
        "id": "freshness.raw_payments.payment_events",
        "entity_id": "source.raw_payments.payment_events",
        "type": "freshness_policy",
        "title": "Freshness policy for raw_payments.payment_events",
        "status": "authoritative",
        "version": "1",
        "valid_from": "2025-01-01",
        "valid_to": None,
        "supersedes": None,
        "acl": ["analyst", "finance"],
        "source_uri": (
            "synthetic://dbt/source.raw_payments.payment_events/freshness"
        ),
        "text": (
            "Source raw_payments.payment_events freshness policy: "
            "warn if loaded_at is older than 2 hours; "
            "error if older than 6 hours. "
            "This is a freshness expectation, not evidence of "
            "the current load timestamp."
        ),
    },
    {
        "id": "wiki.revenue.legacy",
        "entity_id": "doc.revenue_wiki",
        "type": "handbook",
        "title": "Revenue Wiki (superseded)",
        "status": "superseded",
        "version": "2024-02",
        "valid_from": "2024-02-01",
        "valid_to": "2025-01-01",
        "supersedes": None,
        "acl": ["analyst", "finance"],
        "source_uri": "synthetic://wiki/revenue/2024-02",
        "text": (
            "Superseded Revenue Wiki. Revenue equals order total less "
            "refunds: SUM(order_total_usd) - SUM(refund_amount_usd). "
            "This page was superseded on 2025-01-01; "
            "do not use it for later reporting dates."
        ),
    },
    {
        "id": "draft.net_revenue.2025-02",
        "entity_id": "metric.net_revenue",
        "type": "draft_memo",
        "title": "Draft proposal: net revenue treatment",
        "status": "draft",
        "version": "draft-2025-02-10",
        "valid_from": "2025-02-10",
        "valid_to": None,
        "supersedes": None,
        "acl": ["analyst", "finance"],
        "source_uri": "synthetic://drafts/net_revenue/2025-02-10",
        "text": (
            "DRAFT, NOT APPROVED. Proposal: net revenue equals "
            "captured payments minus refunds; exclude chargebacks until "
            "month-end close. This conflicts with the authoritative "
            "semantic metric v2, which subtracts chargebacks for reporting "
            "dates on or after 2025-01-01."
        ),
    },
    {
        "id": "finance.board_revenue.private",
        "entity_id": "metric.board_revenue",
        "type": "finance_policy",
        "title": "Board revenue policy",
        "status": "authoritative",
        "version": "3",
        "valid_from": "2025-01-01",
        "valid_to": None,
        "supersedes": None,
        "acl": ["finance"],
        "source_uri": "synthetic://finance/board_revenue/v3",
        "text": (
            "CONFIDENTIAL FINANCE ONLY. Board revenue is defined as "
            "[synthetic confidential fixture]. This document must not be "
            "returned to analyst-role requesters."
        ),
    },
    {
        "id": "policy.authority_resolution.v1",
        "entity_id": "policy.authority_resolution",
        "type": "governance_policy",
        "title": "Analytical documentation authority and effective-time policy",
        "status": "authoritative",
        "version": "1",
        "valid_from": "2024-01-01",
        "valid_to": None,
        "supersedes": None,
        "acl": ["analyst", "finance"],
        "source_uri": "synthetic://policy/authority_resolution/v1",
        "text": (
            "Resolution policy: enforce requester ACL before retrieval. "
            "For analytical definitions, prefer approved semantic metric "
            "specifications over dbt model descriptions over approved "
            "handbook prose; drafts are non-authoritative. "
            "Apply the version whose valid_from <= reporting_date < valid_to, "
            "with open valid_to meaning current. If reporting_date is "
            "required but missing, ask for clarification. If no accessible "
            "authoritative source exists, abstain."
        ),
    },
]


GOLD: list[dict[str, Any]] = [
    {
        "id": "Q1",
        "query": (
            "For reporting date 2024-06-30, "
            "what definition of net revenue applies?"
        ),
        "as_of": "2024-06-30",
        "role": "analyst",
        "required": ["metric.net_revenue.v1"],
        "selected": "metric.net_revenue.v1",
        "behavior": "answer",
        "target": "metric.net_revenue",
    },
    {
        "id": "Q2",
        "query": (
            "For reporting date 2025-03-31, "
            "what definition of net revenue applies?"
        ),
        "as_of": "2025-03-31",
        "role": "analyst",
        "required": ["metric.net_revenue.v2"],
        "selected": "metric.net_revenue.v2",
        "behavior": "answer",
        "target": "metric.net_revenue",
    },
    {
        "id": "Q3",
        "query": (
            "For net revenue on 2025-03-31, "
            "which dbt model version supports the metric?"
        ),
        "as_of": "2025-03-31",
        "role": "analyst",
        "required": [
            "metric.net_revenue.v2",
            "model.fct_orders.v2",
        ],
        "selected": "model.fct_orders.v2",
        "behavior": "answer",
        "target": "model.fct_orders",
    },
    {
        "id": "Q4",
        "query": (
            "Which raw sources feed fct_orders version 2 for net revenue?"
        ),
        "as_of": "2025-03-31",
        "role": "analyst",
        "required": ["lineage.fct_orders.v2"],
        "selected": "lineage.fct_orders.v2",
        "behavior": "answer",
        "target": "lineage.fct_orders",
    },
    {
        "id": "Q5",
        "query": "What is the grain of fct_orders version 2?",
        "as_of": "2025-03-31",
        "role": "analyst",
        "required": ["model.fct_orders.v2"],
        "selected": "model.fct_orders.v2",
        "behavior": "answer",
        "target": "model.fct_orders",
    },
    {
        "id": "Q6",
        "query": (
            "What freshness expectation applies to "
            "raw_payments.payment_events?"
        ),
        "as_of": "2025-03-31",
        "role": "analyst",
        "required": ["freshness.raw_payments.payment_events"],
        "selected": "freshness.raw_payments.payment_events",
        "behavior": "answer",
        "target": "source.raw_payments.payment_events",
    },
    {
        "id": "Q7",
        "query": "What is the net revenue definition?",
        "as_of": None,
        "role": "analyst",
        "required": [],
        "selected": None,
        "behavior": "clarify",
        "target": "metric.net_revenue",
    },
    {
        "id": "Q8",
        "query": (
            "According to the 2025 draft, net revenue excludes "
            "chargebacks until month-end close, right?"
        ),
        "as_of": "2025-03-31",
        "role": "analyst",
        "required": [
            "draft.net_revenue.2025-02",
            "metric.net_revenue.v2",
        ],
        "selected": "metric.net_revenue.v2",
        "behavior": "conflict",
        "target": "metric.net_revenue",
    },
    {
        "id": "Q9",
        "query": "What is the board revenue definition?",
        "as_of": "2025-03-31",
        "role": "analyst",
        "required": [],
        "selected": None,
        "forbidden": ["finance.board_revenue.private"],
        "behavior": "deny_or_abstain",
        "target": "metric.board_revenue",
    },
    {
        "id": "Q10",
        "query": (
            "What is the recognized revenue definition "
            "for reporting date 2025-03-31?"
        ),
        "as_of": "2025-03-31",
        "role": "analyst",
        "required": [],
        "selected": None,
        "behavior": "abstain",
        "target": "metric.recognized_revenue",
    },
    {
        "id": "Q11",
        "query": (
            "The old Revenue Wiki says revenue is order total minus refunds. "
            "Is that current for 2025-03-31?"
        ),
        "as_of": "2025-03-31",
        "role": "analyst",
        "required": [
            "wiki.revenue.legacy",
            "metric.net_revenue.v2",
        ],
        "selected": "metric.net_revenue.v2",
        "behavior": "conflict",
        "target": "metric.net_revenue",
    },
]


def tokenize(value: str) -> list[str]:
    return re.findall(r"[a-z0-9_\.']+", value.lower())


DOCUMENT_TEXT = [
    item["title"] + " " + item["text"]
    for item in CORPUS
]

TOKENIZED_DOCUMENTS = [
    tokenize(value)
    for value in DOCUMENT_TEXT
]

N = len(CORPUS)
AVG_DOCUMENT_LENGTH = sum(map(len, TOKENIZED_DOCUMENTS)) / N

DOCUMENT_FREQUENCY: dict[str, int] = {}
for tokens in TOKENIZED_DOCUMENTS:
    for token in set(tokens):
        DOCUMENT_FREQUENCY[token] = (
            DOCUMENT_FREQUENCY.get(token, 0) + 1
        )


def bm25_scores(
    query: str,
    k1: float = 1.5,
    b: float = 0.75,
) -> np.ndarray:
    query_tokens = tokenize(query)
    scores: list[float] = []

    for tokens in TOKENIZED_DOCUMENTS:
        counts: dict[str, int] = {}
        for token in tokens:
            counts[token] = counts.get(token, 0) + 1

        document_length = len(tokens)
        score = 0.0

        for token in query_tokens:
            n = DOCUMENT_FREQUENCY.get(token, 0)
            if n == 0:
                continue

            inverse_document_frequency = math.log(
                1 + (N - n + 0.5) / (n + 0.5)
            )

            term_frequency = counts.get(token, 0)
            if term_frequency == 0:
                continue

            denominator = (
                term_frequency
                + k1
                * (
                    1
                    - b
                    + b * document_length / AVG_DOCUMENT_LENGTH
                )
            )

            score += (
                inverse_document_frequency
                * term_frequency
                * (k1 + 1)
                / denominator
            )

        scores.append(score)

    return np.asarray(scores, dtype=float)


# Local dense-vector diagnostic.
TFIDF = TfidfVectorizer(
    lowercase=True,
    token_pattern=r"(?u)\b[\w\.\-']+\b",
    ngram_range=(1, 2),
)

TFIDF_DOCUMENTS = TFIDF.fit_transform(DOCUMENT_TEXT)

N_COMPONENTS = min(
    8,
    TFIDF_DOCUMENTS.shape[0] - 1,
    TFIDF_DOCUMENTS.shape[1] - 1,
)

SVD = TruncatedSVD(
    n_components=N_COMPONENTS,
    random_state=0,
)

DENSE_DOCUMENTS = normalize(
    SVD.fit_transform(TFIDF_DOCUMENTS)
)


def lsa_scores(query: str) -> np.ndarray:
    query_sparse = TFIDF.transform([query])
    query_dense = normalize(SVD.transform(query_sparse))
    return (DENSE_DOCUMENTS @ query_dense.T).ravel()


def reciprocal_rank_fusion(
    query: str,
    rank_constant: int = 60,
) -> np.ndarray:
    """Fuse BM25 and LSA ranks. rank_constant=60 is a test assumption."""
    result = np.zeros(len(CORPUS), dtype=float)

    for score_array in (
        bm25_scores(query),
        lsa_scores(query),
    ):
        order = np.argsort(-score_array)
        ranks = np.empty_like(order)
        ranks[order] = np.arange(1, len(score_array) + 1)

        result += 1.0 / (rank_constant + ranks)

    return result


def valid_on(
    document: dict[str, Any],
    as_of: str | None,
) -> bool:
    if as_of is None:
        return True

    observed = date.fromisoformat(as_of)

    valid_from = (
        date.fromisoformat(document["valid_from"])
        if document["valid_from"]
        else date.min
    )

    valid_to = (
        date.fromisoformat(document["valid_to"])
        if document["valid_to"]
        else date.max
    )

    return valid_from <= observed < valid_to


def retrieve(
    query: str,
    *,
    method: str,
    role: str,
    as_of: str | None,
    top_k: int = 3,
    acl_filter: bool = True,
    metadata_rerank: bool = False,
) -> list[str]:
    if method == "bm25":
        scores = bm25_scores(query).copy()
    elif method == "lsa":
        scores = lsa_scores(query).copy()
    elif method == "hybrid":
        scores = reciprocal_rank_fusion(query).copy()
    else:
        raise ValueError(f"Unsupported method: {method}")

    # Synthetic weights chosen only for this diagnostic.
    for index, document in enumerate(CORPUS):
        if acl_filter and role not in document["acl"]:
            scores[index] = -1e9
            continue

        if metadata_rerank:
            if as_of is not None:
                scores[index] += (
                    0.05 if valid_on(document, as_of) else -0.10
                )

            scores[index] += {
                "authoritative": 0.02,
                "draft": -0.005,
                "superseded": -0.01,
            }.get(document["status"], 0.0)

    order = np.argsort(-scores)

    results: list[str] = []
    for index in order:
        if scores[index] <= -1e8:
            continue

        results.append(CORPUS[index]["id"])

        if len(results) >= top_k:
            break

    return results


DATE_REQUIRED_ENTITY_IDS = {
    "metric.net_revenue",
    "model.fct_orders",
    "lineage.fct_orders",
    "metric.board_revenue",
    "metric.recognized_revenue",
}


def deterministic_lookup(
    entity_id: str,
    *,
    as_of: str | None,
    role: str,
) -> dict[str, str | None]:
    """
    This tests structured resolution only.
    Natural-language -> entity_id extraction is NOT tested.
    """

    candidates = [
        document
        for document in CORPUS
        if (
            document["entity_id"] == entity_id
            and document["status"] == "authoritative"
            and role in document["acl"]
        )
    ]

    if (
        entity_id in DATE_REQUIRED_ENTITY_IDS
        and as_of is None
    ):
        return {
            "status": "clarify_reporting_date",
            "id": None,
        }

    if as_of is not None:
        candidates = [
            document
            for document in candidates
            if valid_on(document, as_of)
        ]

    if not candidates:
        # Deliberately does not distinguish missing from unauthorized.
        return {
            "status": "not_found_or_not_authorized",
            "id": None,
        }

    # Only one applicable authoritative fixture is expected in this corpus.
    return {
        "status": "resolved",
        "id": candidates[-1]["id"],
    }


METHODS = [
    ("BM25 + ACL", "bm25", False),
    ("LSA dense-vector + ACL", "lsa", False),
    ("Hybrid RRF + ACL", "hybrid", False),
    ("Hybrid + metadata rerank", "hybrid", True),
]


def evaluate_retrieval() -> None:
    scored_questions = [
        question
        for question in GOLD
        if question["selected"] is not None
    ]

    print(
        "method,"
        "selected_authoritative_top1,"
        "required_evidence_recall_at_3"
    )

    for label, method, rerank in METHODS:
        correct_top1 = 0
        recalls: list[float] = []

        for question in scored_questions:
            returned = retrieve(
                question["query"],
                method=method,
                role=question["role"],
                as_of=question["as_of"],
                top_k=3,
                acl_filter=True,
                metadata_rerank=rerank,
            )

            if returned[0] == question["selected"]:
                correct_top1 += 1

            required = question["required"]
            recall = (
                sum(
                    evidence_id in returned
                    for evidence_id in required
                )
                / len(required)
            )
            recalls.append(recall)

        top1_rate = correct_top1 / len(scored_questions)
        recall_at_3 = sum(recalls) / len(recalls)

        print(
            f"{label},"
            f"{top1_rate:.3f},"
            f"{recall_at_3:.3f}"
        )


def evaluate_deterministic_resolution() -> None:
    correct = 0

    for question in GOLD:
        result = deterministic_lookup(
            question["target"],
            as_of=question["as_of"],
            role=question["role"],
        )

        if question["behavior"] in {"answer", "conflict"}:
            passed = result["id"] == question["selected"]

        elif question["behavior"] == "clarify":
            passed = (
                result["status"]
                == "clarify_reporting_date"
            )

        elif question["behavior"] in {
            "deny_or_abstain",
            "abstain",
        }:
            passed = (
                result["status"]
                == "not_found_or_not_authorized"
            )

        else:
            passed = False

        correct += int(passed)

        print(
            question["id"],
            result["status"],
            result["id"],
            "PASS" if passed else "FAIL",
        )

    print(
        f"deterministic_gold_behavior="
        f"{correct}/{len(GOLD)}"
    )


def demonstrate_permission_prefilter() -> None:
    question = next(
        question
        for question in GOLD
        if question["id"] == "Q9"
    )

    print("Q9 permission diagnostic")

    for method in ("bm25", "lsa", "hybrid"):
        without_acl = retrieve(
            question["query"],
            method=method,
            role=question["role"],
            as_of=question["as_of"],
            top_k=3,
            acl_filter=False,
        )

        with_acl = retrieve(
            question["query"],
            method=method,
            role=question["role"],
            as_of=question["as_of"],
            top_k=3,
            acl_filter=True,
        )

        print(
            method,
            "without_acl=", without_acl,
            "with_acl=", with_acl,
        )


if __name__ == "__main__":
    evaluate_retrieval()
    print()
    evaluate_deterministic_resolution()
    print()
    demonstrate_permission_prefilter()
```

The corresponding executed aggregate output was:

```text
method,selected_authoritative_top1,required_evidence_recall_at_3
BM25 + ACL,0.375,0.750
LSA dense-vector + ACL,0.375,0.938
Hybrid RRF + ACL,0.375,0.875
Hybrid + metadata rerank,0.625,0.875

deterministic_gold_behavior=11/11
```

Again, `11/11` is **conditional on supplying the gold structured target ID**; the code intentionally does not pretend that natural-language entity resolution was solved.

### Candidate article angles

**“RAG Is the Wrong Place to Decide What Revenue Means.”**  
The sharpest practical angle: similarity search should discover evidence; authority, effective time, lineage, and permissions need explicit system semantics.

**“The Three Clocks an Analytics Copilot Must Not Confuse.”**  
Data freshness, documentation/index freshness, and metric-definition effective time are different states. This has a clean worked revenue example and is unusually concrete.

**“Lineage Is a Graph, Not a Chunk.”**  
A focused explanation of why source/model/column lineage belongs in dbt manifests/OpenLineage-style structures and why RAG should verbalize, not reconstruct, the graph. The dbt filter-dependency caveat makes the article less simplistic. [1](https://docs.getdbt.com/reference/artifacts/manifest-json)

**“High Recall, Wrong Definition.”**  
Use the synthetic diagnostic to show why retrieval recall and source-of-truth fidelity are separate evaluation axes. The headline number must remain explicitly labeled as a tiny synthetic experiment, not a production benchmark.

**“Permissions Before Embeddings.”**  
Why model access, database grants, documentation access, and RAG candidate filtering are separate layers, with the denied-document fixture as an executable regression test. [11](https://docs.getdbt.com/docs/mesh/govern/model-access)

**“The Better Analytics Agent May Use Less RAG.”**  
An agent whose core tools are `resolve_metric`, `trace_lineage`, `check_freshness`, and `execute_metric` can reserve generative retrieval for explanation rather than asking embeddings to solve structured metadata problems.

### Useful reusable assets

The synthetic corpus above can become a **minimum regression suite** for any analytics copilot. A vendor or internal prototype should have to pass the same categories before a demo is taken seriously:

```text
right definition
right historical version
right grain
right model
right lineage
conflicting draft
superseded page
missing date
missing authoritative source
denied document
freshness expectation versus freshness observation
```

A second reusable asset is the **answer evidence envelope** from the worked example. It forces source/version fidelity into an API contract instead of leaving provenance as prose.

A third is the **evaluation split**:

```text
retrieval quality
≠ source resolution
≠ generation quality
≠ calculation correctness
≠ human decision quality
```

That split is consistent with the modular concerns raised by RAG evaluation and citation research. [4](https://arxiv.org/abs/2408.08067)

### Freshness triggers for this dossier

Revisit the conclusions when any of the following materially changes:

- dbt changes its model-version, Semantic Layer, manifest, access, grants, or freshness semantics; the relevant pages were actively updated during 2026. [1](https://docs.getdbt.com/reference/artifacts/manifest-json)
- OpenLineage advances beyond the inspected 1.53.0 specification or changes dataset/job/version/column-lineage semantics. [8](https://openlineage.io/docs/spec/object-model/)
- a production catalog gains native effective-time or source-authority resolution that makes a custom layer unnecessary;
- long-context models show materially better audited multi-document evidence use than the behavior represented by the 2024 “Lost in the Middle” study. [14](https://aclanthology.org/2024.tacl-1.9/)
- target-domain evaluation shows one retriever clearly dominates lexical/hybrid alternatives on real analytical documentation;
- new indirect-prompt-injection results materially change the retrieval-threat model or demonstrate reliable defenses. [15](https://arxiv.org/abs/2302.12173)
- the organization's own authority, ACL, lineage, or metric-versioning policy changes.

**Bottom-line decision:** start with an **authoritative metadata resolver plus ordinary catalog/search baseline**, not a vector database. Add hybrid retrieval where natural-language discovery empirically fails. Add an LLM only after the source resolver can return a permission-safe, time-valid, version-specific evidence bundle. For calculations, invoke the metric/SQL layer separately. RAG's highest-value role here is finding and explaining authoritative evidence—not deciding what the authoritative evidence is.

## Portable citation index

[1] [Manifest JSON file | dbt Developer Hub](https://docs.getdbt.com/reference/artifacts/manifest-json)

[2] [Dense Passage Retrieval for Open-Domain Question Answering - ACL Anthology](https://aclanthology.org/2020.emnlp-main.550/)

[3] [[2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401)

[4] [[2408.08067] RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation](https://arxiv.org/abs/2408.08067)

[5] [Semantic models | dbt Developer Hub](https://docs.getdbt.com/docs/build/semantic-models)

[6] [versions | dbt Developer Hub](https://docs.getdbt.com/reference/resource-properties/versions)

[7] [About documentation | dbt Developer Hub](https://docs.getdbt.com/docs/build/documentation)

[8] [Object Model | OpenLineage](https://openlineage.io/docs/spec/object-model/)

[9] [Column Level Lineage Dataset Facet | OpenLineage](https://openlineage.io/docs/spec/facets/dataset-facets/column_lineage_facet/)

[10] [freshness | dbt Developer Hub](https://docs.getdbt.com/reference/resource-configs/freshness)

[11] [Model access | dbt Developer Hub](https://docs.getdbt.com/docs/mesh/govern/model-access)

[12] [BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://arxiv.org/html/2104.08663v4)

[13] [Enabling Large Language Models to Generate Text with Citations - ACL Anthology](https://aclanthology.org/2023.emnlp-main.398/)

[14] [Lost in the Middle: How Language Models Use Long Contexts - ACL Anthology](https://aclanthology.org/2024.tacl-1.9/)

[15] [[2302.12173] Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://arxiv.org/abs/2302.12173)

[16] [Overcoming the Retrieval Barrier: Indirect Prompt Injection in the Wild for LLM Systems | USENIX](https://www.usenix.org/conference/usenixsecurity26/presentation/chang-hongyan)