For most of its history, data lineage answered a familiar set of questions. Where did this number in the report come from? Which pipelines feed this table? What breaks if I change this column? The lineage graph started in source systems and ended in dashboards, extracts, and machine learning features.
Generative AI has stretched that graph in new directions. Documents are split into chunks, chunks become embeddings, embeddings land in vector indexes, retrieval pipelines assemble context, foundation models produce answers, and AI agents take actions in business systems. When a customer-facing assistant gives a wrong answer, or an agent updates a record it should not have touched, teams need to trace the problem back through all of those steps. This article explains what is changing, what end-to-end lineage for generative AI should capture, and how to build it with tools available in late 2026.
Why Classic Lineage Falls Short for GenAI
Traditional lineage tools were designed around SQL and batch pipelines. They parse queries or observe job runs to connect input tables and columns to outputs. That remains essential, but generative AI introduces several kinds of transformation that classic lineage either misses or represents poorly:
- Unstructured inputs. PDFs, wiki pages, tickets, emails, and transcripts often bypass the warehouse entirely and flow straight into ingestion services for retrieval.
- Lossy transformations. Chunking and embedding transform text in ways that are not reversible by inspecting the output; the link to the source must be recorded explicitly as metadata.
- Runtime composition. A retrieval-augmented answer is assembled at request time from whatever chunks the retriever selects, so the relevant lineage is per request, not only per pipeline.
- Model indirection. Applications call gateways or endpoints that may route to different foundation models, including fallbacks, which changes behavior without any change to application code.
- Actions, not just outputs. Agents call tools and APIs that write to systems of record, extending lineage from analytics into operations.
The result is a gap. A team may have excellent column-level lineage for its lakehouse and still be unable to say which version of a policy document produced a chatbot answer, or which model served a request last Tuesday.
What Is Driving the Change in 2026
Risk Frameworks Ask for Provenance
The NIST AI 600-1 Generative AI Profile, published in July 2024 as a companion to the AI Risk Management Framework, lists twelve risks specific to or amplified by generative AI, including confabulation, information integrity, and value chain and component integration. Among its suggested governance actions is recording data provenance in AI inventory entries, along with the underlying foundation models and versions. The value chain risk explicitly concerns third-party datasets and pre-trained models that are hard to trace. Lineage is the practical mechanism for meeting these expectations.
Platforms Are Extending Lineage to AI Assets
Major data platforms have been widening the scope of their lineage graphs. Databricks documents that Unity Catalog captures lineage automatically for queries run on the platform, down to the column level, and that the lineage graph includes ML model versions, model services, and external model providers alongside tables, notebooks, jobs, and dashboards. Its AI gateway service lineage links model services to the foundation models they serve, marked as primary or fallback, and to the inference tables that log request payloads. Databricks is candid about the limit: because this lineage is captured from service definitions, it does not show which agents call a service at runtime or what data they access through it.
That limitation captures the state of the market well. Platform lineage increasingly covers the static structure of AI systems. Per-request lineage, which shows what a specific answer or action depended on, still requires deliberate engineering.
Open Standards Are Getting Richer
The OpenLineage specification has evolved to describe not just which columns feed an output, but how. Its column lineage facet distinguishes direct transformations, such as identity copies, computations, and aggregations, from indirect influences, such as join keys, filters, grouping, sorting, windows, and conditional expressions. It also includes a masking flag for transformations that obfuscate inputs. Query engines continue to adopt the standard; the Presto project, for example, has described richer column-level lineage emitted through its OpenLineage event listener. These richer semantics matter for AI because they let teams distinguish data that shaped a feature or context window from data that merely filtered it.
The Four Layers of GenAI Lineage
A useful way to design lineage for generative AI is to think in four layers, each answering a different question:
- Data lineage: how structured and unstructured source data moves into curated tables, document stores, and feature tables. This is classic lineage, extended to cover document ingestion.
- Knowledge lineage: how documents become chunks, embeddings, and vector index entries, including chunking strategy, embedding model and version, and index build.
- Model lineage: which foundation models, fine-tuned models, prompts, and gateway routes serve each application, with versions and fallbacks.
- Request lineage: for each answer or action, which retrieved chunks, tools, model version, and prompt template were involved, linked back to the layers above.
The first three layers are mostly static and can be captured from pipeline runs and configuration. The fourth is dynamic and is usually captured through tracing in the application or agent framework, then joined to the static graph using shared identifiers.
For generative AI, lineage must answer a new question: not only where a table came from, but which documents, model version, prompt, and tools shaped a specific answer or action. That requires joining pipeline lineage with request-level traces.
Designing Identifiers That Make Lineage Joinable
The hardest part of GenAI lineage is not collecting data but connecting it. A few identifier conventions make the difference between a usable graph and a pile of logs:
- Stable document IDs and versions. Every source document gets an ID and a version or content hash at ingestion. Chunks inherit both.
- Chunk IDs carried into the index. Vector index entries store the chunk ID, document ID, document version, and embedding model version as metadata.
- Retrieval logging. Each request logs the chunk IDs retrieved and those actually included in the prompt.
- Model and prompt versions. Requests log the resolved model version, gateway route, and prompt template version, not just the endpoint name.
- Tool call records. Agent tool calls log the tool, parameters, target system, and the identity under which they ran.
- A shared trace ID. One identifier ties all of the above together for each request, and links to inference tables or observability stores.
With these conventions, a team can start from a problematic answer and walk back to the exact document versions, then use classic lineage to walk further back to the source systems and pipelines that produced them.
Use Cases That Justify the Investment
Answer Investigation
When a user reports a wrong or harmful answer, request lineage shows which chunks were retrieved and which document versions they came from. Often the root cause is an outdated document, a missing document, or a chunk that lost context during splitting. Without lineage, teams guess.
Impact Analysis Before Change
Retiring a foundation model, switching providers, or changing an embedding model affects every application that depends on it. Model and knowledge lineage show which applications, indexes, and downstream tables are affected. Databricks positions its gateway service lineage for exactly this scenario: understanding impact before deprecating a model or switching provider.
Data Deletion and Rights Requests
If a document must be removed for legal, contractual, or privacy reasons, knowledge lineage identifies every chunk and index entry derived from it. Request lineage can show whether it influenced past answers, which may matter for incident response.
Quality and Freshness Monitoring
Linking indexes to source document versions makes it possible to measure index staleness and alert when important sources have changed but embeddings have not been refreshed.
A Hypothetical Example: An Internal Policy Assistant
Consider a hypothetical professional services firm that deploys an internal assistant answering questions about travel, expense, and security policies. Policies live in a document management system and are updated several times a year. The assistant uses retrieval over a vector index and calls a foundation model through a gateway with a fallback model configured.
An employee reports that the assistant quoted an outdated per diem rate. Because the firm logs chunk IDs and document versions for each request, the support engineer can see within minutes that the answer drew on version three of the travel policy, while version four had been published two weeks earlier. Knowledge lineage shows that the ingestion job for the policy library had been failing silently since a permissions change. The fix involves repairing the job, rebuilding the affected index entries, and adding a freshness check that compares the latest document versions to the index. Model lineage confirms that the fallback model was not involved, ruling out one hypothesis quickly.
Without lineage, the same investigation might have taken days and ended with a vague conclusion that the model hallucinated.
Building It: Practical Steps
- Start from your existing lineage foundation. Make sure classic lineage covers the curated tables and document stores that feed AI applications, using platform-native capture and OpenLineage where tools support it.
- Treat document ingestion as a pipeline. Emit lineage events for document ingestion, chunking, embedding, and index builds, just as you would for SQL jobs.
- Register AI assets in the catalog. Models, endpoints, prompt templates, and vector indexes should be catalog objects with owners and versions.
- Instrument applications and agents. Use tracing in your application or agent framework to capture retrieval results, model routes, and tool calls with a shared trace ID.
- Join static and dynamic lineage. Store request traces where they can be joined to catalog lineage, such as inference tables or an observability store linked by IDs.
- Define retention and access. Request logs may contain sensitive prompts and outputs; apply classification, masking, and retention rules from the start.
- Operationalize. Build runbooks for answer investigation, model retirement, and document deletion that rely on lineage, and test them.
A Maturity Path for GenAI Lineage
Few organizations can implement all four layers at once. A staged path keeps the effort proportional to risk:
- Level 1, Inventory: every AI application is listed with its data sources, models, and owners, as the NIST profile’s inventory actions suggest. Lineage is documented manually.
- Level 2, Pipeline lineage: structured and document ingestion pipelines emit lineage automatically, and indexes record source document versions.
- Level 3, Model lineage: models, prompts, gateway routes, and fallbacks are catalog assets, and impact analysis covers model changes.
- Level 4, Request lineage: each answer and agent action is traceable to retrieved content, model version, and tool calls, joined to the catalog graph.
- Level 5, Lineage-driven controls: lineage drives automated actions, such as blocking retrieval from deprecated documents, triggering re-embedding when sources change, or alerting owners when a model dependency is retired.
Customer-facing assistants and agents that write to business systems usually justify level 4 quickly. Internal, low-risk tools may sit comfortably at level 2 or 3 for some time.
Pitfalls to Avoid
- Logging everything, linking nothing. Large volumes of traces without shared identifiers are expensive and nearly useless.
- Ignoring unstructured pipelines. Document ingestion often lives in application code outside the data platform and is invisible to platform lineage.
- Tracking endpoints instead of models. An endpoint name hides which model version actually responded, especially with routing and fallbacks.
- Forgetting privacy. Prompts and retrieved content can include personal or confidential data; lineage stores need the same controls as the data itself.
- Treating lineage as a compliance artifact only. The biggest payoff is faster incident resolution and safer change, which keeps the system maintained.
Outlook: Lineage as the Memory of AI Systems
Over the next few years, expect platform vendors to close part of the runtime gap by linking agent traces to catalog lineage natively, and expect open standards to add richer representations for unstructured data, embeddings, and model calls. As agents take on more operational work, lineage will increasingly serve as the system of record for why something happened. Organizations that build joinable identifiers and request-level tracing now will be able to adopt those capabilities easily; those that rely on table-level lineage alone will find themselves reconstructing incidents by hand.
Conclusion
Generative AI has turned lineage from a data engineering convenience into a core control for AI reliability, risk management, and change. The extension is manageable: capture document and embedding pipelines as lineage, register models and indexes as governed assets, trace each request with shared identifiers, and join it all to the lineage you already have. Done well, it lets teams answer the question every AI incident eventually raises: why did the system do that?
How Hanoi Technology Can Help
Hanoi Technology designs end-to-end lineage for data and AI systems, from source pipelines to RAG indexes, model endpoints and agent traces, as projects or managed services.