Ask a data team where a number on an executive dashboard comes from, and the honest answer is often a shrug followed by an afternoon of reading SQL. Lineage, the map of how data flows from sources through transformations to reports, models and AI applications, has been on every governance roadmap for a decade. Yet most organizations still have fragmented lineage: one view inside the warehouse, another inside the orchestration tool, a third inside the BI platform, and nothing that connects them.
The reason is structural. Lineage is produced by many tools but consumed in one place, and for years every catalog vendor had to write and maintain its own integration with every engine. OpenLineage, an open specification for emitting lineage events at runtime, set out to break that many-to-many problem. In 2026 it has become the default way serious data platforms exchange lineage. This article explains how the standard works, why it gained traction, and how to adopt it without boiling the ocean.
The Problem OpenLineage Solves
Traditional lineage tools work by harvesting: they connect to a warehouse, parse query logs or stored SQL, and infer dependencies after the fact. Harvesting works well inside a single system but has three weaknesses in a modern stack.
- Tool boundaries break the graph. A Spark job that reads from object storage, a dbt model in the warehouse and a Python script that writes features to a vector store each live in different systems. Parsing one system’s logs cannot see the others.
- Inference is lossy. Parsers struggle with dynamic SQL, procedural code, notebooks and DataFrame APIs, which is exactly where much of the important logic lives.
- Integration cost multiplies. With N producers and M consumers, every pair needs a connector. Catalogs spent their engineering budgets keeping up with engine releases.
OpenLineage reverses the direction. Instead of consumers scraping producers, producers emit standardized events as jobs run, and any compatible backend can consume them. The project was initiated by the team behind Marquez, which is now an LF AI & Data project and the reference implementation of the OpenLineage API, with contributors from projects including Airflow, Spark, dbt, Great Expectations, Iceberg and DataHub, according to Data Council’s overview of the project.
How the Standard Works
The core model is deliberately small. According to the specification, it has three entities and one extension mechanism.
- Job: a process definition that consumes and produces datasets, such as an Airflow task, a Spark application or a dbt model. Jobs are named uniquely within a namespace.
- Run: a single execution of a job, identified by a UUID, with a start and an end.
- Dataset: an abstract representation of data, such as a table, a file path or a topic, named uniquely within the namespace of its data source and derived from its physical location.
- Facets: atomic, versioned pieces of metadata attached to jobs, runs or datasets. Facets carry everything else, from schemas to SQL to data quality results.
Producers emit run events describing a run’s lifecycle. The specification requires at least a START event and a terminal COMPLETE, FAIL or ABORT event; intermediate events are optional, and metadata is additive across the run. Each event lists the job, the run and the input and output datasets observed, with whatever facets the producer can supply. Events are defined in JSON Schema, with an OpenAPI definition for HTTP transport, so a backend can be as simple as a web service that accepts POST requests.
Facets: where the value lives
The facet system is what makes OpenLineage useful beyond a dependency graph. Standard facets documented by the project include:
- Run facets such as parent (linking a task run to the DAG run that spawned it), nominalTime and errorMessage.
- Job facets such as sql, sourceCode, sourceCodeLocation (with a git SHA) and ownership.
- Dataset facets such as schema, dataSource, version (for example an Iceberg snapshot ID), lifecycleStateChange, ownership and columnLineage.
- Input and output facets such as dataQualityMetrics, dataQualityAssertions and outputStatistics with row counts and bytes written.
Custom facets are allowed with a project-specific prefix and a versioned schema URL, which lets vendors and internal teams extend the model without forking it.
Column-level lineage, done precisely
Table-level lineage tells you that a dashboard depends on an orders table. Column-level lineage tells you that the revenue figure is an aggregation of order_amount, filtered by status and joined on customer_id. The OpenLineage column lineage facet captures this with two transformation types: DIRECT, where the output value is derived from the input (with subtypes IDENTITY, TRANSFORMATION and AGGREGATION), and INDIRECT, where the input influences the output without being part of its value (JOIN, GROUP_BY, FILTER, SORT, WINDOW and CONDITIONAL). A masking flag marks transformations, such as hashing, that obfuscate the input. That level of detail matters for impact analysis, privacy reviews and explaining model inputs.
OpenLineage turns lineage from something you reverse-engineer after an incident into something your pipelines report as they run, in a format any tool can read.
Why Adoption Accelerated
Three developments pushed OpenLineage from an interesting project to a practical default.
- Native producers in core tools. Apache Airflow ships an official OpenLineage provider, and integrations exist for Spark, dbt and Flink, so many teams get lineage by configuration rather than code.
- Cloud platforms joined in. In June 2025, AWS announced that Amazon SageMaker contributed a custom transport to the OpenLineage community so that lineage events from OpenLineage plugins can be sent into SageMaker Unified Studio domains, alongside expanded automated lineage for AWS Glue and Amazon Redshift. When a hyperscaler’s catalog accepts the standard, adopting it stops being a bet on a niche project.
- AI raised the stakes. Teams building retrieval-augmented generation, feature pipelines and agents need to show what data fed a model or an answer. Regulators are moving the same way: the EU AI Act’s data governance obligations for high-risk systems, now scheduled under the 2026 Digital Omnibus amendment to apply from December 2027 for stand-alone systems, call for documented data origin and preparation steps. Runtime lineage is the most scalable way to produce that evidence.
Where OpenLineage Fits in Your Architecture
A useful mental model separates four roles.
- Producers: orchestrators, processing engines, transformation tools and custom code that emit events.
- Transport: HTTP, Kafka or cloud-specific transports that move events reliably. Kafka is attractive at scale because it decouples producers from backend availability.
- Backends: Marquez, commercial catalogs or a cloud platform’s catalog that store events and build the graph.
- Consumers of the graph: people doing impact analysis, data observability tools, governance workflows and, increasingly, AI assistants that answer “where does this number come from?”
OpenLineage does not replace harvesting entirely. Warehouse-native lineage, such as what Unity Catalog or a warehouse’s own query history provides, remains valuable for SQL that runs entirely inside one engine. The practical pattern is hybrid: native lineage inside platforms, OpenLineage for everything that crosses boundaries, and a catalog that stitches both together using consistent dataset naming.
Operating the lineage event pipeline
Lineage events are telemetry, and they deserve the same operational care as logs and metrics. Producers should never fail a data job because the lineage backend is unavailable, so configure transports to be asynchronous and non-blocking, and buffer events through a durable queue where volume is high. Monitor event volume per producer: a sudden drop usually means an integration broke after an upgrade, not that pipelines stopped running. Version your custom facets and validate events against the JSON Schema in CI for any internal emitter library. Finally, decide on retention up front. Run-level history is valuable for audits and incident forensics, but most organizations do not need every run event forever; keeping detailed runs for a defined period and the aggregated graph indefinitely is a common compromise.
An Adoption Playbook
- Agree on naming first. OpenLineage’s value depends on two producers naming the same table identically. Adopt the specification’s dataset naming conventions and document how your platforms map to them before turning on integrations.
- Start with the orchestrator. Enabling the OpenLineage integration in Airflow or your scheduler gives you job and run structure across the estate quickly, and parent facets link tasks to pipelines.
- Add engine-level producers. Turn on the Spark and dbt integrations to capture dataset and column lineage from the transformations themselves. For Spark, follow the project’s guidance to enable the compact dataset-level representation of indirect dependencies, which avoids bloated column lineage.
- Instrument custom code. Use the client libraries to emit events from Python services, feature pipelines and ingestion scripts. Wrap this in a shared internal library so teams do not hand-craft events.
- Choose a backend deliberately. Marquez is a good starting point for exploration; production estates usually route events to the catalog the organization already uses, provided it consumes OpenLineage natively.
- Attach quality and ownership. Emit data quality assertion and ownership facets so the lineage graph doubles as an incident map: who owns each node and whether it passed its checks.
- Operationalize it. Put lineage into change review (show downstream impact on every pull request), incident response and access reviews. Lineage that nobody looks at decays.
What good looks like after 90 days
- Every production pipeline in the orchestrator emits START and terminal events, with parent links from tasks to pipelines.
- Column-level lineage is available for the top tier of business-critical datasets and the dashboards and models built on them.
- Each node in the graph has an owner, and data quality results appear alongside lineage.
- At least one workflow, typically pull request impact analysis or incident triage, uses the lineage graph routinely.
Worked Example: Tracing a Broken KPI
Imagine a company whose weekly gross margin KPI drops by an implausible amount on Monday morning. Without cross-tool lineage, the analytics team checks the dashboard, the semantic model and the warehouse tables in turn, then escalates to data engineering, who check the ingestion jobs. Half a day passes.
With OpenLineage events flowing from the orchestrator, Spark and dbt into the catalog, the investigation looks different. The analyst opens the KPI’s column lineage and sees that gross_margin is an aggregation of net_revenue and cost_of_goods, with cost_of_goods derived from a Spark job that joins product costs from an ERP extract. The run facets show that the Spark job completed on schedule, but the output statistics facet shows that it wrote about a third of its usual row count, and the data quality assertion facet records a failed completeness check that nobody was alerted to. The parent facet links the run to the ingestion task, whose error message facet shows a partial extract after a source timeout.
The fix still requires engineering work, but diagnosis takes minutes rather than hours, the right owner is notified immediately, and the team can list every other dashboard and model that consumed the incomplete table. That last step, downstream impact, is often where lineage pays for itself.
The follow-up is just as instructive. The team adds an alert on the completeness assertion, so the next partial extract stops downstream jobs instead of silently flowing into the KPI. Because the lineage graph also shows that a pricing model consumed the same table, the data science team retrains it on corrected data and records the incident against the affected model version. None of those steps would be obvious without a graph that crosses the ingestion, processing, transformation and BI layers.
Pitfalls to Avoid
- Inconsistent dataset names that split one table into several nodes in the graph.
- Lineage without ownership, which shows where a problem is but not who should fix it.
- Blind spots in custom code, where the most business-critical logic often lives.
- Treating lineage as a one-off project rather than a property of every new pipeline, enforced through templates and CI.
- Over-collecting verbose facets everywhere, which inflates storage and slows the catalog without adding insight.
Outlook
Expect three shifts over the next two years. Lineage will increasingly extend past analytics into AI, linking training datasets, feature pipelines, vector indexes and model versions in one graph. Static lineage, describing designed dependencies before jobs run, will complement runtime events for change planning. And lineage will become an input to automation: data observability tools prioritizing alerts by downstream impact, access reviews driven by where sensitive columns flow, and AI assistants explaining metrics by walking the graph. All of these depend on lineage being open and machine-readable, which is why the standard matters more than any single tool.
Conclusion
OpenLineage succeeded because it solved the right problem: not drawing prettier graphs, but giving every tool a common way to report what it read, what it wrote and how. With native producers in mainstream tools, cloud catalogs accepting the standard and AI governance demanding traceable data, open runtime lineage is now table stakes for a credible data platform. Start with naming, turn on the orchestrator and engine integrations, instrument custom code, and wire lineage into the everyday workflows of change review and incident response. The payoff is faster debugging, safer change and evidence you can show an auditor.
How Hanoi Technology Can Help
Hanoi Technology implements OpenLineage-based, column-level lineage across orchestration, Spark, dbt and custom code, and connects it to your catalog and governance processes.
Further Reading
- OpenLineage specification – OpenLineage / GitHub
- Column Level Lineage Dataset Facet – OpenLineage Docs
- Understanding and Using Facets – OpenLineage Docs
- Amazon SageMaker contributes a custom transport to the OpenLineage community – AWS
- OpenLineage – Data Council
- EU Digital Omnibus on AI enters into force – National Law Review