Data Engineering
Data Lineage Services
See exactly where every number comes from, how it was transformed and what depends on it, with automated, column-level lineage across your pipelines, warehouses and reports.
Know Where Your Data Comes From and Where It Goes
When a dashboard shows an unexpected value, the first question is where it came from. When a source column changes, the next is what will break. Data lineage answers both by mapping the flow of data from source systems through ingestion, transformations and models to reports, applications and AI. Without it, teams trace dependencies by reading code and asking colleagues, which is slow and unreliable.
Hanoi Technology implements automated lineage that is harvested from the systems themselves, such as SQL logs, transformation code, orchestration metadata and BI tools, rather than drawn by hand. We connect technical lineage to business terms and owners, so engineers, analysts, auditors and data stewards can all use it.
The Challenge
Why Lineage Is Hard to Get Right
The obstacles that keep lineage incomplete, outdated or unused.
Manual Documentation Goes Stale
Lineage diagrams drawn in spreadsheets or slide decks are out of date the week after they are finished. Nobody trusts them for real decisions.
Gaps Across Tools
Lineage in one tool stops at its boundary. A path that passes through an ingestion tool, stored procedures, a lakehouse and a BI semantic layer shows up as disconnected fragments.
Table-Level Lineage Is Too Coarse
Knowing that one table feeds another does not tell you which report uses a sensitive column or which metric a change will affect.
Regulatory Pressure for Traceability
Regulations and standards such as BCBS 239 for banks, and privacy laws like GDPR, expect organizations to demonstrate how reported figures and personal data flow through their systems.
Business Value
What Automated Lineage Makes Possible
Confident Change Management
Before changing a column, table or pipeline, engineers see every downstream model, report and consumer affected. Breaking changes are caught before release.
Faster Root-Cause Analysis
When a number is wrong, lineage traces it upstream to the transformation or source where the problem began. Incidents are resolved in hours rather than days.
Audit-Ready Compliance
Lineage shows regulators and auditors how reported figures are produced and where personal data flows. Evidence is generated from metadata, not assembled by hand.
Greater Trust in Data
Business users see the origin, owner and quality status of the data behind each report, which builds confidence in self-service analytics.
Smarter Platform Cleanup
Lineage combined with usage reveals tables, pipelines and reports that nobody consumes. Retiring them cuts cost and complexity.
Responsible AI Foundations
Tracing which datasets feed each model and AI assistant supports explainability, bias reviews and compliance with emerging AI rules.
Lineage Is Only Useful If People Use It
The goal is not a beautiful graph but better decisions. We embed lineage into daily workflows: impact checks in pull requests, lineage links in incident alerts, origin details beside dashboards and evidence exports for audits. That is what turns lineage from a one-time documentation project into an everyday tool.
Our Services
Hanoi Technology’s End-to-End Data Lineage Services
Services to design, implement and operationalize data lineage.
Lineage Strategy and Tool Selection
We define the questions lineage must answer, the systems it must cover and the required level of detail, then compare catalog and lineage tools against your stack and budget. We also define how lineage will be kept current and who is responsible for its accuracy.
- Lineage use cases and scope
- Coverage requirements
- Tool evaluation and recommendation
Automated Lineage Harvesting
We configure connectors and parsers that extract lineage from databases, query logs, dbt projects, Spark jobs, orchestration tools and BI platforms, and keep it current automatically. Harvesting runs on a schedule or on deployment, so lineage reflects the platform as it is today.
- Connector configuration
- SQL and code parsing
- Scheduled lineage refresh
Column-Level and Cross-System Lineage
We stitch lineage across ingestion, transformation, warehouse and BI layers, down to columns, filling gaps with OpenLineage events or custom extractors where tools lack native support.
- End-to-end lineage graph
- Column-level mapping
- Custom extractors and OpenLineage integration
Business Lineage and Glossary
We link technical assets to business terms, critical data elements, owners and policies, so non-technical users can follow how a business metric is produced.
- Business glossary
- Critical data element mapping
- Ownership and stewardship model
Regulatory and Privacy Lineage
We trace personal and regulated data from collection to reporting, document controls along the path and produce evidence for audits, BCBS 239 reviews or privacy assessments.
- Sensitive data flow maps
- Control documentation
- Audit evidence exports
Lineage-Driven Operations
We integrate lineage into CI/CD impact analysis, data observability alerts and incident runbooks, so it is used every day by engineers and stewards.
- Pull request impact checks
- Lineage-aware alerting
- Operating procedures
Our Approach
How We Implement Data Lineage
Focused on high-value data first, then expanded across the platform.
- 01
Define Questions and Scope
We start with the decisions lineage must support, such as impact analysis, incident triage or regulatory reporting, and choose the critical reports and data elements to cover first.
Deliverables: Lineage use cases · Priority scope
- 02
Inventory Systems and Metadata
We list every system in the scope’s data path and check what lineage metadata each one exposes natively.
Deliverables: System inventory · Coverage gap analysis
- 03
Harvest and Stitch Lineage
We configure automated extraction, close gaps with custom extractors and validate the resulting graph against known flows with your engineers.
Deliverables: Validated lineage graph · Gap remediation
- 04
Enrich with Business Context
We attach owners, glossary terms, classifications and quality status, and make lineage accessible to analysts and stewards.
Deliverables: Enriched catalog · User guides
- 05
Embed and Expand
We integrate lineage into engineering and governance workflows, measure adoption and extend coverage to further domains.
Deliverables: Workflow integrations · Expansion roadmap
Use Cases
Data Lineage Use Cases
Practical ways our clients use lineage.
Impact Analysis Before Schema Changes
An engineer plans to rename a source column and sees the dashboards and models that depend on it. Owners are notified and changes are coordinated before deployment.
Regulatory Report Traceability
A bank traces each figure in a risk report back to its source systems and transformations. Auditors review evidence exported directly from the lineage tool.
Personal Data Mapping for Privacy
A privacy team identifies every table and report containing customer email addresses. Data subject requests and retention rules are applied completely.
Incident Root-Cause Analysis
A revenue metric drops unexpectedly and lineage points to a failed upstream load. The data team fixes the source and reruns only the affected models.
Cloud Migration Planning
Before migrating a warehouse, lineage shows which objects feed active reports. Unused tables are retired instead of migrated.
AI Model Data Provenance
An AI governance board reviews which datasets train and ground each model. Sensitive sources are identified and approved explicitly.
Technology
Lineage and Catalog Technologies
Tools and standards we implement and integrate.
- OpenLineage
- Marquez
- DataHub
- OpenMetadata
- Unity Catalog
- Microsoft Purview
- Collibra
- Alation
- Atlan
- Informatica CDGC
- Ataccama
- Manta
- dbt
- Apache Airflow
- Power BI
- Tableau
Platforms
Deliver Data Lineage on Your Preferred Platform
- DatabricksUnified lakehouse for data engineering, analytics and AI.Explore
- InformaticaEnterprise data integration, quality and governance.Explore
- AtaccamaData quality, governance and master data in one platform.Explore
- Microsoft AzureModernize data, analytics and AI within your Azure tenant.Explore
- SnowflakeElastic cloud data platform for analytics and data sharing.Explore
Why Hanoi Technology
Why Organizations Choose Hanoi Technology
Senior, Cross-Functional Teams
Strategy, data, AI, cloud and engineering specialists work as one team, so nothing is lost between hand-offs.
Proven Delivery
More than 50 projects delivered for clients in North America, Europe and Asia-Pacific.
Radical Transparency
A written update every week and a demo of working software every sprint, so you always know where things stand.
Cost-Effective Global Delivery
Experienced engineers from our Hanoi delivery hub, with working-hour overlap for every major time zone.
Success Story · Fintech
All-in-one digital banking app for a traditional bank
A traditional financial institution wanted to compete with neobanks for mobile-first customers. We replaced a set of separate banking, investing and payment apps with one product that people use every day.
Read the Full StoryMore Success Stories
Real estate
90%pricing accuracyTransparent AI property valuation for a multi-city marketplace
A growing real estate marketplace wanted buyers to trust its prices. We built a valuation engine and property pages that compare the estimated value with the asking price.
Resources
Featured Resources
Industry trends and practical guidance to help you get more value from data, AI and technology.
FSMA 204 After the Delay: Using Shipment Monitoring to Get Traceability-Ready by 2028
The FDA Food Traceability Rule deadline moved to July 2028, but the data work is large. Here is how to use shipment monitoring and event data to get ready without wasting the extra time.
CSRD After the Omnibus: What the Revised ESRS Mean for Reporting From 2027
The Omnibus and the simplified ESRS reshaped EU sustainability reporting in 2026. Here is who is in scope, what changed in the standards and how to adapt your data and process.
OpenTelemetry in 2026: Building a Vendor-Neutral Observability Strategy
OpenTelemetry has become the default way to collect telemetry. Here is what changed in 2026 and a practical roadmap for a vendor-neutral observability stack.
Make Every Number Traceable
Tell us which reports, regulations or platforms matter most to you. We will reply within 24 hours and set up a free consultation to scope a lineage implementation that delivers value quickly.
Data Lineage FAQ
What is the difference between table-level and column-level lineage?
Table-level lineage shows which tables feed which other tables and reports. Column-level lineage goes further and shows how each individual column is derived, including the transformations applied. Column level is what you need for precise impact analysis, tracing sensitive fields and regulatory traceability, because a single table often feeds many unrelated reports through different columns.
Can lineage be fully automated?
Most of it can. Modern tools extract lineage from SQL query logs, transformation code such as dbt models, Spark jobs, orchestrators and BI tools. Gaps typically remain for stored procedures with dynamic SQL, custom scripts, spreadsheets and some legacy tools. We close those with code parsing, OpenLineage instrumentation or, as a last resort, curated manual entries that are clearly marked and reviewed.
Which lineage tool should we choose?
It depends on your stack and purpose. Platform-native lineage, such as Unity Catalog or Microsoft Purview, is a good start if most data lives in one ecosystem. Enterprise catalogs like Collibra, Alation, Atlan or Informatica suit broad governance programs. Open-source tools like DataHub, OpenMetadata and OpenLineage offer flexibility. We evaluate options against your systems, required detail and budget.
How does lineage help with BCBS 239 or GDPR?
BCBS 239 expects banks to show that risk data is accurate, complete and traceable from source to report, and GDPR requires knowing where personal data is stored and processed. Lineage provides that map automatically and keeps it current. Combined with classification and controls documentation, it turns audit preparation from a manual exercise into reports generated from metadata.
How long does a lineage implementation take?
Initial lineage for a well-defined scope, such as a set of critical reports on one platform, can often be in place within several weeks. Cross-system, column-level coverage across a large estate is delivered incrementally. We start where lineage answers the most valuable questions and expand in phases, with weekly written updates on coverage and adoption.
Who maintains lineage once it is in place?
Because lineage is harvested automatically, it largely maintains itself as pipelines change. Ongoing work involves onboarding new systems, reviewing gaps, maintaining business context such as owners and glossary terms, and monitoring that connectors run. We define clear responsibilities for platform and stewardship teams, and we can support operations as a managed service.
Does lineage replace a data catalog?
No, they complement each other. A catalog helps people find and understand data assets through search, descriptions, owners and classifications. Lineage shows how those assets are connected and how data flows between them. Most modern catalog tools include lineage, and the combination is most useful: a user who finds a table in the catalog can immediately see where its data comes from, what it feeds and whether its upstream sources are healthy.