Data Lineage Explained: Tracking Data Flow Across Your Organization

Scenario context

📊 Data lineage is the complete genealogy of your data — where it comes from, how it's transformed, and where it ends up. It's critical for debugging pipelines, proving compliance, and understanding data dependencies. Implement it using OpenLineage, Apache Atlas, or custom tracking to prevent silent data failures and meet audit requirements.

Think of data lineage like supply chain tracking for goods. When you order…

This walkthrough focuses on What is Data Lineage? The Data Supply Chain.

Before making a design choice, clarify:

• Think of data lineage like supply chain tracking for goods.
• When you order a t-shirt online, you can track it from factory → warehouse → truck → doorstep.
• Data lineage does the same: it tracks data from source → transformation → storage → consumption.

Concepts in play: data-warehousing-lakehouse, graph-traversal-algorithms, spark-data-engineering, caching

Stage 1 — The Silent Crisis: When Data Disappears Into the Black Box

• Three months into the fiscal year, your finance team discovers the revenue report shows $2M more revenue than it actually did.
• "Where did this number come from?" No one knows.
• The pipeline ran successfully.
• The warehouse accepted the data.

Observable outcome: The warehouse accepted the data.

Stage 2 — How Data Lineage Works: The Tracking Mechanism

• There are three primary approaches to capturing data lineage: Approach 1: Query/Log Parsing (Passive) Parse database logs, Spark job logs, and SQL queries to infer lineage after the fact.
• When a Spark job runs, log the SQL it executes 2.
• Parse the SQL to extract table and column references 3.
• Build a graph: source table.column → target table.column 4.

Observable outcome: Build a graph: source table.column → target table.column 4.

Stage 3 — Visualizing the Data Pipeline: From Source to Report

• This diagram shows how data flows through a medallion architecture (bronze → silver → gold), and how lineage tracking instruments each layer to build the complete dependency graph.
• How to read this diagram: - The left side shows the traditional data pipeline flow (ingestion → transformation → consumption) - The lineage collector (shown in gray) sits alongside the pipeline and tracks every step - T…

graph TD
    A["📦 Source Systems"] --> B["🔌 Ingestion Layer"]
    B --> C["⚙️ Bronze Layer"]
    C --> D["🔄 Transformation Layer"]
    D --> E["🏛️ Silver Layer"]
    E --> F["📊 Analytics Layer"]
    F --> G["📈 BI / Reports / Dashboards"]

    H["🔍 Lineage Collector<br/>(OpenLineage)"] -.->|tracks all flows| B
    H -.->|tracks all flows| D
    H -.->|tracks all flows| E
    H -.->|tracks all flows| G

    style A fill:#e8f4f8
    style G fill:#fff4e6
    style H fill:#f0f0f0

Observable outcome: How to read this diagram: - The left side shows the traditional data pipeline flow (ingestion → transformation → consumption) - The lineage collector (shown in gray) sits alongside the pipeline and tracks every step - T…

Stage 4 — OpenLineage: How Industry Standard Lineage Works in Practice

• OpenLineage is an open-source standard (sponsored by Databricks, Collibra, Google) for capturing and sharing lineage metadata.
• It's the most practical way to implement lineage across modern data stacks.
• What OpenLineage Does OpenLineage defines a standard event format that tools can emit to describe: - Job execution (what ran, when, with what parameters) - Data movement (which tables were inputs, outputs, and transform…
• Query "What tables feed into revenue by user?" and you get the entire lineage chain.

from openlineage.client.run import RunEvent, RunState
from openlineage.client.client import OpenLineageClient
import datetime

# Initialize OpenLineage client (connects to Airflow, Marquez, or custom backend)
client = OpenLineageClient(url="http://localhost:5000")

# Define a data transformation job
job_name = "user_deduplication"
run_id = "run-123"

# Create a RunEvent describing what your job does
run_event = RunEvent(
    eventTime=datetime.datetime.now().isoformat(),
    run={"runId": run_id},
    job={"namespace": "data-pipeline", "name": job_name},
    eventType=RunState.START,
    inputs=[
        {
            "namespace": "postgres",
            "name": "public.raw_users",
            "facets": {
                "schema": {
                    "fields": [
                        {"name": "user_id", "type": "int"},
                        {"name": "email", "type": "string"},
                    ]
                }
            },
        }
    ],
    outputs=[
        {
            "namespace": "postgres",
            "name": "public.users_deduplicated",
            "facets": {
                "schema": {
                    "fields": [
                        {"name": "user_id", "type": "int"},
                        {"name": "email", "type": "string"},
                    ]
                }
            },
        }
    ],
    producer="https://github.com/mycompany/data-pipelines",
)

# Emit the event to the lineage collector
client.emit(run_event)

# Now your data pipeline is tracked and visible in:
# - Marquez (open-source lineage UI)
# - Databricks (Unity Catalog)
# - Collibra (enterprise governance platform)

raw_users (input) 
  ↓
[user_deduplication job]
  ↓
users_deduplicated (output)

Observable outcome: Query "What tables feed into revenue by user?" and you get the entire lineage chain.

Stage 5 — Practical Implementation Patterns

• Pattern 1: Lineage in Airflow DAGs Most Airflow integrations automatically emit lineage when you use SQL operators: When this DAG runs, OpenLineage automatically: 1.
• Extracts raw users , raw orders as inputs 3.
• Extracts analytics.user metrics as output 4.
• Emits a RunEvent with full lineage 5.

from airflow import DAG
from airflow.providers.apache.spark.operators.spark_sql import SparkSqlOperator

with DAG("daily_user_metrics", start_date="2026-01-01") as dag:
    transform = SparkSqlOperator(
        task_id="create_user_metrics",
        sql="""
            SELECT 
                u.user_id,
                COUNT(*) as purchase_count,
                SUM(o.amount) as total_spent
            FROM raw_users u
            LEFT JOIN raw_orders o ON u.user_id = o.user_id
            GROUP BY u.user_id
        """,
        output_table="analytics.user_metrics"  # OpenLineage auto-extracts lineage
    )

import pandas as pd
from openlineage.client import run
from datetime import datetime

def load_and_deduplicate():
    """Load raw users, deduplicate by email, save results."""

    # Emit START event
    with run.RunEventAsContext() as ctx:
        # Register input
        ctx.add_input_dataset(
            namespace="postgres",
            name="public.raw_users",
            schema=[("user_id", "int"), ("email", "string"), ("name", "string")]
        )

        # Load data
        users = pd.read_sql("SELECT * FROM raw_users", conn)

        # Transform (deduplicate by email, keep first occurrence)
        deduped = users.drop_duplicates(subset=['email'], keep='first')

        # Save results
        deduped.to_sql("users_deduplicated", conn, if_exists='replace')

        # Register output
        ctx.add_output_dataset(
            namespace="postgres",
            name="public.users_deduplicated",
            schema=[("user_id", "int"), ("email", "string"), ("name", "string")]
        )

# When this function runs, lineage is automatically tracked
load_and_deduplicate()

Observable outcome: Emits a RunEvent with full lineage 5.

Stage 6 — Real-World Example: E-Commerce Revenue Pipeline with Lineage

• Imagine you're at an e-commerce company.
• Revenue is calculated like this: At step 6, someone notices revenue is 10% lower than expected.
• Query: "Show me all transformations that fed into revenue by customer" 2.
• See step 5 (clean orders) is the immediate source 3.

1. Orders source system (external)
   ↓
2. Ingest to Kafka
   ↓
3. Stream to Bronze layer (raw_orders table)
   ↓
4. Spark job: Clean/validate orders
   ↓
5. Silver layer (clean_orders table)
   ↓
6. SQL: Join with customer_dim, calculate revenue
   ↓
7. Gold layer (revenue_by_customer table)
   ↓
8. BI Dashboard (revenue by region, time period, customer tier)

Observable outcome: See step 5 (clean orders) is the immediate source 3.

End-to-end outcome

After working through the stages, you should be able to explain:

• Don't implement all of technical + business + column-level lineage day one.
• Begin with table-level lineage (which tables feed which tables).

Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

Failure and degraded modes

Lineage Trade-offs: Accuracy vs Overhead

• Detection: Lineage Trade-offs: Accuracy vs Overhead
• User or operator impact: Lineage Trade-offs: Accuracy vs Overhead