Big Data 101: The 5 Vs, Ecosystem, and Why Scale Breaks Everything

Scenario context

Traditional databases fail at big data scale for three concrete reasons — storage saturation, compute bottleneck, and write-lock contention. The 5 Vs (Volume, Velocity, Variety, Veracity, Value) frame what makes data "big." A layered ecosystem — Kafka for ingestion, S3 for storage, Spark for processing, Snowflake for serving — assigns one tool per failure mode.

Observable outcome: A single PostgreSQL server has one disk (or disk array).

Stage 2 — The Big Data Ecosystem Map: From Raw Events to Business Insights

• The big data ecosystem looks intimidating at first because there are dozens of tools.
• The key insight is that every tool belongs to one of seven functional layers, and each layer solves one failure mode.
• You do not need all of these on day one.
• A team just outgrowing PostgreSQL typically starts with three layers: Kafka (ingestion), S3 (storage), and Redshift or BigQuery (serving).

graph TD
    A[Event Sources Apps  Sensors  DBs] --> B[Ingestion Layer Kafka  Kinesis  Fluentd]
    B --> C[Storage Layer HDFS  S3  GCS  Azure Blob]
    C --> D1[Batch Processing Apache Spark  Hadoop MapReduce]
    C --> D2[Stream Processing Apache Flink  Kafka Streams]
    D1 --> E[Table Formats Delta Lake  Apache Iceberg  Apache Hudi]
    D2 --> E
    E --> F[Serving Layer Redshift  BigQuery  Snowflake  ClickHouse]
    C --> G[Orchestration Airflow  Prefect  Dagster]
    G --> D1
    C --> H[Catalog & Governance Apache Atlas  AWS Glue]
    F --> I[BI & ML Tableau  dbt  ML Models]

Observable outcome: A team just outgrowing PostgreSQL typically starts with three layers: Kafka (ingestion), S3 (storage), and Redshift or BigQuery (serving).

Stage 3 — Real-World Applications: How Batch Reports and Millisecond Alerts Demand Different Stacks

• Understanding the ecosystem becomes concrete when you look at the two fundamentally different processing models every production data platform needs: batch and streaming .
• Batch processing is the practice of accumulating data over a period (an hour, a day) and processing it in a single large job.
• This is how nightly revenue reports work: collect all orders from the past 24 hours, join with the product catalog, aggregate by region, and write the result to a dashboard.
• The data is always somewhat stale — a "T+1 report" delivered the morning after the events occurred.

Observable outcome: The data is always somewhat stale — a "T+1 report" delivered the morning after the events occurred.

Stage 4 — Deep Dive: What a Distributed Job Actually Does Differently

• You do not need to understand cluster internals to work with big data tools, but a brief look at why distributed processing works helps the ecosystem map make sense.
• A traditional SQL query on PostgreSQL runs on one machine: one CPU reads data from one disk, applies transformations in RAM, and returns results.
• The throughput ceiling is whatever one machine can do.
• A distributed Spark job breaks the dataset into partitions — chunks of roughly equal size, typically 128 MB each.

Observable outcome: A distributed Spark job breaks the dataset into partitions — chunks of roughly equal size, typically 128 MB each.

Stage 5 — From MySQL to a Big Data Stack: An E-Commerce Case Study

• Consider an e-commerce company that started with a single MySQL database.
• Here is the journey from hitting the wall to a functioning big data stack.
• Phase 1 — MySQL hits the wall.
• The product catalog, orders, and user events all live in one MySQL instance.

graph TD
    A[User App Page Views  Clicks  Orders] --> B[Apache Kafka Event Stream 50k events/sec]
    C[MySQL Orders  Inventory] --> B
    B --> D[Amazon S3 Raw Data Lake  Parquet + Partitioned]
    D --> E[Apache Spark Nightly ETL  Join  Enrich  Aggregate]
    E --> F[S3 Gold Layer Cleaned Enriched Tables]
    F --> G[Amazon Redshift Columnar Warehouse  BI SQL]
    G --> H[BI Dashboard Tableau  Looker  Superset]
    B --> I[Kafka Streams Real-time Fraud Alerts]

# pandas approach: works at small scale, fails with MemoryError on large files
import pandas as pd

# This crashes with MemoryError on a laptop if the CSV is >16 GB
df = pd.read_csv("s3://my-bucket/events/2024-01-01.csv")
event_counts = df.groupby("event_type")["user_id"].count()
print(event_counts)

Observable outcome: The product catalog, orders, and user events all live in one MySQL instance.

Stage 6 — Apache Hadoop: The Pioneer That Shaped Modern Big Data

• No survey of big data is complete without acknowledging Apache Hadoop — the framework that proved distributed processing at commodity scale was possible.
• in 2006, Hadoop introduced two foundational ideas that every modern big data tool builds on: HDFS (Hadoop Distributed File System) splits large files into 128 MB blocks and distributes copies of those blocks across comm…
• If one server fails, the blocks are re-replicated from another copy.
• This is the concept behind Amazon S3, Google Cloud Storage, and every modern data lake.

Observable outcome: This is the concept behind Amazon S3, Google Cloud Storage, and every modern data lake.

End-to-end outcome

Imagine a startup that built its analytics on PostgreSQL.

Alternatives considered

The 5 Vs: A Framework for Naming Big Data Problems

The "5 Vs" is the most widely used framework for describing what makes data "big." Each V identifies a dimension of the problem, and each dimension maps to a specific class of tools.

Decision Guide: When Big Data Architecture Actually Makes Sense

Resist the urge to adopt the full big data stack before you need it.

Failure and degraded modes

Trade-offs and Failure Modes in Big Data Projects

• Detection: The big data ecosystem solves real scaling problems, but it introduces its own class of failure modes that beginners frequently underestimate.
• User or operator impact: The big data ecosystem solves real scaling problems, but it introduces its own class of failure modes that beginners frequently underestimate.