Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

Safe Event Schema Evolution
with Protobuf & Avro.

A streaming analytics platform processes 50,000 events/sec across Kafka using Avro and Protobuf. A team deploys a producer update that renames a required Avro field (without aliases or defaults) and reuses a retired Protobuf tag number (field 4 changed from int64 to string). Downstream Flink and Python consumers immediately throw DeserializationException / WireFormatException, crash-looping and blocking all subsequent valid messages in the partitions. Design a schema governance, CI/CD compatibility gate, and poison-pill-resilient consumer architecture.
Start the journey5 stages
Loading diagram…
Safe Event Schema Evolution with Protobuf & Avro architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

A streaming analytics platform processes 50,000 events/sec across Kafka using Avro and Protobuf. A team deploys a producer update that renames a required Avro field (without `aliases` or defaults) and reuses a retired Protobuf tag number (`field 4` changed from `int64` to `string`). Downstream Flink and Python consumers immediately throw `DeserializationException` / `WireFormatException`, crash-looping and blocking all subsequent valid messages in the partitions. Design a schema governance, CI/CD compatibility gate, and poison-pill-resilient consumer architecture.

From patterns to decisions.

A streaming analytics platform processes 50,000 events/sec across Kafka using Avro and Protobuf. A team deploys a producer update that renames a required Avro field (without `aliases` or defaults) and reuses a retired Protobuf tag number (`field 4` changed from `int64` to `string`). Downstream Flink and Python consumers immediately throw `DeserializationException` / `WireFormatException`, crash-looping and blocking all subsequent valid messages in the partitions. Design a schema governance, CI/CD compatibility gate, and poison-pill-resilient consumer architecture.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

Protobuf encodes (tag << 3).

Protobuf encodes (tag << 3) | wire_type on the wire; changing int64 (wire type 0) to string (wire type 2) breaks binary framing, while Avro needs defaults to project missing fields.

ChoiceWhat it buysWhat it costs
Trade-off 1CI compatibility checks catch breaking changes before merge, adding <1s to build pipelines.

Explain Protobuf wire types

0 (varint) vs 2 (length-delimited).

Explain Avro schema resolution

reader schema requires a default if the writer schema omitted a field.

Step 3

Show why FULL_TRANSITIVE catches both BACKWARD and FORWARD breaks.

Alternative 1

Using TopicNameStrategy vs RecordNameStrategy when multiple event types share one topic.

Failure Mode 1

Using BACKWARD alone allows a producer to add a required field that crashes older consumers.

# CI check against Confluent Schema Registry compatibility API
import requests

def check_schema_compat(sr_url: str, subject: str, avsc_str: str) -> bool:
    resp = requests.post(
        f"{sr_url}/compatibility/subjects/{subject}/versions?verbose=true",
        json={"schema": avsc_str, "schemaType": "AVRO"},
        timeout=5,
    )
    resp.raise_for_status()
    data = resp.json()
    if not data.get("is_compatible"):
        raise RuntimeError(f"Breaking schema change: {data.get('messages')}")
    return True
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

Consume raw bytes so poll().

Consume raw bytes so poll() never throws; decode inside try/except, route failures to <topic>.dlq with headers, and commit offsets.

ChoiceWhat it buysWhat it costs
Trade-off 1Handling deserialization explicitly prevents partition stalls, at the cost of out-of-order DLQ replay.

Step 1

Parse the 5-byte header (magic == 0x00 + 4-byte schema_id) or delegate to AvroDeserializer/ProtobufDeserializer.

Step 2

On SerializationError or DecodeError, produce the raw msg.value() to <topic>.dlq with error headers.

Step 3

Commit the source offset only after dlq_producer.flush() succeeds.

Alternative 1

Spring Kafka ErrorHandlingDeserializer + DeadLetterPublishingRecoverer in Java.

Failure Mode 1

Committing the source offset before dlq_producer acknowledges the write drops the poison pill permanently.

import struct
from confluent_kafka.serialization import SerializationContext, MessageField, SerializationError

def process_message(msg, deserializer, dlq_producer, handler):
    raw = msg.value()
    try:
        if not raw or len(raw) < 5 or raw[0] != 0:
            raise SerializationError("Missing Confluent 5-byte wire header")
        schema_id = struct.unpack(">I", raw[1:5])[0]
        ctx = SerializationContext(msg.topic(), MessageField.VALUE)
        event = deserializer(raw, ctx)
        handler(event)
    except Exception as exc:
        dlq_producer.produce(
            topic=f"{msg.topic()}.dlq",
            key=msg.key(),
            value=raw,
            headers={
                "error": str(exc),
                "source_partition": str(msg.partition()),
                "source_offset": str(msg.offset()),
            },
        )
        dlq_producer.flush()
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

Use the expand-and-contract pattern.

Use the expand-and-contract pattern: add a new optional field, dual-write, migrate readers, then deprecate and reserve the old tag.

ChoiceWhat it buysWhat it costs
Trade-off 1Three-step rollout takes longer than an in-place edit, but guarantees zero downtime and zero decoder crashes.

Step 1

Add the new field (string user_id_v2 = 8;) with a default/optional marker.

Step 2

Producer dual-writes both old and new fields.

Step 3

Upgrade all consumers to prefer user_id_v2, then stop writing old field and mark tag 4 as reserved.

Alternative 1

Publishing to a brand-new versioned topic (orders.v2) when breaking changes affect >50% of fields.

Failure Mode 1

Removing reserved later allows a future developer to accidentally reuse tag 4.

# Protobuf expand-and-contract example
# Never reuse tag 4 after deprecating int64 user_id
# syntax = "proto3";
# message OrderEvent {
#   reserved 4;
#   reserved "user_id";
#   string user_id_v2 = 8;
# }
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

Implementation & Guardrails (4).

Switch to KafkaRecordDeserializationSchema with a Collector<Either<BadRecord, ValidEvent>> (or Flink Side Output) that routes bad bytes to a DLQ sink.

ChoiceWhat it buysWhat it costs
Trade-off 1Keeps Flink checkpoints healthy at 50k/sec without restarting TaskManagers.

Step 1

Catch Throwable inside KafkaRecordDeserializationSchema.deserialize(ConsumerRecord, Collector).

Step 2

Wrap bad bytes in a DeserializationResult tagged for a Flink OutputTag<RawRecord>("dlq").

Step 3

Send the side output to <topic>.dlq so checkpoints and watermarks continue advancing.

Alternative 1

Temporarily resetting consumer offset past the bad message (loses data if not copied first).

Failure Mode 1

Returning null silently from deserialize() drops data without any audit trail or alert.

// Flink-style safe deserialization pattern in Java
public void deserialize(ConsumerRecord<byte[], byte[]> record, Collector<DecodedOrError> out) {
    try {
        Event evt = avroDecoder.decode(record.value());
        out.collect(DecodedOrError.ok(evt));
    } catch (Exception ex) {
        out.collect(DecodedOrError.dlq(record.topic(), record.partition(), record.offset(), record.value(), ex.getMessage()));
    }
}
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

If auto.

If auto.register.schemas=true, a misconfigured producer can silently register an unreviewed schema or fail in production at runtime instead of CI.

ChoiceWhat it buysWhat it costs
Trade-off 1Requires schema registration in CI/CD before app rollout, preventing rogue schema mutations in prod.

Step 1

Register schemas explicitly during CI/CD deployment.

Step 2

Set auto.register.schemas=false and use.latest.version=true (or pinned schema ID) in production.

Alternative 1

Using RBAC on Schema Registry so production app service accounts have READ access only.

Failure Mode 1

Accidental auto.register.schemas=true combined with NONE compatibility lets any typo create a broken schema version.

producer_conf = {
    "bootstrap.servers": "kafka:9092",
    "schema.registry.url": "https://sr.internal",
    "auto.register.schemas": False,
}

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Enforce BACKWARD or FULL_TRANSITIVE compatibility in Schema Registry with CI/CD gating, embed 5-byte Confluent wire-format schema IDs, and isolate poison-pill SerializationException records to a DLQ.
  • 02Trade-off tip: Three-step rollout takes longer than an in-place edit, but guarantees zero downtime and zero decoder crashes.
  • 03Watch out for: Returning null silently from deserialize() drops data without any audit trail or alert.
Behavioral Progression · Safe Event Schema Evolution Protobuf Avro

Safe Event Schema Evolution Protobuf Avro — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Enforce BACKWARD or FULLTRANSITIVE compatibility in Schema Registry with CI/CD gating, embed 5-byte Confluent wire-format schema IDs, and isolate poison-pill SerializationException records to a DLQ.

Synthesis / One connected design

Putting Safe Event Schema Evolution with Protobuf & Avro Together

Enforce `BACKWARD` or `FULL_TRANSITIVE` compatibility in Schema Registry with CI/CD gating, embed 5-byte Confluent wire-format schema IDs, and isolate poison-pill `SerializationException` records to a DLQ.

A candidate architecture, not a universal answer

Putting Safe Event Schema Evolution with Protobuf & Avro Together

Step 1
Step 2
Step 3
  1. Parse the 5-byte header (magic == 0x00 + 4-byte schema_id) or delegate to AvroDeserializer/ProtobufDeserializer.
  2. On SerializationError or DecodeError, produce the raw msg.value() to <topic>.dlq with error headers.
  3. Commit the source offset only after dlq_producer.flush() succeeds.
Switch to `KafkaRecordDeserializationSchema` with a `Collector<Either<BadRecord, ValidEvent>>` (or Flink Side Output) that routes bad bytes to a DLQ sink.
Interview Readiness Check

Switch to KafkaRecordDeserializationSchema with a Collector<Either<BadRecord, ValidEvent>> (or Flink Side Output) that routes bad bytes to a DLQ sink.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Safe Event Schema Evolution with Protobuf & Avro

Enforce BACKWARD or FULL_TRANSITIVE compatibility in Schema Registry with CI/CD gating, embed 5-byte Confluent wire-format schema IDs, and isolate poison-pill SerializationException records to a DLQ.