A streaming analytics platform processes 50,000 events/sec across Kafka using Avro and Protobuf. A team deploys a producer update that renames a required Avro field (without `aliases` or defaults) and reuses a retired Protobuf tag number (`field 4` changed from `int64` to `string`). Downstream Flink and Python consumers immediately throw `DeserializationException` / `WireFormatException`, crash-looping and blocking all subsequent valid messages in the partitions. Design a schema governance, CI/CD compatibility gate, and poison-pill-resilient consumer architecture.
From patterns to decisions.
A streaming analytics platform processes 50,000 events/sec across Kafka using Avro and Protobuf. A team deploys a producer update that renames a required Avro field (without `aliases` or defaults) and reuses a retired Protobuf tag number (`field 4` changed from `int64` to `string`). Downstream Flink and Python consumers immediately throw `DeserializationException` / `WireFormatException`, crash-looping and blocking all subsequent valid messages in the partitions. Design a schema governance, CI/CD compatibility gate, and poison-pill-resilient consumer architecture.
Protobuf encodes (tag << 3).
Protobuf encodes (tag << 3) | wire_type on the wire; changing int64 (wire type 0) to string (wire type 2) breaks binary framing, while Avro needs defaults to project missing fields.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | CI compatibility checks catch breaking changes before merge, adding <1s to build pipelines. |
Explain Protobuf wire types
0 (varint) vs 2 (length-delimited).
Explain Avro schema resolution
reader schema requires a default if the writer schema omitted a field.
Step 3
Show why FULL_TRANSITIVE catches both BACKWARD and FORWARD breaks.
Alternative 1
Using TopicNameStrategy vs RecordNameStrategy when multiple event types share one topic.
Failure Mode 1
Using BACKWARD alone allows a producer to add a required field that crashes older consumers.
# CI check against Confluent Schema Registry compatibility API
import requests
def check_schema_compat(sr_url: str, subject: str, avsc_str: str) -> bool:
resp = requests.post(
f"{sr_url}/compatibility/subjects/{subject}/versions?verbose=true",
json={"schema": avsc_str, "schemaType": "AVRO"},
timeout=5,
)
resp.raise_for_status()
data = resp.json()
if not data.get("is_compatible"):
raise RuntimeError(f"Breaking schema change: {data.get('messages')}")
return True
Consume raw bytes so poll().
Consume raw bytes so poll() never throws; decode inside try/except, route failures to <topic>.dlq with headers, and commit offsets.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Handling deserialization explicitly prevents partition stalls, at the cost of out-of-order DLQ replay. |
Step 1
Parse the 5-byte header (magic == 0x00 + 4-byte schema_id) or delegate to AvroDeserializer/ProtobufDeserializer.
Step 2
On SerializationError or DecodeError, produce the raw msg.value() to <topic>.dlq with error headers.
Step 3
Commit the source offset only after dlq_producer.flush() succeeds.
Alternative 1
Spring Kafka ErrorHandlingDeserializer + DeadLetterPublishingRecoverer in Java.
Failure Mode 1
Committing the source offset before dlq_producer acknowledges the write drops the poison pill permanently.
import struct
from confluent_kafka.serialization import SerializationContext, MessageField, SerializationError
def process_message(msg, deserializer, dlq_producer, handler):
raw = msg.value()
try:
if not raw or len(raw) < 5 or raw[0] != 0:
raise SerializationError("Missing Confluent 5-byte wire header")
schema_id = struct.unpack(">I", raw[1:5])[0]
ctx = SerializationContext(msg.topic(), MessageField.VALUE)
event = deserializer(raw, ctx)
handler(event)
except Exception as exc:
dlq_producer.produce(
topic=f"{msg.topic()}.dlq",
key=msg.key(),
value=raw,
headers={
"error": str(exc),
"source_partition": str(msg.partition()),
"source_offset": str(msg.offset()),
},
)
dlq_producer.flush()
Use the expand-and-contract pattern.
Use the expand-and-contract pattern: add a new optional field, dual-write, migrate readers, then deprecate and reserve the old tag.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Three-step rollout takes longer than an in-place edit, but guarantees zero downtime and zero decoder crashes. |
Step 1
Add the new field (string user_id_v2 = 8;) with a default/optional marker.
Step 2
Producer dual-writes both old and new fields.
Step 3
Upgrade all consumers to prefer user_id_v2, then stop writing old field and mark tag 4 as reserved.
Alternative 1
Publishing to a brand-new versioned topic (orders.v2) when breaking changes affect >50% of fields.
Failure Mode 1
Removing reserved later allows a future developer to accidentally reuse tag 4.
# Protobuf expand-and-contract example
# Never reuse tag 4 after deprecating int64 user_id
# syntax = "proto3";
# message OrderEvent {
# reserved 4;
# reserved "user_id";
# string user_id_v2 = 8;
# }
Implementation & Guardrails (4).
Switch to KafkaRecordDeserializationSchema with a Collector<Either<BadRecord, ValidEvent>> (or Flink Side Output) that routes bad bytes to a DLQ sink.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Keeps Flink checkpoints healthy at 50k/sec without restarting TaskManagers. |
Step 1
Catch Throwable inside KafkaRecordDeserializationSchema.deserialize(ConsumerRecord, Collector).
Step 2
Wrap bad bytes in a DeserializationResult tagged for a Flink OutputTag<RawRecord>("dlq").
Step 3
Send the side output to <topic>.dlq so checkpoints and watermarks continue advancing.
Alternative 1
Temporarily resetting consumer offset past the bad message (loses data if not copied first).
Failure Mode 1
Returning null silently from deserialize() drops data without any audit trail or alert.
// Flink-style safe deserialization pattern in Java
public void deserialize(ConsumerRecord<byte[], byte[]> record, Collector<DecodedOrError> out) {
try {
Event evt = avroDecoder.decode(record.value());
out.collect(DecodedOrError.ok(evt));
} catch (Exception ex) {
out.collect(DecodedOrError.dlq(record.topic(), record.partition(), record.offset(), record.value(), ex.getMessage()));
}
}
If auto.
If auto.register.schemas=true, a misconfigured producer can silently register an unreviewed schema or fail in production at runtime instead of CI.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Requires schema registration in CI/CD before app rollout, preventing rogue schema mutations in prod. |
Step 1
Register schemas explicitly during CI/CD deployment.
Step 2
Set auto.register.schemas=false and use.latest.version=true (or pinned schema ID) in production.
Alternative 1
Using RBAC on Schema Registry so production app service accounts have READ access only.
Failure Mode 1
Accidental auto.register.schemas=true combined with NONE compatibility lets any typo create a broken schema version.
producer_conf = {
"bootstrap.servers": "kafka:9092",
"schema.registry.url": "https://sr.internal",
"auto.register.schemas": False,
}
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Enforce
BACKWARDorFULL_TRANSITIVEcompatibility in Schema Registry with CI/CD gating, embed 5-byte Confluent wire-format schema IDs, and isolate poison-pillSerializationExceptionrecords to a DLQ. - 02Trade-off tip: Three-step rollout takes longer than an in-place edit, but guarantees zero downtime and zero decoder crashes.
- 03Watch out for: Returning
nullsilently fromdeserialize()drops data without any audit trail or alert.
Safe Event Schema Evolution Protobuf Avro — Normal → Load / Fault → Recovery
Enforce BACKWARD or FULLTRANSITIVE compatibility in Schema Registry with CI/CD gating, embed 5-byte Confluent wire-format schema IDs, and isolate poison-pill SerializationException records to a DLQ.
Putting Safe Event Schema Evolution with Protobuf & Avro Together
Enforce `BACKWARD` or `FULL_TRANSITIVE` compatibility in Schema Registry with CI/CD gating, embed 5-byte Confluent wire-format schema IDs, and isolate poison-pill `SerializationException` records to a DLQ.
Putting Safe Event Schema Evolution with Protobuf & Avro Together
- Parse the 5-byte header (
magic == 0x00+ 4-byteschema_id) or delegate toAvroDeserializer/ProtobufDeserializer. - On
SerializationErrororDecodeError, produce the rawmsg.value()to<topic>.dlqwith error headers. - Commit the source offset only after
dlq_producer.flush()succeeds.
Interview Readiness Check
Switch to KafkaRecordDeserializationSchema with a Collector<Either<BadRecord, ValidEvent>> (or Flink Side Output) that routes bad bytes to a DLQ sink.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Safe Event Schema Evolution with Protobuf & Avro
Enforce BACKWARD or FULL_TRANSITIVE compatibility in Schema Registry with CI/CD gating, embed 5-byte Confluent wire-format schema IDs, and isolate poison-pill SerializationException records to a DLQ.
