A 20-pod Spring Boot order-ingestion service (200 Tomcat worker threads/pod, gp3 EBS 3,000 IOPS limit) enables verbose DEBUG/JSON request logging during an incident. At 1,800 RPS/pod, each request writes 6 log lines (~4.5 KB). With Logback's synchronous `FileAppender` (`immediateFlush=true`), disk queue depth hits 128 and write latency spikes to 45 ms per line—blocking all 200 worker threads in `FileOutputStream.writeBytes` and dropping throughput to near zero.
From patterns to decisions.
A 20-pod Spring Boot order-ingestion service (200 Tomcat worker threads/pod, gp3 EBS 3,000 IOPS limit) enables verbose DEBUG/JSON request logging during an incident. At 1,800 RPS/pod, each request writes 6 log lines (~4.5 KB). With Logback's synchronous `FileAppender` (`immediateFlush=true`), disk queue depth hits 128 and write latency spikes to 45 ms per line—blocking all 200 worker threads in `FileOutputStream.writeBytes` and dropping throughput to near zero.
Synchronous FileAppender acquires a global.
Synchronous FileAppender acquires a global lock around OutputStream.write() and flushes every line, turning EBS IOPS saturation into a 200-thread mutex pileup.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | immediateFlush=true guarantees each line hits the OS page cache immediately at the cost of destroying throughput. |
Step 1
Show 1,800 RPS * 6 lines = 10,800 syscalls/sec per pod.
Step 2
Explain how gp3 3,000 IOPS limit drives write latency to 45 ms/line.
Step 3
Note that all 200 worker threads queue behind FileOutputStream.writeBytes.
Alternative 1
Writing to stdout (containerd json-file driver), which still hits the same node EBS volume.
Failure Mode 1
Complete service freeze whenever log verbosity increases or EBS bursts run out.
<!-- Broken synchronous Logback config -->
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/var/log/app/app.log</file>
<immediateFlush>true</immediateFlush> <!-- Forces 10,800 flushes/sec on worker threads -->
</appender>
Implementation & Guardrails (2).
Uses Log4j2 AsyncLoggerContextSelector with 65,536-slot ring buffer, Discard policy for INFO/DEBUG, and 256 KB buffered RollingFileAppender with immediateFlush=false.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Using Discard drops INFO/DEBUG logs if disk stalls > 6s, prioritizing API availability over verbose logs. |
Step 1
Set AsyncLoggerContextSelector and asyncLoggerRingBufferSize=65536.
Step 2
Set asyncQueueFullPolicy=Discard and discardThreshold=INFO.
Step 3
Configure RollingFile with bufferSize=262144 and immediateFlush=false.
Alternative 1
Logback AsyncAppender with neverBlock=true and discardingThreshold=20.
Failure Mode 1
Forgetting asyncQueueFullPolicy=Discard still blocks worker threads when the 65k ring fills.
<!-- log4j2-spring.xml -->
<Configuration status="WARN">
<Appenders>
<RollingFile name="AppFile" fileName="/var/log/app/app.log"
filePattern="/var/log/app/app-%i.log"
bufferSize="262144" immediateFlush="false">
<JsonTemplateLayout eventTemplateUri="classpath:EcsLayout.json"/>
<Policies>
<SizeBasedTriggeringPolicy size="100 MB"/>
</Policies>
<DefaultRolloverStrategy max="5"/>
</RollingFile>
</Appenders>
<Loggers>
<Root level="INFO">
<AppenderRef ref="AppFile"/>
</Root>
</Loggers>
</Configuration>
<!-- JVM Flags:
-Dlog4j2.contextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector
-Dlog4j2.asyncLoggerRingBufferSize=65536
-Dlog4j2.asyncQueueFullPolicy=Discard
-Dlog4j2.discardThreshold=INFO -->
Implementation & Guardrails (3).
Log4j2's AsyncLogger automatically flushes the 256 KB buffer whenever the Disruptor consumer hits endOfBatch (when the queue is momentarily drained), keeping lag in milliseconds.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | End-of-batch flushing gives both low latency at low RPS and high batching efficiency at high RPS. |
Explain endOfBatch semantics
if 50 events are queued, Log4j2 writes all 50, then flushes once.
Step 2
Show that under low load (1 event), endOfBatch=true immediately, so lag is < 1 ms.
Step 3
Under high load, batching turns 10,800 syscalls into ~32–150 flushes/sec.
Alternative 1
Periodic timer-based flushing every 500 ms.
Failure Mode 1
Setting includeLocation=true in AsyncLogger forces a stack-trace walk on every log call, destroying throughput.
// Conceptual Disruptor consumer batching
public void onEvent(LogEvent event, long sequence, boolean endOfBatch) {
appender.append(event); // writes into 256 KB BufferedOutputStream
if (endOfBatch) {
appender.flush(); // 1 syscall for the entire drained batch
}
}
Implementation & Guardrails (4).
Capturing caller class/line number requires walking the JVM stack (StackWalker / Throwable.getStackTrace()) synchronously on the worker thread before publishing to the Disruptor.
| Choice | What it buys | What it costs |
|---|---|---|
| Disabling line numbers removes file | line from logs in exchange for 10x–20x higher logging throughput. |
Step 1
Measure stack-walk cost (~15–30 us per log line * 10,800 lines = ~250 ms CPU per second per thread).
Step 2
Show that location calculation cannot be deferred to the background Disruptor thread.
Step 3
Disable includeLocation=false and rely on logger name + structured correlation IDs.
Alternative 1
Printing stack traces only on ERROR-level exceptions.
Failure Mode 1
CPU-bound GC pressure from millions of short-lived StackTraceElement arrays.
<Root level="INFO" includeLocation="false">
<AppenderRef ref="AppFile"/>
</Root>
Implementation & Guardrails (5).
Export Log4j2 AsyncLogger RingBufferRemainingCapacity via JMX/Micrometer and alert on Linux node_disk_io_time_seconds_total.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Auto-reverting DEBUG log levels after 15 minutes prevents accidental permanent verbose logging. |
Step 1
Alert if RingBufferRemainingCapacity drops below 20% of 65,536.
Step 2
Monitor EBS VolumeQueueLength > 10 or await > 10 ms.
Step 3
Expose a runtime /actuator/loggers endpoint with auto-revert TTL for temporary DEBUG.
Alternative 1
Dynamic tail-based log sampling.
Failure Mode 1
Leaving DEBUG enabled after an incident slowly fills EBS and triggers Fluent Bit backpressure.
-- PromQL alerts for log ring buffer and disk saturation
log4j2_ringbuffer_remaining_capacity / 65536 < 0.2
rate(node_disk_write_time_seconds_total[1m]) / rate(node_disk_writes_completed_total[1m]) > 0.01
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Switch to Log4j2 AsyncLogger (LMAX Disruptor ring buffer) with batched flushing,
immediateFlush=false, anddiscardThresholdbackpressure so disk stalls never block request threads. - 02Trade-off tip: End-of-batch flushing gives both low latency at low RPS and high batching efficiency at high RPS.
- 03Watch out for: CPU-bound GC pressure from millions of short-lived StackTraceElement arrays.
Async Structured Logging Pipelines — Normal → Load / Fault → Recovery
Switch to Log4j2 AsyncLogger (LMAX Disruptor ring buffer) with batched flushing, immediateFlush=false, and discardThreshold backpressure so disk stalls never block request threads.
Putting Non-Blocking Async & Structured Logging Together
Switch to Log4j2 AsyncLogger (LMAX Disruptor ring buffer) with batched flushing, `immediateFlush=false`, and `discardThreshold` backpressure so disk stalls never block request threads.
Putting Non-Blocking Async & Structured Logging Together
- Set AsyncLoggerContextSelector and asyncLoggerRingBufferSize=65536.
- Set asyncQueueFullPolicy=Discard and discardThreshold=INFO.
- Configure RollingFile with bufferSize=262144 and immediateFlush=false.
Interview Readiness Check
Capturing caller class/line number requires walking the JVM stack (StackWalker / Throwable.getStackTrace()) synchronously on the worker thread before publishing to the Disruptor.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Non-Blocking Async & Structured Logging
Switch to Log4j2 AsyncLogger (LMAX Disruptor ring buffer) with batched flushing, immediateFlush=false, and discardThreshold backpressure so disk stalls never block request threads.
