Skip to main content
← PROBLEMS / ENGINEERING
An end-to-end domain learning journey

Non-Blocking Async &
Structured Logging.

A 20-pod Spring Boot order-ingestion service (200 Tomcat worker threads/pod, gp3 EBS 3,000 IOPS limit) enables verbose DEBUG/JSON request logging during an incident. At 1,800 RPS/pod, each request writes 6 log lines (~4.5 KB). With Logback's synchronous FileAppender (immediateFlush=true), disk queue depth hits 128 and write latency spikes to 45 ms per line—blocking all 200 worker threads in FileOutputStream.writeBytes and dropping throughput to near zero.
Start the journey5 stages
Non-Blocking Async & Structured Logging architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

A 20-pod Spring Boot order-ingestion service (200 Tomcat worker threads/pod, gp3 EBS 3,000 IOPS limit) enables verbose DEBUG/JSON request logging during an incident. At 1,800 RPS/pod, each request writes 6 log lines (~4.5 KB). With Logback's synchronous `FileAppender` (`immediateFlush=true`), disk queue depth hits 128 and write latency spikes to 45 ms per line—blocking all 200 worker threads in `FileOutputStream.writeBytes` and dropping throughput to near zero.

From patterns to decisions.

A 20-pod Spring Boot order-ingestion service (200 Tomcat worker threads/pod, gp3 EBS 3,000 IOPS limit) enables verbose DEBUG/JSON request logging during an incident. At 1,800 RPS/pod, each request writes 6 log lines (~4.5 KB). With Logback's synchronous `FileAppender` (`immediateFlush=true`), disk queue depth hits 128 and write latency spikes to 45 ms per line—blocking all 200 worker threads in `FileOutputStream.writeBytes` and dropping throughput to near zero.

01
ENGINEERING · STAGE 01L4 · FUNDAMENTALS

Synchronous FileAppender acquires a global.

Synchronous FileAppender acquires a global lock around OutputStream.write() and flushes every line, turning EBS IOPS saturation into a 200-thread mutex pileup.

ChoiceWhat it buysWhat it costs
Trade-off 1immediateFlush=true guarantees each line hits the OS page cache immediately at the cost of destroying throughput.

Step 1

Show 1,800 RPS * 6 lines = 10,800 syscalls/sec per pod.

Step 2

Explain how gp3 3,000 IOPS limit drives write latency to 45 ms/line.

Step 3

Note that all 200 worker threads queue behind FileOutputStream.writeBytes.

Alternative 1

Writing to stdout (containerd json-file driver), which still hits the same node EBS volume.

Failure Mode 1

Complete service freeze whenever log verbosity increases or EBS bursts run out.

<!-- Broken synchronous Logback config -->
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
  <file>/var/log/app/app.log</file>
  <immediateFlush>true</immediateFlush> <!-- Forces 10,800 flushes/sec on worker threads -->
</appender>
02
ENGINEERING · STAGE 02L5 · APPLIED DESIGN

Implementation & Guardrails (2).

Uses Log4j2 AsyncLoggerContextSelector with 65,536-slot ring buffer, Discard policy for INFO/DEBUG, and 256 KB buffered RollingFileAppender with immediateFlush=false.

ChoiceWhat it buysWhat it costs
Trade-off 1Using Discard drops INFO/DEBUG logs if disk stalls > 6s, prioritizing API availability over verbose logs.

Step 1

Set AsyncLoggerContextSelector and asyncLoggerRingBufferSize=65536.

Step 2

Set asyncQueueFullPolicy=Discard and discardThreshold=INFO.

Step 3

Configure RollingFile with bufferSize=262144 and immediateFlush=false.

Alternative 1

Logback AsyncAppender with neverBlock=true and discardingThreshold=20.

Failure Mode 1

Forgetting asyncQueueFullPolicy=Discard still blocks worker threads when the 65k ring fills.

<!-- log4j2-spring.xml -->
<Configuration status="WARN">
  <Appenders>
    <RollingFile name="AppFile" fileName="/var/log/app/app.log"
                 filePattern="/var/log/app/app-%i.log"
                 bufferSize="262144" immediateFlush="false">
      <JsonTemplateLayout eventTemplateUri="classpath:EcsLayout.json"/>
      <Policies>
        <SizeBasedTriggeringPolicy size="100 MB"/>
      </Policies>
      <DefaultRolloverStrategy max="5"/>
    </RollingFile>
  </Appenders>
  <Loggers>
    <Root level="INFO">
      <AppenderRef ref="AppFile"/>
    </Root>
  </Loggers>
</Configuration>
<!-- JVM Flags:
-Dlog4j2.contextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector
-Dlog4j2.asyncLoggerRingBufferSize=65536
-Dlog4j2.asyncQueueFullPolicy=Discard
-Dlog4j2.discardThreshold=INFO -->
03
ENGINEERING · STAGE 03L6 · SCALE & TRADE-OFFS

Implementation & Guardrails (3).

Log4j2's AsyncLogger automatically flushes the 256 KB buffer whenever the Disruptor consumer hits endOfBatch (when the queue is momentarily drained), keeping lag in milliseconds.

ChoiceWhat it buysWhat it costs
Trade-off 1End-of-batch flushing gives both low latency at low RPS and high batching efficiency at high RPS.

Explain endOfBatch semantics

if 50 events are queued, Log4j2 writes all 50, then flushes once.

Step 2

Show that under low load (1 event), endOfBatch=true immediately, so lag is < 1 ms.

Step 3

Under high load, batching turns 10,800 syscalls into ~32–150 flushes/sec.

Alternative 1

Periodic timer-based flushing every 500 ms.

Failure Mode 1

Setting includeLocation=true in AsyncLogger forces a stack-trace walk on every log call, destroying throughput.

// Conceptual Disruptor consumer batching
public void onEvent(LogEvent event, long sequence, boolean endOfBatch) {
    appender.append(event); // writes into 256 KB BufferedOutputStream
    if (endOfBatch) {
        appender.flush();   // 1 syscall for the entire drained batch
    }
}
04
ENGINEERING · STAGE 04L6 · SCALE & TRADE-OFFS

Implementation & Guardrails (4).

Capturing caller class/line number requires walking the JVM stack (StackWalker / Throwable.getStackTrace()) synchronously on the worker thread before publishing to the Disruptor.

ChoiceWhat it buysWhat it costs
Disabling line numbers removes fileline from logs in exchange for 10x–20x higher logging throughput.

Step 1

Measure stack-walk cost (~15–30 us per log line * 10,800 lines = ~250 ms CPU per second per thread).

Step 2

Show that location calculation cannot be deferred to the background Disruptor thread.

Step 3

Disable includeLocation=false and rely on logger name + structured correlation IDs.

Alternative 1

Printing stack traces only on ERROR-level exceptions.

Failure Mode 1

CPU-bound GC pressure from millions of short-lived StackTraceElement arrays.

<Root level="INFO" includeLocation="false">
  <AppenderRef ref="AppFile"/>
</Root>
05
ENGINEERING · STAGE 05L7 · STAFF / PRINCIPAL

Implementation & Guardrails (5).

Export Log4j2 AsyncLogger RingBufferRemainingCapacity via JMX/Micrometer and alert on Linux node_disk_io_time_seconds_total.

ChoiceWhat it buysWhat it costs
Trade-off 1Auto-reverting DEBUG log levels after 15 minutes prevents accidental permanent verbose logging.

Step 1

Alert if RingBufferRemainingCapacity drops below 20% of 65,536.

Step 2

Monitor EBS VolumeQueueLength > 10 or await > 10 ms.

Step 3

Expose a runtime /actuator/loggers endpoint with auto-revert TTL for temporary DEBUG.

Alternative 1

Dynamic tail-based log sampling.

Failure Mode 1

Leaving DEBUG enabled after an incident slowly fills EBS and triggers Fluent Bit backpressure.

-- PromQL alerts for log ring buffer and disk saturation
log4j2_ringbuffer_remaining_capacity / 65536 < 0.2
rate(node_disk_write_time_seconds_total[1m]) / rate(node_disk_writes_completed_total[1m]) > 0.01

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Switch to Log4j2 AsyncLogger (LMAX Disruptor ring buffer) with batched flushing, immediateFlush=false, and discardThreshold backpressure so disk stalls never block request threads.
  • 02Trade-off tip: End-of-batch flushing gives both low latency at low RPS and high batching efficiency at high RPS.
  • 03Watch out for: CPU-bound GC pressure from millions of short-lived StackTraceElement arrays.
Behavioral Progression · Async Structured Logging Pipelines

Async Structured Logging Pipelines — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Switch to Log4j2 AsyncLogger (LMAX Disruptor ring buffer) with batched flushing, immediateFlush=false, and discardThreshold backpressure so disk stalls never block request threads.

Synthesis / One connected design

Putting Non-Blocking Async & Structured Logging Together

Switch to Log4j2 AsyncLogger (LMAX Disruptor ring buffer) with batched flushing, `immediateFlush=false`, and `discardThreshold` backpressure so disk stalls never block request threads.

A candidate architecture, not a universal answer

Putting Non-Blocking Async & Structured Logging Together

Step 1
Step 2
Step 3
  1. Set AsyncLoggerContextSelector and asyncLoggerRingBufferSize=65536.
  2. Set asyncQueueFullPolicy=Discard and discardThreshold=INFO.
  3. Configure RollingFile with bufferSize=262144 and immediateFlush=false.
Capturing caller class/line number requires walking the JVM stack (`StackWalker` / `Throwable.getStackTrace()`) synchronously on the worker thread before publishing to the Disruptor.
Interview Readiness Check

Capturing caller class/line number requires walking the JVM stack (StackWalker / Throwable.getStackTrace()) synchronously on the worker thread before publishing to the Disruptor.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Non-Blocking Async & Structured Logging

Switch to Log4j2 AsyncLogger (LMAX Disruptor ring buffer) with batched flushing, immediateFlush=false, and discardThreshold backpressure so disk stalls never block request threads.