A high-frequency Java telemetry ingestion service runs 8 worker threads on an 8-core x86_64 server, each incrementing its own dedicated counter in a shared long[8] array (or adjacent volatile fields in a metrics object) at 100M events/sec. Despite zero locks and zero logical data sharing, CPU utilization spikes to 100% sys/kernel-like stall cycles and total throughput collapses from ~800M ops/sec to ~25M ops/sec. Diagnose the hardware root cause and redesign the counter structure in Java to restore linear multi-core scaling.
From patterns to decisions.
A high-frequency Java telemetry ingestion service runs 8 worker threads on an 8-core x86_64 server, each incrementing its own dedicated counter in a shared long[8] array (or adjacent volatile fields in a metrics object) at 100M events/sec. Despite zero locks and zero logical data sharing, CPU utilization spikes to 100% sys/kernel-like stall cycles and total throughput collapses from ~800M ops/sec to ~25M ops/sec. Diagnose the hardware root cause and redesign the counter structure in Java to restore linear multi-core scaling.
Adjacent 8-byte longs share a.
Adjacent 8-byte longs share a 64-byte L1 cache line, causing MESI Read-For-Ownership invalidations across cores.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Compact 64-byte footprint maximizes cache density for read-only data, but serializes concurrent writes. |
Calculate memory layout
8 longs * 8 bytes = 64 bytes (1 cache line).
Step 2
Trace MESI transitions (M -> I -> M) when Core 0 and Core 1 alternate writes.
Step 3
Verify with JMH + -prof perfnorm (L1-dcache-load-misses) or Linux perf c2c (HITM).
Alternative 1
Use perf stat -e cache-misses,L1-dcache-load-misses if perf c2c is unavailable in a container.
Failure Mode 1
Misdiagnosing false sharing as GC pause or OS scheduler contention because CPU % looks busy.
// Contended layout: all 8 threads hammer the same 64-byte cache line
public final class ContendedCounters {
private final AtomicLongArray counters = new AtomicLongArray(8);
public void increment(int workerId) {
counters.getAndIncrement(workerId);
}
}
Stride each worker slot by.
Stride each worker slot by 16 longs (128 bytes) in an AtomicLongArray, or use LongAdder for unbounded dynamic threads.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | `getPlain` + `setRelease` avoids `LOCK XADD` (~10 ns) on x86_64, dropping increment cost to ~1-2 ns when each slot has a single writer. | |
| Trade-off 2 | Consumes 128 bytes per worker instead of 8 bytes (1 KB for 8 workers). |
Step 1
Allocate new AtomicLongArray(numWorkers * 16) so each worker owns a 128-byte region.
Step 2
Use lazySet / setRelease when only one thread writes each slot and exact atomic RMW is unnecessary.
Step 3
Sum across i = 0..numWorkers-1 at index i * 16 for the 100 ms scraper.
Alternative 1
Use java.util.concurrent.atomic.LongAdder when threads are spawned dynamically from ForkJoinPool or virtual threads.
Failure Mode 1
Calling incrementSingleWriter from two threads with the same workerId loses updates; use getAndIncrement if multiple threads share a slot.
import java.util.concurrent.atomic.AtomicLongArray;
public final class PaddedWorkerCounters {
private static final int STRIDE_LONGS = 16; // 16 * 8B = 128B (defeats 64B line + spatial prefetcher)
private final int numWorkers;
private final AtomicLongArray slots;
public PaddedWorkerCounters(int numWorkers) {
this.numWorkers = numWorkers;
this.slots = new AtomicLongArray(numWorkers * STRIDE_LONGS);
}
// Single-writer per workerId: avoid expensive LOCK XADD by reading plain/opaque + setRelease
public void incrementSingleWriter(int workerId) {
int idx = workerId * STRIDE_LONGS;
long next = slots.getPlain(idx) + 1;
slots.setRelease(idx, next);
}
public long sum() {
long total = 0;
for (int i = 0; i < numWorkers; i++) {
total += slots.getAcquire(i * STRIDE_LONGS);
}
return total;
}
}
Strided arrays and LongAdder work.
Strided arrays and LongAdder work out of the box on JDK 17+, whereas @Contended requires -XX:-RestrictContended for application classes.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Inheritance padding works without special JVM flags because the JVM spec lays out superclass fields before subclass fields. | |
| Trade-off 2 | Increases object count if thousands of counters are allocated. |
Verify JVM flag requirements
@Contended is restricted to java.base unless -XX:-RestrictContended is passed.
Step 2
Use class-hierarchy padding (PrePad -> Value -> PostPad) when padding individual objects like Disruptor Sequence.
Alternative 1
Use LongAdder whenever you do not need per-worker slot indexing.
Failure Mode 1
Putting all padding fields in one flat class allows HotSpot field-packing to place value adjacent to another object's header.
abstract class LhsPad { long p1, p2, p3, p4, p5, p6, p7; }
abstract class ValueHolder extends LhsPad { volatile long value; }
public final class PaddedSequence extends ValueHolder {
long p9, p10, p11, p12, p13, p14, p15;
}
Never bind per-thread padded buffers.
Never bind per-thread padded buffers to ThreadLocal for 100k virtual threads; use LongAdder or carrier-thread-bounded striping.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | `Striped64` bounds stripe memory to O(CPU_CORES) regardless of virtual thread count. | |
| Trade-off 2 | `sum()` may slightly lag concurrent writes without a global stop-the-world lock. |
Step 1
Recognize that 100,000 virtual threads * 128B ThreadLocal = 12.8 MB of fragmented heap.
Step 2
LongAdder (Striped64) caps its internal Cell[] table at the nearest power of 2 >= Runtime.getRuntime().availableProcessors().
Alternative 1
Shard by Thread.currentThread().threadId() & (STRIPE_COUNT - 1) with STRIPE_COUNT = nextPowerOf2(cores * 2).
Failure Mode 1
Allocating per-virtual-thread ThreadLocal padded objects causes OOM or severe GC pressure.
import java.util.concurrent.atomic.LongAdder;
public final class VirtualThreadSafeCounter {
private final LongAdder adder = new LongAdder();
public void recordEvent() {
adder.increment();
}
public long snapshot() {
return adder.sum();
}
}
Pad both the head and.
Pad both the head and tail sequence cursors onto separate 128-byte cache lines and cache the remote cursor locally in each thread.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Eliminates cross-core traffic on every single ring-buffer offer/poll, cutting p99 latency from ~60 ns to < 10 ns. |
Step 1
Place producerSeq and consumerSeq on distinct 128-byte cache lines.
Step 2
Cache cachedConsumerSeq inside the producer's cache line so the producer only reads the volatile consumerSeq when the ring appears full.
Alternative 1
Use AtomicLongArray with head at index 16 and tail at index 32.
Failure Mode 1
Forgetting to pad the ring buffer's underlying Object[] array ends or letting adjacent slots share a line when elements are small primitives.
public final class SpscRingCursors {
// Producer cache line
long p1, p2, p3, p4, p5, p6, p7;
volatile long producerSeq;
long cachedConsumerSeq;
long p8, p9, p10, p11, p12, p13;
// Consumer cache line
volatile long consumerSeq;
long cachedProducerSeq;
long p14, p15, p16, p17, p18, p19;
}
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Isolate hot per-thread counters onto separate 64-byte cache lines using LongAdder, thread-local buffers, or @Contended/manual padding so cores never issue Read-For-Ownership (RFO) invalidations.
- 02Trade-off tip: Inheritance padding works without special JVM flags because the JVM spec lays out superclass fields before subclass fields.
- 03Trade-off tip: Increases object count if thousands of counters are allocated.
- 04Watch out for: Allocating per-virtual-thread
ThreadLocalpadded objects causes OOM or severe GC pressure.
Eliminating Cpu Cache False Sharing — Normal → Load / Fault → Recovery
Isolate hot per-thread counters onto separate 64-byte cache lines using LongAdder, thread-local buffers, or @Contended/manual padding so cores never issue Read-For-Ownership (RFO) invalidations.
Putting Eliminating CPU Cache-Line False Sharing Together
Isolate hot per-thread counters onto separate 64-byte cache lines using LongAdder, thread-local buffers, or @Contended/manual padding so cores never issue Read-For-Ownership (RFO) invalidations.
Putting Eliminating CPU Cache-Line False Sharing Together
- Allocate
new AtomicLongArray(numWorkers * 16)so each worker owns a 128-byte region. - Use
lazySet/setReleasewhen only one thread writes each slot and exact atomic RMW is unnecessary. - Sum across
i = 0..numWorkers-1at indexi * 16for the 100 ms scraper.
Interview Readiness Check
Never bind per-thread padded buffers to ThreadLocal for 100k virtual threads; use LongAdder or carrier-thread-bounded striping.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Eliminating CPU Cache-Line False Sharing
Isolate hot per-thread counters onto separate 64-byte cache lines using LongAdder, thread-local buffers, or @Contended/manual padding so cores never issue Read-For-Ownership (RFO) invalidations.
