Lock-Free Union‑Find with Atomic Path Compression: Scaling Disjoint‑Set Operations for Massive Parallel Graph Workloads

Scenario context

Combining lock‑free Union‑Find with atomic path compression enables contention‑free disjoint‑set operations, delivering scalable performance for massively parallel graph algorithms without sacrificing correctness.

The core challenge addressed in this article is the design of a lock‑free Union‑Find data structure that supports atomic path compression while scaling to millions of concurrent operations on grap…

This walkthrough focuses on Problem Statement & System Requirements.

Before making a design choice, clarify:

• The core challenge addressed in this article is the design of a lock‑free Union‑Find data structure that supports atomic path compression while scaling to millions of concurrent operations on graph workloads that span t…
• In many modern analytics pipelines—social‑network community detection, real‑time recommendation engines, or distributed graph databases—disjoint‑set operations become a critical bottleneck when implemented with traditio…
• The goal is to eliminate contention hotspots, preserve linear‑time amortized complexity, and enable seamless integration into a Spring‑based microservice ecosystem that already leverages Resilience4j, Axon, and Kafka fo…

Concepts in play: event-driven-architecture, circuit-breaker, graph-traversal-algorithms, jvm-internals-and-garbage-collection

Stage 1 — Architectural Overview & System Mechanics

• The lock‑free Union‑Find component is positioned as a dedicated Set Service within the fraud‑detection micro‑service ecosystem.
• It receives edge streams from the Transaction Ingestion Service , applies atomic path‑compressed unions, and exposes a lightweight REST API for find queries.
• The service is built on Spring Boot, leverages Resilience4j for fault tolerance, and consumes Kafka events via Spring Kafka.
• Event‑driven state changes are captured with Axon, enabling eventual consistency and auditability.

The **data flow** is illustrated below. The ingestion service streams raw transaction edges to Kafka. Each Set Service instance subscribes to a dedicated topic partition, ensuring that the union‑find array is partitioned by node hash. This partitioning eliminates cross‑instance contention and allows each instance to maintain a local, NUMA‑aware parent array.

### System Mechanics

* **Atomic Path Compression**: Each `find` operation performs a read‑modify‑write cycle on the parent pointer using `VarHandle.compareAndSet`. The algorithm compresses the path in a single pass, guaranteeing that subsequent finds traverse a flat tree. Because the operation is lock‑free, multiple threads can concurrently compress distinct subtrees without blocking.

* **NUMA‑Aware Allocation**: The parent array is allocated in a thread‑local memory region aligned with the NUMA node of the owning thread pool. This reduces remote memory access latency and aligns with the contention‑free design.

* **Resilience & Retry**: Resilience4j’s circuit breaker wraps the Kafka consumer and REST endpoint. If a downstream snapshot operation to PostgreSQL fails, the breaker trips, and the service retries with exponential back‑off, preventing cascading failures.

* **Event Sourcing with Axon**: Every union operation is published as a domain event (`EdgeAdded`). Axon’s event store persists these events, enabling replay for state recovery and audit trails. Snapshots are taken periodically to avoid replaying the entire event history.

### Trade‑offs

| Aspect | Lock‑Free Union‑Find | Lock‑Based (e.g., `synchronized`) |
|--------|---------------------|-----------------------------------|
| Throughput | 10× higher on multi‑core systems | Limited by global lock contention |
| Latency | Deterministic O(α(n)) per operation | Variable, can spike under load |
| Memory | Slight overhead for atomic fields | Minimal overhead |
| Complexity | Requires careful use of `VarHandle` and memory fences | Simpler to implement but less scalable |

The architectural design balances raw performance with operational resilience. By isolating the Union‑Find logic into a horizontally scalable service, the system can ingest millions of edges per second while still servicing low‑latency `find` queries, meeting the stringent demands of real‑time fraud detection.

## 💡 Production Code Implementation & Patterns

In a production‑grade deployment the lock‑free union‑find core is wrapped in a thin, declarative façade that exposes the data structure over HTTP, streams edge events to Kafka, and safeguards the service with Resilience4j. The following subsections illustrate how to glue these pieces together while preserving the O(α(n)) guarantees of the underlying algorithm.

### 1. Spring REST façade

The `UnionFindController` delegates to a stateless `UnionFindService`. The service holds a singleton `AtomicUnionFind` instance and exposes two endpoints:

Observable outcome: Event‑driven state changes are captured with Axon, enabling eventual consistency and auditability.

End-to-end outcome

The core challenge addressed in this article is the design of a lock‑free Union‑Find data structure that supports atomic path compression while scaling to millions of concurrent operations on graph workloads that span tens of thousands of nodes.

Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

Failure and degraded modes

The source does not isolate a separate failure mode. Limits that matter for the scenario are called out in the stage decisions above.

• The lock‑free Union‑Find component is positioned as a dedicated Set Service within the fraud‑detection micro‑service ecosystem.