Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

Fixing Elasticsearch Deep
Pagination with search_after.

A multi-tenant e-commerce catalog cluster on Elasticsearch 8.11 (12 data nodes, 64 GB RAM / 31 GB JVM heap each, 20 primary shards with 1 replica) experiences 18-second G1GC stop-the-world pauses and HTTP 503 circuit-breaker rejections whenever scrapers or export jobs paginate past page 500 (from=50000, size=100). Redesign the pagination architecture, query limits, and client export path to eliminate coordinator OOMs and GC pauses while preserving deterministic ordering under concurrent writes.
Start the journey5 stages
Loading diagram…
Fixing Elasticsearch Deep Pagination with search_after architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

A multi-tenant e-commerce catalog cluster on Elasticsearch 8.11 (12 data nodes, 64 GB RAM / 31 GB JVM heap each, 20 primary shards with 1 replica) experiences 18-second G1GC stop-the-world pauses and HTTP 503 circuit-breaker rejections whenever scrapers or export jobs paginate past page 500 (`from=50000, size=100`). Redesign the pagination architecture, query limits, and client export path to eliminate coordinator OOMs and GC pauses while preserving deterministic ordering under concurrent writes.

From patterns to decisions.

A multi-tenant e-commerce catalog cluster on Elasticsearch 8.11 (12 data nodes, 64 GB RAM / 31 GB JVM heap each, 20 primary shards with 1 replica) experiences 18-second G1GC stop-the-world pauses and HTTP 503 circuit-breaker rejections whenever scrapers or export jobs paginate past page 500 (`from=50000, size=100`). Redesign the pagination architecture, query limits, and client export path to eliminate coordinator OOMs and GC pauses while preserving deterministic ordering under concurrent writes.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

In scatter-gather execution, every shard.

In scatter-gather execution, every shard must compute and return from + size sort tuples because shards do not know the global rank distribution.

ChoiceWhat it buysWhat it costs
Trade-off 1Capping `max_result_window` at 2000 protects JVM heap immediately but breaks any legacy client expecting `from > 2000`.

Trace the Query Phase

coordinator broadcasts from + size = 50,100 to all 20 primary/replica shards.

Calculate per-query merge memory

20 shards * 50,100 entries = 1,002,000 ScoreDoc tuples allocated on the coordinator.

Step 3

Connect large contiguous array allocations to G1GC humongous region pressure and stop-the-world Full GC pauses.

Alternative 1

Using _scroll (deprecated for real-time user queries and pins old segments without flexible sorting).

Failure Mode 1

Raising max_result_window to 100,000+ merely hides the validation error while guaranteeing coordinator OOMs under concurrency.

# Index template guardrail to block deep offset pagination
index_patterns: ["catalog-*"]
template:
  settings:
    index:
      max_result_window: 2000
      number_of_shards: 20
      number_of_replicas: 1
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

A Python service that opens.

A Python service that opens a 2-minute PIT, paginates with search_after + product_id tie-breaker, refreshes the PIT via the response, and closes it on completion.

ChoiceWhat it buysWhat it costs
Trade-off 1Using `product_id` instead of `_shard_doc` adds doc-values lookup overhead on ties but allows seamless recovery if the PIT expires.

Step 1

Open a PIT handle with keep_alive=2m before the first page.

Query `/_search` passing `pit

{id, keep_alive} and track_total_hits: false`.

Step 3

Use product_id as a deterministic tie-breaker so cursor recovery works even if a PIT expires.

Step 4

Close the PIT with DELETE /_pit in a finally block for batch exporters.

Alternative 1

Using _shard_doc when PIT expiration recovery is not required, as Elasticsearch appends it automatically and reads it faster.

Forgetting `track_total_hits

false` forces Lucene to traverse all matching postings to compute exact totals.

import base64, json
from elasticsearch import Elasticsearch

def fetch_page(es: Elasticsearch, index: str, cursor: str | None, size: int = 100):
    if cursor:
        state = json.loads(base64.urlsafe_b64decode(cursor))
        pit_id, search_after = state["pit_id"], state["sort"]
    else:
        pit = es.open_point_in_time(index=index, keep_alive="2m")
        pit_id, search_after = pit["id"], None

    body = {
        "size": size,
        "pit": {"id": pit_id, "keep_alive": "2m"},
        "track_total_hits": False,
        "sort": [{"updated_at": "desc"}, {"product_id": "asc"}],
    }
    if search_after:
        body["search_after"] = search_after

    resp = es.search(body=body)
    hits = resp["hits"]["hits"]
    next_cursor = None
    if len(hits) == size:
        next_state = {"pit_id": resp["pit_id"], "sort": hits[-1]["sort"]}
        next_cursor = base64.urlsafe_b64encode(json.dumps(next_state).encode()).decode()
    return hits, next_cursor
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

search_after + PIT is the.

search_after + PIT is the only pattern offering O(size) heap, snapshot consistency, and safe stateless API cursors.

ChoiceWhat it buysWhat it costs
Trade-off 1Hybrid routing preserves clickable page numbers 1-20 for UX while protecting the cluster from deep-offset abuse.

Step 1

Keep from/size strictly for UI pages 1-20 (offset <= 2000) where users click page numbers.

Step 2

Use search_after + PIT for infinite scroll, API cursors, and parallel sliced exports.

Step 3

Eliminate _scroll due to stateful server-side context pinning and lack of real-time query flexibility.

Alternative 1

Cursor-only APIs from page 1, simplifying backend code at the cost of removing page-number buttons.

Failure Mode 1

Exposing raw pit_id without HMAC signing or size validation allows malicious clients to pass size=10000.

# Hybrid router: allow shallow page jumps up to 2000, enforce cursor beyond
MAX_OFFSET = 2000
def validate_pagination(offset: int, size: int, cursor: str | None):
    if cursor is None and (offset + size) > MAX_OFFSET:
        raise ValueError(f"Deep offset {offset + size} exceeds {MAX_OFFSET}; use cursor pagination.")
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

Open PITs pin old Lucene.

Open PITs pin old Lucene segments after merges, causing disk bloat and file-descriptor exhaustion if left unclosed or uncapped.

ChoiceWhat it buysWhat it costs
Trade-off 1Reopening a PIT on expiry sacrifices strict point-in-time isolation across the boundary but prevents export job failure.

Step 1

Cap keep_alive at 2 minutes and monitor indices.search.open_contexts per node.

Step 2

Isolate bulk exports to dedicated read-only replica/search nodes or throttled async job queues.

Step 3

Handle search_context_missing_exception by reopening a PIT and resuming from the deterministic (updated_at, product_id) sort key.

Alternative 1

Taking a daily snapshot to S3 and running offline exports via Spark/DuckDB instead of hitting the live search cluster.

Failure Mode 1

Leaving orphaned PITs open with keep_alive=30m prevents Lucene from reclaiming deleted documents, filling NVMe disks.

from elasticsearch import BadRequestError, NotFoundError

def safe_fetch_page(es, index, cursor, size=100):
    try:
        return fetch_page(es, index, cursor, size)
    except (BadRequestError, NotFoundError) as exc:
        if "search_context_missing_exception" in str(exc) and cursor:
            state = json.loads(base64.urlsafe_b64decode(cursor))
            fresh_pit = es.open_point_in_time(index=index, keep_alive="2m")["id"]
            rebuilt = base64.urlsafe_b64encode(json.dumps({"pit_id": fresh_pit, "sort": state["sort"]}).encode()).decode()
            return fetch_page(es, index, rebuilt, size)
        raise
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

Use PIT slice partitioning (slice.

Use PIT slice partitioning (slice: {id, max}) aligned to a divisor of the shard count, combined with token-bucket rate limiting.

ChoiceWhat it buysWhat it costs
Trade-off 1Slicing with `max=10` speeds up exports 10x, but unthrottled workers can saturate data-node disk I/O and CPU.

Step 1

Open a single shared PIT for the entire export job so all workers read the exact same snapshot.

Step 2

Partition the export into max=5 or max=10 slices (dividing the 20 shards evenly) so each worker scans a subset of documents.

Step 3

Close the shared PIT once all slice workers finish.

Alternative 1

Reading directly from the source-of-truth Kafka topic or database CDC stream for downstream warehouse syncs.

Failure Mode 1

Setting max_slices larger than number_of_shards forces expensive _id hash filtering per document instead of per-shard partitioning.

def fetch_slice(es: Elasticsearch, pit_id: str, slice_id: int, max_slices: int, size: int = 500):
    search_after = None
    while True:
        body = {
            "size": size,
            "pit": {"id": pit_id, "keep_alive": "2m"},
            "slice": {"id": slice_id, "max": max_slices},
            "sort": [{"_shard_doc": "asc"}],
        }
        if search_after:
            body["search_after"] = search_after
        hits = es.search(body=body)["hits"]["hits"]
        if not hits:
            break
        yield hits
        search_after = hits[-1]["sort"]

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Replace offset-based from/size with stateless search_after paired with a Point-in-Time (_pit) ID and a tie-breaker sort key (_shard_doc or id) to keep memory O(size) per page.
  • 02Trade-off tip: Hybrid routing preserves clickable page numbers 1-20 for UX while protecting the cluster from deep-offset abuse.
  • 03Watch out for: Leaving orphaned PITs open with keep_alive=30m prevents Lucene from reclaiming deleted documents, filling NVMe disks.
Behavioral Progression · Elasticsearch Search After Vs Deep Pagination

Elasticsearch Search After Vs Deep Pagination — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Replace offset-based from/size with stateless searchafter paired with a Point-in-Time (pit) ID and a tie-breaker sort key (sharddoc or id) to keep memory O(size) per page.

Synthesis / One connected design

Putting Fixing Elasticsearch Deep Pagination with search_after Together

Replace offset-based `from/size` with stateless `search_after` paired with a Point-in-Time (`_pit`) ID and a tie-breaker sort key (`_shard_doc` or `id`) to keep memory O(size) per page.

A candidate architecture, not a universal answer

Putting Fixing Elasticsearch Deep Pagination with search_after Together

Step 1
Step 2
Step 3
Step 4
  1. Open a PIT handle with keep_alive=2m before the first page.
  2. Query /_search passing pit: {id, keep_alive} and track_total_hits: false.
  3. Use product_id as a deterministic tie-breaker so cursor recovery works even if a PIT expires.
  4. Close the PIT with DELETE /_pit in a finally block for batch exporters.
Open PITs pin old Lucene segments after merges, causing disk bloat and file-descriptor exhaustion if left unclosed or uncapped.
Interview Readiness Check

Open PITs pin old Lucene segments after merges, causing disk bloat and file-descriptor exhaustion if left unclosed or uncapped.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Fixing Elasticsearch Deep Pagination with search_after

Replace offset-based from/size with stateless search_after paired with a Point-in-Time (_pit) ID and a tie-breaker sort key (_shard_doc or id) to keep memory O(size) per page.