List Comprehensions, Generators, and Lazy Evaluation in Python

Scenario context

Python list comprehensions offer clean, readable syntax for eager sequence construction, but allocate memory for all elements immediately. Swapping brackets for parentheses (...) creates a generator expression, which evaluates lazily — computing items only on demand. For complex, stateful pipelines, infinite sequences, or large files, use generator functions with the yield keyword to stream data with O(1) memory…

This walkthrough focuses on The MemoryError That Launched a Thousand Generators.

Before making a design choice, clarify:

• She is a data engineer at a logistics company, tasked with crunching a 10 GB CSV of shipping events.
• She opens her laptop, writes what feels like perfectly reasonable Python, and hits run: Three seconds later: Her laptop has 16 GB of RAM.
• Python tried to load 10 GB of CSV rows into a single list, and then the list comprehension allocated a second copy.

Concepts in play: python-concurrency-nogil, tree-data-structures, database-indexing, graph-traversal-algorithms

Stage 1 — Building Sequences the Pythonic Way — Comprehensions and Generator Expressions

• Before diving into yield , let's cover the simpler forms of lazy-leaning syntax that Python developers use every day: comprehensions and generator expressions .
• List Comprehensions A list comprehension is a concise, readable way to build a new list from an iterable.
• The pattern is: Compare the classic loop approach with the comprehension: You can also nest comprehensions to flatten 2D structures: Dictionary and Set Comprehensions The same syntax works for dict and set literals: Gen…
• Swap [...] for (...) and you get a generator expression instead of a list: The generator expression object is just a recipe — a suspended computation.

result = [expression for item in iterable if condition]

# Classic loop
squares = []
for x in range(10):
    squares.append(x ** 2)

# List comprehension — same result, one line
squares = [x ** 2 for x in range(10)]
# [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Observable outcome: Swap [...] for (...) and you get a generator expression instead of a list: The generator expression object is just a recipe — a suspended computation.

Stage 2 — How yield Turns Functions Into Lazy Factories

• List comprehensions and generator expressions are convenient, but they have limits.
• For complex multi-step logic, conditional branching, or sequences that depend on external state (like a file handle or a database cursor), you need a generator function — any function that contains the yield keyword.
• The yield Keyword — Pause, Return, Resume When Python encounters yield inside a function, it does something remarkable: it suspends the entire function frame — local variables, the current line number, open loops, every…
• The function is not done; it is paused.

def count_up(start, stop):
    current = start
    while current < stop:
        print(f"  [generator] about to yield {current}")
        yield current
        print(f"  [generator] resumed after yield, incrementing")
        current += 1
    print("  [generator] loop finished — StopIteration coming")

gen = count_up(1, 4)

print("Calling next() the first time:")
val = next(gen)       # Runs until the first yield, then pauses
print(f"Got: {val}")

print("Calling next() the second time:")
val = next(gen)       # Resumes from after yield, runs until next yield
print(f"Got: {val}")

print("Calling next() the third time:")
val = next(gen)
print(f"Got: {val}")

print("Calling next() a fourth time — will raise StopIteration:")
try:
    next(gen)
except StopIteration:
    print("StopIteration raised — generator is exhausted")

Calling next() the first time:
  [generator] about to yield 1
Got: 1
Calling next() the second time:
  [generator] resumed after yield, incrementing
  [generator] about to yield 2
Got: 2
Calling next() the third time:
  [generator] resumed after yield, incrementing
  [generator] about to yield 3
Got: 3
Calling next() a fourth time — will raise StopIteration:
  [generator] loop finished — StopIteration coming
StopIteration raised — generator is exhausted

Observable outcome: The function is not done; it is paused.

Stage 3 — Under the Hood: Python Frame Objects and the Generator Protocol

• The Internals of Generator Execution When Python calls a regular function, it creates a frame object ( PyFrameObject in CPython) on the call stack.
• The frame holds local variables, the code object, the current instruction pointer, and the evaluation stack.
• When the function returns, the frame is discarded.
• Generator functions work differently.

import dis
import inspect

def simple_gen():
    yield 1
    yield 2

gen = simple_gen()

# The generator object holds a live frame
print(gen.gi_frame)          # <frame object at 0x...>
print(gen.gi_frame.f_lineno) # current line (points to first yield)

next(gen)   # run to first yield

print(gen.gi_frame.f_lineno) # now points to the second yield
print(gen.gi_suspended)      # True — frame is frozen between yields

next(gen)   # run to second yield
next(gen)   # exhausts the generator

print(gen.gi_frame)          # None — frame has been released

import sys
import tracemalloc

def measure_list_vs_generator(n: int = 1_000_000) -> None:
    print(f"Comparison for n={n:,} items\n{'─'*45}")

    # --- List comprehension ---
    tracemalloc.start()
    result_list = [x * 2 for x in range(n)]
    _, list_peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()

    # --- Generator expression ---
    tracemalloc.start()
    result_gen = (x * 2 for x in range(n))
    _, gen_peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()

    print(f"List comprehension")
    print(f"  sys.getsizeof : {sys.getsizeof(result_list):>12,} bytes")
    print(f"  peak heap     : {list_peak:>12,} bytes")
    print()
    print(f"Generator expression")
    print(f"  sys.getsizeof : {sys.getsizeof(result_gen):>12,} bytes")
    print(f"  peak heap     : {gen_peak:>12,} bytes")
    print()

    ratio = list_peak / gen_peak if gen_peak > 0 else float("inf")
    print(f"Memory ratio (list / gen): {ratio:,.0f}x")

measure_list_vs_generator()

Observable outcome: Generator functions work differently.

Stage 4 — Where Generators Shine in the Real World

• Infinite Sequence Generation Because a generator only computes the next value on demand, it can represent sequences with no defined end.
• A while True loop inside a generator produces values forever without consuming infinite memory.
• The caller controls when to stop.
• Streaming File Processing Any file that is too large to fit in RAM is a natural candidate for a generator reader.

import itertools

def natural_numbers(start: int = 0):
    """Generates 0, 1, 2, 3, ... indefinitely."""
    n = start
    while True:
        yield n
        n += 1

# Safe: itertools.islice stops after 5 pulls
first_five = list(itertools.islice(natural_numbers(), 5))
print(first_five)   # [0, 1, 2, 3, 4]

def read_large_csv(filepath: str):
    """Yields one parsed row dict at a time from a large CSV."""
    import csv
    with open(filepath, newline="", encoding="utf-8") as f:
        reader = csv.DictReader(f)
        for row in reader:
            yield row

# Downstream code never loads the whole file
for row in read_large_csv("events_2024.csv"):
    process(row)

Observable outcome: Streaming File Processing Any file that is too large to fit in RAM is a natural candidate for a generator reader.

Stage 5 — When Lazy Evaluation Becomes a Liability

• Generators are not always the right tool.
• Knowing when not to use them is just as important.
• Generators Are Single-Pass — You Cannot Rewind Once a generator is exhausted, it is gone.
• There is no reset() , no seek(0) .

data_gen = (float(x) for x in range(100))

# First pass: compute mean
total = sum(data_gen)          # works fine
# Second pass: generator is now exhausted!
count = sum(1 for _ in data_gen)   # returns 0 — the generator gave nothing

# Debugging trick: force a stage to a list to inspect it
active_users = list(filter_active(parse_rows(raw)))  # now inspectable
print(f"Active users count: {len(active_users)}")

Observable outcome: There is no reset() , no seek(0) .

Stage 6 — Choosing Between a List Comprehension, Generator Expression, and yield Generator

• Use this table to make the decision quickly:

Observable outcome: Use this table to make the decision quickly:

End-to-end outcome

After working through the stages, you should be able to explain:

• Lesson 1 — The 104-byte illusion can mislead you.
• sys.getsizeof(gen) reports the generator object shell, not the frame.

Alternatives considered

Lazy Pull vs. Eager Push — Visualizing the Generator Pipeline

The key architectural difference between an eager list pipeline and a lazy generator pipeline is who drives the data flow .

Failure and degraded modes

The source does not isolate a separate failure mode. Limits that matter for the scenario are called out in the stage decisions above.