Generator Functions in Python: How to Pause a Function Mid-Execution

Here's something Python can do that JavaScript can't, at least not natively, without async/await:

def count_up():
    yield 1
    yield 2
    yield 3

Call that function and you don't get 1. You also don't get [1, 2, 3]. You get a generator object; a paused function waiting to be resumed.

gen = count_up()
print(next(gen))  # 1
print(next(gen))  # 2
print(next(gen))  # 3

That's the core idea of generators: functions that can be paused at a specific point and resumed later, picking up exactly where they left off (local variables, execution position, everything...)

This article is about understanding generators from the ground up. And there's a practical reason to care beyond generators themselves: Python's @contextmanager decorator, the cleanest way to build context managers, is powered entirely by this mechanism. Once you understand generators, @contextmanager stops feeling like magic and start feeling obvious. We'll get there in Part 4.

The Problem Generators Solve

Before we get into the mechanics, let's understand why generators exist.

Imagine you write a function that generates the first n Fibonacci numbers. The naive approach returns a list:

def fibonacci(n):
    numbers = []
    a, b = 0, 1
    for _ in range(n):
        numbers.append(a)
        a, b = b, a + b
    return numbers

for num in fibonacci(10):
    print(num)

This works, but it has a problem: it builds the entire list in memory before returning. If n is 10 million, you're holding 10 million numbers in RAM before you've processed a single one.

What if you only need to process them one at a time? What if you only need the first few? You're paying the full memory cost upfront for no reason.

The generator function would look like this:

def fibonacci(n):
    a, b = 0, 1
    for _ in range(n):
        yield a
        a, b = b, a + b

for num in fibonacci(10):
    print(num)

This produces the same output, but never holds more than two numbers in memory at once. Each time the for loop asks for the next value, the function runs until it hits yield, hands the value over, and pauses. On the next iteration, it picks up right after the yield.

This is the key insight: a generator computes values on demand, not all at once.

What yield Actually Does

yield is the keyword that turns a regular function into a generator function. But it does more than just return a value: it suspends the entire execution state of the function.

Let's trace through a simple example step by step:

def simple_gen():
    print("before first yield")
    yield 1
    print("before second yield")
    yield 2
    print("after last yield")

gen = simple_gen()   # function body doesn't run yet — returns a generator object

val = next(gen)
# prints: "before first yield"
# pauses at `yield 1`
# val = 1

val = next(gen)
# resumes after `yield 1`
# prints: "before second yield"
# pauses at `yield 2`
# val = 2

val = next(gen)
# resumes after `yield 2`
# prints: "after last yield"
# function body finishes - raises StopIteration

Notice: calling simple_gen() doesn't execute a single line of the function body. It just create the generator object. The body only starts running when you call next() for the first time.

StopIteration - How Generators Signal "Done"

When a generator's function body finishes, either by reaching the end of hitting a return statement, it raises StopIteration. This is Python's built-in signal that an iterator has no more values.

def two_values():
    yield "first"
    yield "second"
    # function ends here — StopIteration raised automatically

gen = two_values()
print(next(gen))  # "first"
print(next(gen))  # "second"
print(next(gen))  # raises StopIteration 💥

In practice, you almost never call next() manually and catch StopIteration yourself. The for loop handles it automatically:

for value in two_values():
    print(value)
# "first"
# "second"
# loop exits cleanly when StopIteration is raised

This is why generators work seamlessly in for loops. The loop protocol calls next() under the hood and catches StopIteration to know when to stop.

Generators Are Lazy

This is the core behavioral difference from a regular function returning a list: generators are lazy. They don't compute values until they're asked for.

def big_range(n):
    i = 0
    while i < n:
        yield i
        i += 1

# This creates a generator instantly — no computation yet
gen = big_range(1_000_000_000)

# Only NOW does computation happen — and only for one value
print(next(gen))  # 0
print(next(gen)) # 1

Compare that to list(range(1_000_000_000)), which would try to allocate roughly 8GB of memory immediately.

Laziness makes generators ideal for:

• Large datasets: Process a file line by line without loading it all into memory

• Infinite sequences: A generator can produce values forever; a list can't.

• Pipelines: Chain generators together so that data flows through transformations one value at a time

Generators Remember Their State

This is what makes generators genuinely different from callbacks or regular functions. Every local variable, every loop counter, the exact position in the code, all of it is preserved between calls to next().

def stateful_counter(start, step):
    current = start
    while True:
        yield current
        current += step   # this runs after each yield, before the next one

counter = stateful_counter(10, 3)
print(next(counter))  # 10
print(next(counter))  # 13
print(next(counter))  # 16
print(next(counter))  # 19

current persists between calls. No class, no instance variables, no global state. The generator object itself carries the state.

Generator Expressions

Just like list comprehensions give you a concise way to build lists, generator expressions give you a concise way to build generators, with the same lazy evaluation.

The syntax is identical to a list comprehension, but with parentheses instead of square brackets:

# List comprehension — builds entire list in memory immediately
squares_list = [x ** 2 for x in range(10)]

# Generator expression — lazy, computes one value at a time
squares_gen = (x ** 2 for x in range(10))

print(squares_list)  # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
print(squares_gen)   # <generator object <genexpr> at 0x...>
print(next(squares_gen))  # 0
print(next(squares_gen)) # 1

They work identically in for loops:

for sq in (x ** 2 for x in range(5)):
    print(sq)

And they shine when passed directly to functions that consume iterables:

# List comprehension — computes all squares, builds list, then sums
total = sum([x ** 2 for x in range(1_000_000)])

# Generator expression — sums one value at a time, never holds the full list
total = sum(x ** 2 for x in range(1_000_000))

The second version uses a fraction of the memory. Notice that you can drop the extra parentheses when a generator expression is the only argument to a function call.

When to Use Each

A good rule of thumb: If you're building a collection to iterate over it once, a generator is probably the right tool.

Real-World Usage 1: Reading Large Files

The most practical everyday use of generators is reading files line by line without loading the whole file into memory:

def read_large_file(filepath):
    with open(filepath, "r", encoding="utf-8") as f:
        for line in f:
            yield line.strip()

for line in read_large_file("access.log"):
    if "ERROR" in line:
        print(line)

The file is read one line at a time. If the log file is 10GB, this uses essentially no extra memory, compared to f.readlines() which would load all 10GB into RAM.

Real-World Usage 2: Infinite Sequences

Generators are the natural tool for sequences that have no defined end:

def unique_ids(prefix="user"):
    count = 0
    while True:
        yield f"{prefix}_{count}"
        count += 1

id_gen = unique_ids()
print(next(id_gen))  # user_0
print(next(id_gen))  # user_1
print(next(id_gen))  # user_2

You'd never build an infinite list. But an infinite generator is perfectly reasonable. You just take as many values as you need.

The JavaScript Parallel

If you're coming from JavaScript, generators will look familiar. Javascript has them too, with the function* syntax, and the same yield keyword:

// JavaScript generator
function* fibonacci() {
    let a = 0, b = 1;
    while (true) {
        yield a;
        [a, b] = [b, a + b];
    }
}

const gen = fibonacci();
console.log(gen.next().value);  // 0
console.log(gen.next().value);  // 1
console.log(gen.next().value);  // 1

Python's generators came first, introduced by PEP 255 in Python 2.2 and JavaScript borrowed the concept later. The mechanics are nearly identical, with one key difference: in Python, next(gen) is a built-function, while in JavaScript it's gen.next(), a method on the generator object.

What You Can't Do with a Generator

A few important constraints worth knowing:

You can only iterate once. A generator is exhausted after you've consumed all its values. You can't reset it or iterate it again, you have to create a new one.

gen = (x ** 2 for x in range(3))
print(list(gen))  # [0, 1, 4]
print(list(gen))  # [] — already exhausted

You can't index into a generator. Unlike lists, generators don't support gen[2]. They only know "next".

gen = (x for x in range(10))
print(gen[3])  # TypeError: 'generator' object is not subscriptable

If you need random access, convert to a list first: list(gen)[3].

The Bridge to Context Managers

Here's why generators matter for this series.

When Python see yield inside a function, it treats the function completely differently. It becomes a generator function.

The @contextmanager decorator from contextlib takes exactly this mechanism and wires to __enter__ and __exit__:

from contextlib import contextmanager

@contextmanager
def managed_resource():
    print("setup")    # __enter__ — runs up to yield
    yield             # your with block runs here
    print("teardown") # __exit__ — runs after yield

The yield is literally a pause point. @contextmanager calls next() to run setup, then calls next() again after your with block finishes to run teardown.

You don't need to understand the full wiring yet, that's Article 4. But now you know the mechanism it relies on, and it's not magic. It's just a generator being driven by a decorator.

The Mental Model to Take Away

A generator function is a resumable function. Every time you ask for the next value, it runs until the next yield, hands the value, and freezes, preserving every local variable and its position in the code exactly as they were.

Three things to remember:

• yield pauses the function and send a value out

• next() resumes it from exactly where it paused

• StopIteration signals the generator is exhausted and for loops handle this automatically

Acronyms Used in This Article

• PEP - Python Enhancement Proposal: It's a design document used by the Python community to propose and discuss new language features. PEP 255 introduced generators and PEP 289 introduced generator expressions.

• RAM - Random Access Memory: It's the working memory your computer uses to store data while a program is running.

This is part 3 of 5 in the Python Context Managers Series.

Next up: Part 4 - Building Context Managers: __enter__, __exit__ and @contextmanager