# Generator Functions in Python: How to Pause a Function Mid-Execution

Here's something Python can do that JavaScript can't, at least not natively, without async/await:

```python
def count_up():
    yield 1
    yield 2
    yield 3
```

Call that function and you don't get `1`. You also don't get `[1, 2, 3]`. You get **a generator object**; a paused function waiting to be resumed.

```python
gen = count_up()
print(next(gen))  # 1
print(next(gen))  # 2
print(next(gen))  # 3
```

That's the core idea of generators: functions that can be **paused at a specific point and resumed later**, picking up exactly where they left off (local variables, execution position, everything...)

This article is about understanding generators from the ground up. And there's a practical reason to care beyond generators themselves: **Python's** `@contextmanager` **decorator, the cleanest way to build context managers, is powered entirely by this mechanism**. Once you understand generators, `@contextmanager` stops feeling like magic and start feeling obvious. We'll get there in Part 4.

* * *

## The Problem Generators Solve

Before we get into the mechanics, let's understand why generators exist.

Imagine you write a function that generates the first `n` Fibonacci numbers. The naive approach returns a list:

```python
def fibonacci(n):
    numbers = []
    a, b = 0, 1
    for _ in range(n):
        numbers.append(a)
        a, b = b, a + b
    return numbers

for num in fibonacci(10):
    print(num)
```

This works, but it has a problem: it builds the **entire list in memory** before returning. If `n` is 10 million, you're holding 10 million numbers in RAM before you've processed a single one.

What if you only need to process them one at a time? What if you only need the first few? You're paying the full memory cost upfront for no reason.

The generator function would look like this:

```python
def fibonacci(n):
    a, b = 0, 1
    for _ in range(n):
        yield a
        a, b = b, a + b

for num in fibonacci(10):
    print(num)
```

This produces the same output, but never holds more than two numbers in memory at once. Each time the `for` loop asks for the next value, the function runs until it hits `yield`, hands the value over, and **pauses**. On the next iteration, it picks up right after the `yield`.

This is the key insight: a generator computes values **on demand**, not all at once.

* * *

## What `yield` Actually Does

`yield` is the keyword that turns a regular function into a generator function. But it does more than just return a value: it **suspends the entire execution state** of the function.

Let's trace through a simple example step by step:

```python
def simple_gen():
    print("before first yield")
    yield 1
    print("before second yield")
    yield 2
    print("after last yield")
```

```python
gen = simple_gen()   # function body doesn't run yet — returns a generator object
```

```python
val = next(gen)
# prints: "before first yield"
# pauses at `yield 1`
# val = 1
```

```python
val = next(gen)
# resumes after `yield 1`
# prints: "before second yield"
# pauses at `yield 2`
# val = 2
```

```python
val = next(gen)
# resumes after `yield 2`
# prints: "after last yield"
# function body finishes - raises StopIteration
```

Notice: calling `simple_gen()` doesn't execute a single line of the function body. It just create the generator object. The body only starts running when you call `next()` for the first time.

* * *

## `StopIteration` - How Generators Signal "Done"

When a generator's function body finishes, either by reaching the end of hitting a `return` statement, it raises `StopIteration`. This is Python's built-in signal that an iterator has no more values.

```python
def two_values():
    yield "first"
    yield "second"
    # function ends here — StopIteration raised automatically

gen = two_values()
print(next(gen))  # "first"
print(next(gen))  # "second"
print(next(gen))  # raises StopIteration 💥
```

In practice, you almost never call `next()` manually and catch `StopIteration` yourself. The `for` loop handles it automatically:

```python
for value in two_values():
    print(value)
# "first"
# "second"
# loop exits cleanly when StopIteration is raised
```

This is why generators work seamlessly in `for` loops. The loop protocol calls `next()` under the hood and catches `StopIteration` to know when to stop.

* * *

## Generators Are Lazy

This is the core behavioral difference from a regular function returning a list: generators are **lazy**. They don't compute values until they're asked for.

```python
def big_range(n):
    i = 0
    while i < n:
        yield i
        i += 1

# This creates a generator instantly — no computation yet
gen = big_range(1_000_000_000)

# Only NOW does computation happen — and only for one value
print(next(gen))  # 0
print(next(gen)) # 1
```

Compare that to `list(range(1_000_000_000))`, which would try to allocate roughly 8GB of memory immediately.

Laziness makes generators ideal for:

*   **Large datasets**: Process a file line by line without loading it all into memory
    
*   **Infinite sequences**: A generator can produce values forever; a list can't.
    
*   **Pipelines**: Chain generators together so that data flows through transformations one value at a time
    

* * *

## Generators Remember Their State

This is what makes generators genuinely different from callbacks or regular functions. Every local variable, every loop counter, the exact position in the code, all of it is **preserved between calls to** `next()`.

```python
def stateful_counter(start, step):
    current = start
    while True:
        yield current
        current += step   # this runs after each yield, before the next one

counter = stateful_counter(10, 3)
print(next(counter))  # 10
print(next(counter))  # 13
print(next(counter))  # 16
print(next(counter))  # 19
```

`current` persists between calls. No class, no instance variables, no global state. The generator object itself carries the state.

* * *

## Generator Expressions

Just like list comprehensions give you a concise way to build lists, **generator expressions** give you a concise way to build generators, with the same lazy evaluation.

The syntax is identical to a list comprehension, but with parentheses instead of square brackets:

```python
# List comprehension — builds entire list in memory immediately
squares_list = [x ** 2 for x in range(10)]

# Generator expression — lazy, computes one value at a time
squares_gen = (x ** 2 for x in range(10))
```

```python
print(squares_list)  # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
print(squares_gen)   # <generator object <genexpr> at 0x...>
print(next(squares_gen))  # 0
print(next(squares_gen)) # 1
```

They work identically in `for` loops:

```python
for sq in (x ** 2 for x in range(5)):
    print(sq)
```

And they shine when passed directly to functions that consume iterables:

```python
# List comprehension — computes all squares, builds list, then sums
total = sum([x ** 2 for x in range(1_000_000)])

# Generator expression — sums one value at a time, never holds the full list
total = sum(x ** 2 for x in range(1_000_000))
```

The second version uses a fraction of the memory. Notice that you can drop the extra parentheses when a generator expression is the only argument to a function call.

* * *

## When to Use Each

| Situation | Use |
| --- | --- |
| Need all values at once, or index into them | List comprehension or `list()` |
| Processing values one at a time | Generator expression |
| Complex stateful logic between values | Generator function with `yield` |
| Potentially infinite sequence | Generator function |
| Passing to `sum()`, `max()`, `min()`, `any()`, `all()` | Generator expression |

A good rule of thumb: If you're building a collection to iterate over it once, a generator is probably the right tool.

* * *

## Real-World Usage 1: Reading Large Files

The most practical everyday use of generators is reading files line by line without loading the whole file into memory:

```python
def read_large_file(filepath):
    with open(filepath, "r", encoding="utf-8") as f:
        for line in f:
            yield line.strip()

for line in read_large_file("access.log"):
    if "ERROR" in line:
        print(line)
```

The file is read one line at a time. If the log file is 10GB, this uses essentially no extra memory, compared to `f.readlines()` which would load all 10GB into RAM.

* * *

## Real-World Usage 2: Infinite Sequences

Generators are the natural tool for sequences that have no defined end:

```python
def unique_ids(prefix="user"):
    count = 0
    while True:
        yield f"{prefix}_{count}"
        count += 1

id_gen = unique_ids()
print(next(id_gen))  # user_0
print(next(id_gen))  # user_1
print(next(id_gen))  # user_2
```

You'd never build an infinite list. But an infinite generator is perfectly reasonable. You just take as many values as you need.

* * *

## The JavaScript Parallel

If you're coming from JavaScript, generators will look familiar. Javascript has them too, with the `function*` syntax, and the same `yield` keyword:

```javascript
// JavaScript generator
function* fibonacci() {
    let a = 0, b = 1;
    while (true) {
        yield a;
        [a, b] = [b, a + b];
    }
}

const gen = fibonacci();
console.log(gen.next().value);  // 0
console.log(gen.next().value);  // 1
console.log(gen.next().value);  // 1
```

Python's generators came first, introduced by [PEP 255 in Python 2.2](https://peps.python.org/pep-0255/) and JavaScript borrowed the concept later. The mechanics are nearly identical, with one key difference: in Python, `next(gen)` is a built-function, while in JavaScript it's `gen.next()`, a method on the generator object.

* * *

## What You Can't Do with a Generator

A few important constraints worth knowing:

**You can only iterate once.** A generator is exhausted after you've consumed all its values. You can't reset it or iterate it again, you have to create a new one.

```python
gen = (x ** 2 for x in range(3))
print(list(gen))  # [0, 1, 4]
print(list(gen))  # [] — already exhausted
```

**You can't index into a generator.** Unlike lists, generators don't support `gen[2]`. They only know "next".

```python
gen = (x for x in range(10))
print(gen[3])  # TypeError: 'generator' object is not subscriptable
```

If you need random access, convert to a list first: `list(gen)[3]`.

* * *

## The Bridge to Context Managers

Here's why generators matter for this series.

When Python see `yield` inside a function, it treats the function completely differently. It becomes a generator function.

The `@contextmanager` decorator from `contextlib` takes exactly this mechanism and wires to `__enter__` and `__exit__`:

```python
from contextlib import contextmanager

@contextmanager
def managed_resource():
    print("setup")    # __enter__ — runs up to yield
    yield             # your with block runs here
    print("teardown") # __exit__ — runs after yield
```

The `yield` is literally a pause point. `@contextmanager` calls `next()` to run setup, then calls `next()` again after your `with` block finishes to run teardown.

You don't need to understand the full wiring yet, that's Article 4. But now you know the mechanism it relies on, and it's not magic. It's just a generator being driven by a decorator.

* * *

## The Mental Model to Take Away

A generator function is a **resumable function**. Every time you ask for the next value, it runs until the next `yield`, hands the value, and freezes, preserving every local variable and its position in the code exactly as they were.

Three things to remember:

*   `yield` pauses the function and send a value out
    
*   `next()` resumes it from exactly where it paused
    
*   `StopIteration` signals the generator is exhausted and `for` loops handle this automatically
    

* * *

## Acronyms Used in This Article

*   **PEP -** Python Enhancement Proposal: It's a design document used by the Python community to propose and discuss new language features. [PEP 255](https://peps.python.org/pep-0255/) introduced generators and [PEP 289](https://peps.python.org/pep-0289/#abstract) introduced generator expressions.
    
*   **RAM** - Random Access Memory: It's the working memory your computer uses to store data while a program is running.
    

* * *

> **This is part 3 of 5 in the Python Context Managers Series.**
> 
> Next up: **Part 4 - Building Context Managers:** `__enter__`, `__exit__` and `@contextmanager`