Generator Functions In-Depth

Introduction

Note: This article assumes basic familiarity with generators in Python.

It hit me recently that I didn't understand something about how generator functions in Python work. In short, I was unsure exactly when and how they are determined by a Python implementation to be generators as opposed to "standard" functions, i.e. a function ending in return (whether explicit or falling off the end of the function).¹ That question may not make very much sense at this point, but I'm not sure how to succintly word it without background information. To that end, there are three key facts about generator functions that led me to this question, described below.

Before going on, I must note that I'm not suggesting that any behavior of generators is faulty or ill-designed. Each point below is clearly expressed as the intended behavior within documentation, and the experience to the typical user is intuitive and pleasant. My interest lies in how the behaviors are implemented under the hood in CPython, as will be seen.

That disclaimer aside, consider the following facts about generator functions.

1. Generator functions are functions

That may seem like just a tautology, but it's important to note that a generator function object has the same type, function, as a standard function.

>>> def gen(): yield 0

>>> def func(): return 0

>>> type(gen)
<class 'function'>
>>> type(func)
<class 'function'>

As seen above, a user-defined generator function is an object of class function just like a user-defined standard function. Both are created using the keydword def, so this seems intuitive enough. This fact may start to feel a little wonky, however, when considering the following two points.

2. Calling a generator function returns a generator object implicitly

An actual generator object (the iterable which controls execution of the function) doesn't come into play until a generator function is called. This is seen below.

>>> def gen(): yield 0

>>> type(gen)
<class 'function'>
>>> type(gen())
<class 'generator'>

The generator object return value is entirely implicit. As seen in gen above, a generator function may have no return statement, yet when called it will always return an instance of generator

This behavior is arguably odd because it is in stark contrast to a standard function, with which it would be a huge surprise if any object (other than None) was returned without any directive in the source whatsoever (discounting a wrapper designed for that purpose).²

3. The body of a generator function is not executed when a generator function is called

This is illustrated in the following example.

>>> def gen():
    print('gen starting...')
    yield from range(3)

>>> gen()
<generator object gen at 0x000000000348E480>
>>> for i in gen(): print(i)

gen starting...
0
1
2

When gen is merely called in the example above, the print call inside the function was not executed. It is not until the return value of gen is iterated through that 'gen starting...' is printed. Again, this is contrary to standard functions - calling a standard function should always be expected to execute the body of the function.

The Question

With the three behaviors above in mind, the question driving this article can be discussed - how and when does Python (an individual implementation of Python, really - more on that later) "realize" that a generator function should be treated as such? A generator function seems to be "just another function" for all intents and purposes after definition; yet, Python "knows" to not execute the function when called, and to instead return a generator object that controls execution of the function.

I had never previously given it much thought, but in my mind a generator function executed normally until it reached a yield, after which point a generator object is returned and the function henceforth treated as a generator, through some magic. Recall from above, however, that the body of a generator function is not executed when called, so this cannot be so.

The conclusion then, is that a function containing yield must be recognized as a generator function at definition time.³ But how exactly?

Looking to the Docs

The next logical question is what exactly happens with a function at definition time? The docs say

A function definition is an executable statement. Its execution binds the function name in the current local namespace to a function object (a wrapper around the executable code for the function). This function object contains a reference to the current global namespace as the global namespace to be used when the function is called.

The function definition does not execute the function body; this gets executed only when the function is called.

That quote doesn't help all that much; it makes no reference to identifying whether a function contains a yield, or to examining the function body at all. The data model entry for generator functions, glossary definition for generator function, and the yield expressions section are similarly unhelpful on this point.

Looking to the original proposal for generators in Python, PEP 255, one may find an important reference to what's happening under the hood at definition time. It states:

A generator function is an ordinary function object in all respects, but has the new CO_GENERATOR flag set in the code object's co_flags member.

The above quote finally provides a way forward - a function's code object.

The code object

So what exactly is a code object? The docs state it well:

Code objects represent byte-compiled executable Python code, or bytecode. The difference between a code object and a function object is that the function object contains an explicit reference to the function’s globals (the module in which it was defined), while a code object contains no context; also the default argument values are stored in the function object, not in the code object (because they represent values calculated at run-time). Unlike function objects, code objects are immutable and contain no references (directly or indirectly) to mutable objects.

Fortunately, code objects can be examined. As per the user-defined functions section of the data model page, all functions provide an attribute named __code__ which points to the function's code object.

Sidebar: Implementation vs. language specification

Code objects are getting into the low-level innards of Python, where it becomes necessary to differentiate between language and implementation. The Python language is the syntax and semantics thought of when one says he/she is "writing Python." However, that language needs to be translated into instructions that a computer can understand; that is where the implementation comes in.

The most common implementation of Python is CPython. In CPython, Python code is compiled into bytecode, a sort of intermediate machine language understandable and executed by the CPython bytecode interpreter (aka the Python virtual machine), written in C. CPython bytecode is saved automatically as .pyc files when a program is first run (or has changed).

Other major implementations of Python follow the general pattern of compiling a .py file into bytecode which is then interpreted, but do not necessarily compile to the CPython version of bytecode. For example, Jython compiles Python to Java bytecode, and IronPython compiles to .NET bytecode. PyPy does things differently, using just-in-time compilation, a topic too far out of scope for this article.

In the context of this article, what this means is that how generator functions behave to the end-user is all that is defined by the Python language. How exactly that behavior is accomplished is left to individual implementations. Therefore, this discussion should be considered to be specific to the CPython implementation.

The CO_GENERATOR flag

Getting back to code objects, the following comes from the Code Objects description in the docs. It provides a specific location of the CO_GENERATOR flag bit mentioned in PEP 255.

co_flags is an integer encoding a number of flags for the interpreter. The following flag bits are defined for co_flags: [...] bit 0x20 is set if the function is a generator.

As stated prior, the assumption is that this flag will be set appropriately just after definition time. This can be verified by examing __code__ attributes:

def generator_flag_is_set(func):
    generator_flag = 0x20
    return bool(func.__code__.co_flags & generator_flag)

def gen():
    yield from range(3)

def func():
    return 1

print(generator_flag_is_set(gen))  # True
print(generator_flag_is_set(func)) # False

The AND (&) operation in generator_flag_is_set is simply checking if the bit at position 0x20, defined in the docs to be position of the generator flag, is set. As expected, it is set for gen and not for func (and it's worth noting that both are only defined, never called).

There is actually an easier built-in way to see the flags on a function. The dis module provides tools to disassemble and examine bytecode. The dis.code_info report lists the flags in a readable fashion:

>>> def gen(): yield 1

>>> import dis
>>> print(dis.code_info(gen.__code__))
Name:              gen
Filename:          <pyshell#1>
Argument count:    0
Kw-only arguments: 0
Number of locals:  0
Stack size:        1
Flags:             OPTIMIZED, NEWLOCALS, GENERATOR, NOFREE
Constants:
   0: None
   1: 1

The important implication about __code__ existing and the flag being set after definition is that compilation of a function body into bytecode must happen at definition time. Definition and compile time are then somewhat interchangeable terms in this context.

Digging Deeper

It is now established that at definition/compile time, a particular flag is set on a function's code object if the function is a generator function. This flag is presumably how the interpreter knows to treat the function as a generator function, with all the aforementioned caveat behavior that entails.

The question is thus mostly answered, but just for fun it can be taken further. Here is where the CO_GENERATOR flag is set in the CPython compiler (line 14177 of Python/compile.c).⁴

static int
compute_code_flags(struct compiler *c)
{
    PySTEntryObject *ste = c->u->u_ste;
    int flags = 0;
    Py_ssize_t n;
    if (ste->ste_type == FunctionBlock) {
        flags |= CO_NEWLOCALS;
        if (!ste->ste_unoptimized)
            flags |= CO_OPTIMIZED;
        if (ste->ste_nested)
            flags |= CO_NESTED;
        if (ste->ste_generator)
            flags |= CO_GENERATOR; // CO_GENERATOR is set here
        if (ste->ste_varargs)
            flags |= CO_VARARGS;
        if (ste->ste_varkeywords)
            flags |= CO_VARKEYWORDS;
    }

// Rest of function omitted...
}

As seen above, the compiler is setting the CO_GENERATOR flag based on a member of a PySTEntryObject, a symbol table entry. This object comes from the compiler unit (c->u) representing a particular block of code (in this context, a function).

Per the docs, "[s]ymbol tables are generated by the compiler from AST [abstract syntax trees] just before bytecode is generated." In other words, the "this function is a generator" flag is being set by the compiler when it is generating bytecode for a given function at definition/compile time.

Creating a symbol table naturally involves traversing the syntax tree of some hypothetical generator function, so it's reasonable to assume the flag is set when a yield (or yield from) statement/expression is encountered. This is verified by studying the point at which a PySTEntryObject's ste_generator flag is set, seen in an excerpt from Python/symtable.c, line 11342.

static int
symtable_visit_expr(struct symtable *st, expr_ty e)
{
    // Recursion limit check omitted...
    switch (e->kind) {
        // Many cases omitted...
        case Yield_kind:
            if (e->v.Yield.value)
                VISIT(st, expr, e->v.Yield.value);
            st->st_cur->ste_generator = 1;
            break;
        case YieldFrom_kind:
            VISIT(st, expr, e->v.YieldFrom.value);
            st->st_cur->ste_generator = 1;
            break;
        // Many more cases omitted...
        }
    // Rest of function omitted...
}

As expected, if a yield or yield_from expression is encountered in a function while traversing its syntax tree, the ste_generator flag is set, which later results in the CO_GENERATOR flag being set on the code object.⁵

Conclusion

The short answer to how CPython determines a function is a generator function at definition time is "it sees a yield expression in the function," which is amusingly straightforward given one has to dive down to the compiler level to find where it happens.

Notes