Advanced Python

Generators#

A regular function computes and returns a value once. A generator function uses yield instead of return and produces a stream of values lazily — one at a time, only when asked — instead of building an entire collection in memory up front:

def count_up_to(n):
    i = 1
    while i <= n:
        yield i
        i += 1

for number in count_up_to(5):
    print(number)   # 1 2 3 4 5, one per iteration

Calling count_up_to(5) doesn't run the function body at all — it returns a generator object. Each call to next() (which a for loop does implicitly) resumes the function right where it last left off, runs until the next yield, and pauses again:

gen = count_up_to(3)
print(next(gen))   # 1
print(next(gen))   # 2
print(next(gen))   # 3
print(next(gen))   # StopIteration — the generator is exhausted

This matters for memory: processing a billion-line file with a generator holds one line in memory at a time; building a list of a billion lines first would not.

def read_large_file(path):
    with open(path) as f:
        for line in f:
            yield line.strip()

# Only one line is ever in memory at once, no matter the file's size
for line in read_large_file("huge_log.txt"):
    if "ERROR" in line:
        print(line)

A generator expression — the lazy sibling of a list comprehension — uses parentheses instead of brackets:

squares = (n ** 2 for n in range(1_000_000))   # nothing computed yet
total = sum(squares)                            # computed lazily, one value at a time

Decorators#

A decorator is a function that wraps another function to add behavior — logging, timing, caching, access control — without modifying the original function's code:

import time
from functools import wraps

def timed(func):
    @wraps(func)   # preserves func's original name/docstring for introspection
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = func(*args, **kwargs)
        elapsed = time.perf_counter() - start
        print(f"{func.__name__} took {elapsed:.4f}s")
        return result
    return wrapper

@timed
def slow_square(n):
    time.sleep(0.1)
    return n * n

print(slow_square(5))
# slow_square took 0.1002s
# 25

@timed above def slow_square is exactly equivalent to writing slow_square = timed(slow_square). Once decorated, every call to slow_square(...) actually calls wrapper(...), which runs your extra logic around the original function.

Decorators can also take their own arguments by adding one more layer of nesting:

def retry(times):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(1, times + 1):
                try:
                    return func(*args, **kwargs)
                except ConnectionError:
                    if attempt == times:
                        raise
                    print(f"Retrying ({attempt}/{times})...")
        return wrapper
    return decorator

@retry(times=3)
def fetch_data():
    ...  # something that might raise ConnectionError

Context managers (with)#

The with statement guarantees cleanup code runs — even if an exception is raised — by relying on a context manager: any object implementing __enter__ and __exit__.

with open("data.txt") as f:
    contents = f.read()
# f.close() is called automatically here, even if read() raised an exception

Writing your own context manager means implementing both dunder methods:

class Timer:
    def __enter__(self):
        import time
        self.start = time.perf_counter()
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        import time
        elapsed = time.perf_counter() - self.start
        print(f"Elapsed: {elapsed:.4f}s")
        return False   # False means: don't suppress exceptions

with Timer():
    total = sum(range(10_000_000))

For simple cases, contextlib.contextmanager turns a generator function into a context manager without a class at all:

from contextlib import contextmanager

@contextmanager
def timer():
    import time
    start = time.perf_counter()
    yield                              # code inside the `with` block runs here
    print(f"Elapsed: {time.perf_counter() - start:.4f}s")

with timer():
    total = sum(range(10_000_000))

The GIL — an honest explanation#

CPython (the standard Python implementation) has a Global Interpreter Lock (GIL): a single mutex that ensures only one thread executes Python bytecode at any given instant, even on a multi-core machine.

Why it exists: CPython manages memory with reference counting, and the GIL makes that reference counting thread-safe without needing fine-grained locks scattered throughout the interpreter's internals. It's a pragmatic trade-off that has kept single-threaded Python fast and the C API simple for decades.

What it means in practice:

• CPU-bound work (heavy number crunching in pure Python) does not speed up by adding more threading threads — they still take turns on one core, fighting over the GIL.
• I/O-bound work (network calls, disk reads, waiting on a database) does benefit from threading, because a thread releases the GIL while it's blocked waiting on I/O, letting another thread run.
• The GIL only affects a single process. multiprocessing sidesteps it entirely by running separate Python processes, each with its own interpreter and its own GIL — true parallelism, at the cost of higher memory use and slower inter-process communication.
• asyncio sidesteps the problem differently: a single thread cooperatively switches between many async tasks during their await points (I/O waits), achieving high I/O concurrency without threads or the GIL contention at all.

import asyncio

async def fetch(name, delay):
    await asyncio.sleep(delay)   # simulates a network call
    print(f"Fetched {name}")

async def main():
    await asyncio.gather(
        fetch("users", 1),
        fetch("orders", 1),
        fetch("products", 1),
    )
    # all three run concurrently — total time ≈ 1s, not 3s

asyncio.run(main())

A newer CPython feature (3.13+, still experimental as of this writing) allows building Python without the GIL ("free-threaded" builds) — worth being aware of, but not yet the default or production-standard configuration.

Common mistakes#

• Reaching for threading to speed up a CPU-bound loop and being confused when it's no faster (sometimes slower, due to context-switching overhead) — that's a multiprocessing job.
• Forgetting a decorator changes __name__/__doc__ unless you use functools.wraps, which breaks introspection and debugging tools that rely on function metadata.
• Iterating over a generator twice, expecting it to restart — once exhausted, a generator is done; you need to call the generator function again to get a fresh one.

Interview questions#

Q: What's the practical difference between a list comprehension and a generator expression? A list comprehension ([x for x in ...]) builds the entire list in memory immediately. A generator expression ((x for x in ...)) produces values lazily, one at a time, on demand — far more memory-efficient for large or unbounded sequences, at the cost of only being iterable once.

Q: Does the GIL mean Python can't do anything in parallel? No — it means a single CPython process can't run Python bytecode on more than one core at once. True CPU parallelism is still achievable via multiprocessing (separate processes, each with its own GIL) or by using C-extension libraries like NumPy, which release the GIL during heavy native computation.