Python · easy

What are Python generators and when would you use them?

Asked in Data Engineer interviews, in the Python round.

Short answer

Lazy iterators using yield; use for large/streamed data to save memory.

How to answer it

A generator is a function that yields values one at a time instead of returning a list. Nothing runs until you iterate, and only one item is in memory at once.

def read_records(path):
    with open(path) as f:
        for line in f:
            yield parse(line)

total = sum(r["amount"] for r in read_records("orders.jsonl"))   # never holds the file

Use one when the data is larger than memory, when it arrives over time (a socket, a queue, a paginated API), or when the consumer may stop early, so building the whole list would be wasted work. Generators chain: filter into map into sum, each pulling from the last, which is a pipeline with constant memory.

The costs to mention: a generator is single-pass, so you cannot ask its length or iterate it twice without recreating it; and it is lazy, so an exception inside surfaces where the value is consumed, not where the generator was written. A generator expression, (f(x) for x in xs), is the inline form; square brackets instead of parentheses builds the whole list.

Related questions

Practice this for real