Python ยท easy
What are Python generators and when would you use them?
Asked in Data Engineer interviews, in the Python round.
Short answer
Lazy iterators built with yield, used for large or streamed data to save memory.
How to answer it
Definition
A generator is a function that produces values one at a time using yield instead of building and returning a whole collection. Calling it returns an iterator; nothing runs until you iterate, and only one value exists in memory at a time.
Syntax
def squares(n):
for i in range(n):
yield i * i # yield, not return
gen = (x * x for x in range(n)) # generator expression, same ideaExample
def read_large_file(path):
with open(path) as f:
for line in f: # the file object is itself lazy
yield line.rstrip()
total = 0
for line in read_large_file("50gb.csv"):
total += len(line)Output
>>> g = squares(5)
>>> g
<generator object squares at 0x0000021F...>
>>> list(g)
[0, 1, 4, 9, 16]
>>> list(g)
[] # exhausted: a generator runs onceMemory
[x * x for x in range(10_000_000)] # ~400 MB, built immediately
(x * x for x in range(10_000_000)) # ~200 bytes, built lazilyThe list holds every value at once. The generator holds one value and the state needed to compute the next.
Key points
- Constant memory regardless of how much data flows through, which is the whole point.
- Lazy: nothing is computed until iterated, so an unconsumed generator does no work.
- Single use. Once exhausted it is empty, and that surprises people the first time.
- No
len()and no indexing. If you need either, you wanted a list. - Ideal for a pipeline: each stage yields into the next and nothing is ever fully materialised.
FAQ
- Generator or list comprehension? List when you need it more than once, or need
lenor indexing. Generator when streaming. - What does
yield fromdo? Delegates to another iterable, which flattens nested generators neatly. - Are they faster? Not per item. They save memory and give you the first result sooner, which is often what matters.
The 30-second answer
A generator is a function that uses yield instead of return, so it hands back one value at a time and keeps its state between them. Calling it does not run the body; iterating it does.
I use them whenever the data is larger than memory or arrives as a stream: reading a huge file line by line, or chaining processing stages so nothing is ever fully materialised. The memory profile stays flat no matter how many items pass through.
The trade-offs I would mention are that a generator is single use, so iterating twice gives nothing the second time, and that there is no length and no indexing. If I need either of those, I wanted a list.
In the interview
- What they are checking: "lazy, one at a time, constant memory", and that you know the single-pass limitation.
- Common mistake: calling
list()on the generator "to look at it", which loads the thing you were trying not to load.