Generators and lazy iteration

What it is

A function containing yield returns a generator object; each next() runs until the next yield and suspends. Generator expressions (f(x) for x in xs) do the same inline. The itertools module offers building blocks such as islice, chain, groupby and batched that operate on any iterable lazily.

Why it matters

Processing a multi-gigabyte log line by line, paginating an API, or reading a database cursor in chunks all fit in constant memory when each stage yields items instead of returning lists. Laziness also lets a pipeline stop early (islice, any) without computing the rest.

How to apply

• Write processing stages as generators and connect them; materialise with list() only at the end and only if needed.
• Use yield from to delegate to sub-generators.
• Close generators that hold resources (gen.close() or a with block inside the generator) so that finally clauses run.
• Sort or group only after filtering; groupby requires sorted input.

Pitfalls

A generator can be consumed once; re-iterating silently yields nothing. Exceptions inside a generator surface at the consumer's next() call, far from the cause. Mixing eager sorted() into a lazy pipeline forces everything into memory.