When we talk about Python Generators, we're talking about one of the most powerful and elegant features in the language. Generators allow you to create data sequences on demand, without needing to store everything in memory at once. This is especially useful when working with large data volumes or infinite streams.
In this complete guide, you'll learn everything from basics to advanced generator techniques in Python, with practical examples you can apply immediately in your projects.
What Are Python Generators?
Generators are special functions that use the yield keyword to return values lazily. Unlike regular functions that return all values at once with return, generators pause execution and return one value at a time, maintaining state between calls.
The main advantage of generators is memory efficiency. While a normal list loads all elements into memory, a generator produces each element only when requested. This is crucial for processing large files, data streams, or infinite sequences.
Difference Between Regular Functions and Generators
Let's understand the difference in practice:
# Regular function - returns everything at once
def regular_function(n):
result = []
for i in range(n):
result.append(i * 2)
return result
# Generator - returns on demand
def simple_generator(n):
for i in range(n):
yield i * 2
# Using regular function
print(regular_function(5)) # [0, 2, 4, 6, 8] - everything in memory
# Using generator
gen = simple_generator(5)
print(next(gen)) # 0
print(next(gen)) # 2
print(next(gen)) # 4Notice that the generator doesn't execute any lines when called - it only returns an iterator object. Each next() call executes the code until the next yield.
Source: Python Documentation - Generators
Creating Your First Generator
The syntax of a generator is very similar to a regular function, but with one crucial difference: use yield instead of return.
def counter(max_value):
"""Generator that counts from 0 to max_value-1"""
count = 0
while count < max_value:
yield count
count += 1
# Using the generator
for number in counter(5):
print(f"Count: {number}")This code produces:
Count: 0 Count: 1 Count: 2 Count: 3 Count: 4
Now you must be wondering: "This looks a lot like list comprehension". And you're right! Generators are a related concept, but with an important difference.
Source: Real Python - Introduction to Python Generators
Generator Expressions
Just like list comprehensions, generators have a more concise version: generator expressions. The syntax is almost identical, but uses parentheses instead of brackets.
# List comprehension - creates full list in memory
my_list = [x**2 for x in range(1000000)] # Memory problem!
# Generator expression - creates lazy iterator
gen = (x**2 for x in range(1000000)) # Efficient!
# Using the generator
print(sum(x**2 for x in range(10))) # 285Generator expressions are perfect when you need a simple sequence and don't need to store all values. Use them in functions that expect iterables, like sum(), max(), min(), etc.
Source: W3Schools - Python Generators
Why Use Generators?
Now that you know how to create generators, let's understand why they're so important:
1. Memory Efficiency
The most obvious benefit of generators is memory savings. Let's compare:
import sys
# List of 1000 numbers
my_list = [i for i in range(1000)]
print(f"List size: {sys.getsizeof(my_list)} bytes")
# Equivalent generator
gen = (i for i in range(1000))
print(f"Generator size: {sys.getsizeof(gen)} bytes")The generator takes up much less memory because it doesn't need to store all values - it generates them on demand.
2. Better Performance
For operations that don't need all data at once, generators are faster because there's no upfront cost of creating the entire data structure.
import time
# Measuring time - list vs generator
start = time.time()
list_sum = sum([i for i in range(10000000)])
list_time = time.time() - start
start = time.time()
gen_sum = sum(i for i in range(10000000))
gen_time = time.time() - start
print(f"List: {list_time:.4f}s")
print(f"Generator: {gen_time:.4f}s")3. Representing Infinite Streams
Generators can represent infinite sequences, something impossible with regular lists:
def fibonacci_numbers():
"""Infinite generator of Fibonacci numbers"""
a, b = 0, 1
while True:
yield a
a, b = b, a + b
fib = fibonacci_numbers()
for i in range(10):
print(next(fib), end=" ") # 0 1 1 2 3 5 8 13 21 34This generator can run forever because it doesn't store values - it calculates each one on demand.
Source: Stack Overflow - Python Generator Use Cases
Advanced Generators
Delegation with yield from
The yield from allows you to delegate execution to another generator, creating more modular code:
def generate_numbers():
"""Main generator that delegates to others"""
yield from range(5) # 0, 1, 2, 3, 4
yield from ['a', 'b', 'c'] # a, b, c
yield from (x*2 for x in range(3)) # 0, 2, 4
for item in generate_numbers():
print(item, end=" ") # 0 1 2 3 4 a b c 0 2 4This technique is extremely useful when you need to combine multiple generators or iterators.
Sending Values to Generators
Generators can receive values through the send() method:
def state_manager():
"""Generator that receives values and continues"""
value = yield "Starting..."
yield f"Received: {value}"
yield "Finished"
gen = state_manager()
print(next(gen)) # Starting...
print(gen.send("Hello")) # Received:Hello
print(next(gen)) # FinishedThis functionality allows you to create generators with complex internal state, useful for implementing state machines or coroutines.
Source: GeeksforGeeks - Generators in Python
Generator Pipelines
One of the most powerful applications of generators is creating data processing pipelines, similar to Unix pipes:
def read_numbers():
"""Simulates reading numbers"""
for i in range(1, 101):
yield i
def filter_evens(numbers):
"""Filters only even numbers"""
for num in numbers:
if num % 2 == 0:
yield num
def double_values(numbers):
"""Doubles each value"""
for num in numbers:
yield num * 2
def limit(numbers, n):
"""Limits the number of results"""
for i, num in enumerate(numbers):
if i >= n:
break
yield num
# Creating the pipeline
pipeline = limit(double_values(filter_evens(read_numbers())), 10)
print("Pipeline result:", list(pipeline))
# [4, 8, 12, 16, 20, 24, 28, 32, 36, 40]This pattern is extremely efficient for processing large data volumes because each step only processes data as the next step requests it.
Generators vs Iterators
It's important not to confuse generators with iterators. While generators are a way to create iterators, not every iterator is a generator:
# Generator - uses the yield keyword
def gen():
yield 1
yield 2
yield 3
# Manual iterator - uses __iter__ and __next__
class Iterator:
def __init__(self):
self.value = 0
def __iter__(self):
return self
def __next__(self):
self.value += 1
if self.value > 3:
raise StopIteration
return self.valueThe advantage of generators is that they're much simpler to write - you don't need to create a complete class!
Source: Programiz - Python Generator
Real-World Use Cases
1. Processing Large Files
def read_large_file(path):
"""Reads file line by line without loading everything into memory"""
with open(path, 'r') as file:
for line in file:
yield line.strip()
# Efficient use with GB-sized files
for line in read_large_file('huge_file.txt'):
process(line)2. Web Scraping Pagination
def fetch_pages(base_url, pages):
"""Simulates fetching multiple pages"""
for page in range(1, pages + 1):
data = fetch(f"{base_url}?page={page}")
yield data3. Real-Time Data Processing
def process_sensor_stream():
"""Simulates reading sensors in real-time"""
import random
while True:
yield random.randint(0, 100)
# Processes data as it arrives
sensor = process_sensor_stream()
for reading in sensor:
if reading > 80:
print(f"Alert: {reading}")Throwing Exceptions in Generators
Generators also support exception handling:
def generator_with_exception_handling():
"""Generator with try-except"""
try:
for i in range(10):
yield i
except ValueError:
yield "Error caught!"
# You can throw exceptions into generators
gen = generator_with_exception_handling()
for _ in range(5):
print(next(gen))
gen.throw(ValueError) # Throws exception into generatorBest Practices with Generators
Now here are some best practices when working with generators:
- Use generators for large data - When you have millions of records, generators are essential.
- Document your generator - Write what it produces and what its parameters are.
- Avoid side effects inside generators - They work better when they're pure.
- Combine with itertools - The standard library offers many complementary tools.
- Use generator expressions when possible - They're more concise for simple cases.
from itertools import islice, count
# Advanced example with itertools
def prime_numbers():
"""Generator of prime numbers"""
primes = []
for n in count(2):
if all(n % p != 0 for p in primes[:int(n**0.5)+1]):
primes.append(n)
yield n
# Get the first 10 primes
print(list(islice(prime_numbers(), 10))) # [2, 3, 5, 7, 11, 13, 17, 19, 23, 29]Source: Python itertools Documentation
Conclusion
Python Generators are a fundamental tool for any Python developer working with data. They offer an elegant and efficient way to create iterators, process large data volumes, and implement functional programming patterns.
Mastering generators opens doors to writing cleaner, more efficient, and more scalable code. Whether you're processing GB-sized files, implementing data pipelines, or creating infinite streams, generators are the right solution.
Keep learning with the free guides at Universo Python: explore advanced decorators and generators, asynchronous programming, and much more to take your code to the next level!