If you have ever used a for loop in Python, you have already used iterators. But do you know what happens behind the scenes? Iterators are one of Python's core pillars, enabling data structures like lists, tuples, dictionaries, and sets to be traversed in a uniform and memory-efficient way. Understanding how they work is essential for writing idiomatic and performant Python code.
In this complete guide on Python iterators, you will learn everything from the iteration protocol to building custom iterators, exploring the powerful itertools module and understanding the crucial differences between iterators and generators. Every concept comes with practical examples to solidify your understanding.
What Are Iterators in Python?
An iterator is an object that implements the iteration protocol, which consists of two methods: __iter__() and __next__(). The __iter__() method returns the iterator object itself, while __next__() returns the next value in the sequence. When there are no more items, __next__() raises a StopIteration exception to signal the end of the iteration.
In practice, this means any object that can be looped over with a for loop is an iterable that produces an iterator under the hood. The for loop is essentially syntactic sugar that calls iter() on the object and then repeatedly calls next() until StopIteration is raised.
# What happens internally when you write:
for item in [1, 2, 3]:
print(item)
# Is equivalent to:
iterator = iter([1, 2, 3]) # Calls __iter__() internally
while True:
try:
item = next(iterator) # Calls __next__() internally
print(item)
except StopIteration:
breakSource: Official Python Documentation - Iterators
Iterables vs Iterators: What's the Difference?
This is one of the most common questions among Python developers. Even though they are closely related, iterables and iterators are distinct concepts:
- Iterable: Any object that can be iterated over, meaning it implements
__iter__()(or__getitem__()). Lists, tuples, strings, dictionaries, and sets are all iterables. An iterable can be used in aforloop and can be converted into an iterator with theiter()function. - Iterator: An object that implements both
__iter__()and__next__(). Every iterator is an iterable (since it implements__iter__()), but not every iterable is an iterator.
The most important practical difference is that an iterator can only be traversed once. Once all elements are consumed, the iterator is exhausted and cannot be restarted. An iterable, on the other hand, can generate fresh iterators as many times as needed.
my_list = [1, 2, 3] # my_list is an iterable, NOT an iterator
iterator1 = iter(my_list) # iterator1 is an iterator
iterator2 = iter(my_list) # iterator2 is a fresh, independent iterator
print(next(iterator1)) # 1
print(next(iterator1)) # 2
print(next(iterator1)) # 3
# print(next(iterator1)) # StopIteration - iterator exhausted
print(next(iterator2)) # 1 - iterator2 is still at the beginningSource: Python Glossary - Iterable
The Iteration Protocol in Detail
The iteration protocol is the foundation of Python's entire iteration system. Let's break down each component:
The __iter__() Method
This method must return an iterator object. For iterables like lists, it returns a specialized iterator object. For iterators, it simply returns self (the object itself).
The __next__() Method
This method must return the next element in the sequence. When there are no more elements, it must raise StopIteration. This signals the for loop that the iteration has finished.
The StopIteration Exception
StopIteration is Python's way of communicating that an iterator has been fully consumed. While you can use it directly in your code, it is more commonly handled indirectly through for loops.
Source: PEP 234 - Iterators
Creating Custom Iterators
Building your own iterators is a valuable Python skill. You can encapsulate complex iteration logic into clean, reusable classes. Let's see how it's done.
Even Numbers Iterator
class EvenNumbers:
"""Iterator that returns even numbers up to a limit."""
def __init__(self, limit):
self.limit = limit
self.value = 0
def __iter__(self):
return self
def __next__(self):
if self.value > self.limit:
raise StopIteration
result = self.value
self.value += 2
return result
# Using the custom iterator
for even in EvenNumbers(10):
print(even) # 0, 2, 4, 6, 8, 10Fibonacci Iterator
class Fibonacci:
"""Iterator that generates the Fibonacci sequence."""
def __init__(self, max_count):
self.max_count = max_count
self.a, self.b = 0, 1
self.counter = 0
def __iter__(self):
return self
def __next__(self):
if self.counter >= self.max_count:
raise StopIteration
result = self.a
self.a, self.b = self.b, self.a + self.b
self.counter += 1
return result
fib = Fibonacci(10)
print(list(fib)) # [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]Resettable Iterable
Unlike a pure iterator, we can create classes that are iterables (not iterators) and produce a fresh iterator every time iter() is called, allowing multiple traversals:
class TextWords:
"""Iterable that produces a new iterator for each traversal."""
def __init__(self, text):
self.text = text
def __iter__(self):
return WordsIterator(self.text)
class WordsIterator:
def __init__(self, text):
self.words = text.split()
self.index = 0
def __iter__(self):
return self
def __next__(self):
if self.index >= len(self.words):
raise StopIteration
result = self.words[self.index]
self.index += 1
return result
sentence = TextWords("Python is a powerful language")
for w in sentence:
print(w) # Python, is, a, powerful, language
# Can traverse again
for w in sentence:
print(w.upper()) # PYTHON, IS, A, POWERFUL, LANGUAGELearn more about magic methods like __iter__ and __next__ in our complete guide on Python magic methods.
Source: Real Python - Python Iterators and Iterables
Iterators in Native Data Structures
Each data structure in Python implements iteration differently. Let's explore how iterators work for the most common types:
Lists and Tuples
They produce iterators that traverse elements in insertion order. The list iterator maintains an internal index that advances with each next() call.
my_list = [10, 20, 30]
it = iter(my_list)
print(next(it)) # 10
print(next(it)) # 20Dictionaries
By default, iterating over a dictionary traverses its keys. You can use .values() for values and .items() for key-value pairs.
d = {'a': 1, 'b': 2, 'c': 3}
for key in d: # Keys
print(key)
for value in d.values(): # Values
print(value)
for k, v in d.items(): # Key and value
print(f"{k}: {v}")Strings
Strings are iterables and produce an iterator that traverses each character individually, including spaces and special characters.
for char in "Python":
print(char) # P, y, t, h, o, nFiles
Files are iterable in Python! Each call to next() returns one line, making large file reads extremely memory-efficient.
with open("data.txt", "r") as file:
for line in file: # Reads one line at a time
print(line.strip())Source: Python Wiki - Iterator
Working with itertools
The itertools module is a standard library that provides functions for creating advanced iterators. It is an indispensable tool for any Python developer working with iteration. Let's explore the most useful functions:
count() - Infinite Counter
from itertools import count
for i in count(5): # 5, 6, 7, 8, 9, ...
if i > 10:
break
print(i)cycle() - Infinite Cycle
from itertools import cycle
colors = cycle(["red", "blue", "green"])
for _ in range(6):
print(next(colors)) # red, blue, green, red, blue, greenchain() - Chaining Iterators
from itertools import chain
list1 = [1, 2, 3]
list2 = [4, 5, 6]
for item in chain(list1, list2):
print(item) # 1, 2, 3, 4, 5, 6islice() - Iterator Slicing
from itertools import islice
# Grab the first 5 elements from an infinite counter
for i in islice(count(10), 5):
print(i) # 10, 11, 12, 13, 14product(), permutations(), combinations()
These functions generate powerful mathematical iterators for Cartesian products, permutations, and combinations:
from itertools import product, permutations, combinations
# Cartesian product
print(list(product([1, 2], ["a", "b"])))
# [(1, 'a'), (1, 'b'), (2, 'a'), (2, 'b')]
# Permutations
print(list(permutations([1, 2, 3], 2)))
# [(1, 2), (1, 3), (2, 1), (2, 3), (3, 1), (3, 2)]
# Combinations
print(list(combinations([1, 2, 3], 2)))
# [(1, 2), (1, 3), (2, 3)]Source: Official Documentation - itertools
Iterators vs Generators: When to Use Each
This is a frequent question in technical interviews and daily practice. Both iterators and generators serve to produce sequences of data on demand, but each has its place.
Generators (using yield) are a simplified way to create iterators. While a custom iterator requires a class with __iter__() and __next__(), a generator is written as a regular function using yield. For most cases, generators are the more practical choice.
Custom iterators are more appropriate when you need:
- Complex state to maintain between iterations
- Additional methods beyond iteration
- A rich abstraction with specific behavior
- State reinitialization or reset (via a separate iterable and iterator)
# When to use a generator (simpler):
def squares(n):
for i in range(n):
yield i ** 2
# When to use a custom iterator (more control):
class Squares:
def __init__(self, n, prefix=""):
self.n = n
self.i = 0
self.prefix = prefix
def __iter__(self):
return self
def __next__(self):
if self.i >= self.n:
raise StopIteration
result = f"{self.prefix}{self.i ** 2}"
self.i += 1
return result
def reset(self):
self.i = 0Check out our complete guide on Python Generators to dive deeper into this powerful alternative to iterators.
Source: GeeksforGeeks - Iterators in Python
Built-in Functions That Work with Iterators
Python offers several native functions that operate directly on iterators:
numbers = [1, 2, 3, 4, 5]
# map - applies a function to each element
doubled = map(lambda x: x * 2, numbers)
print(list(doubled)) # [2, 4, 6, 8, 10]
# filter - filters elements
evens = filter(lambda x: x % 2 == 0, numbers)
print(list(evens)) # [2, 4]
# zip - groups elements from multiple iterables
names = ["Alice", "Bob", "Eve"]
ages = [25, 30, 28]
for name, age in zip(names, ages):
print(f"{name} is {age} years old")
# enumerate - enumerates elements with an index
for i, name in enumerate(names, start=1):
print(f"{i}: {name}")
# reversed - iterates in reverse order
for num in reversed([1, 2, 3]):
print(num) # 3, 2, 1
# sorted - iterates in sorted order
for num in sorted([3, 1, 2]):
print(num) # 1, 2, 3
# any and all - check conditions on iterators
print(any(x > 3 for x in [1, 2, 3, 4])) # True
print(all(x > 0 for x in [1, 2, 3, 4])) # TrueSource: Programiz - Python Iterators
Best Practices with Iterators
Working efficiently with iterators requires attention to a few key points:
1. Single-Consumption Awareness
Remember: iterators can only be consumed once. If you need to traverse the data multiple times, convert the iterator to a list (if memory allows) or create an iterable that produces fresh iterators.
# Problem: iterator consumed after first loop
it = iter([1, 2, 3])
for i in it:
print(i)
# Second loop won't run - it is exhausted
for i in it:
print(i)
# Solution 1: convert to a list
my_list = list(iter([1, 2, 3]))
# Solution 2: create an iterable that yields new iterators
class MyNumbers:
def __iter__(self):
return iter([1, 2, 3])2. Efficiency with Large Files
Take advantage of the fact that files are iterable to process large volumes without loading everything into memory:
with open("large_file.csv") as f:
for line in f:
process(line) # One line at a time in memory3. Generator Expressions vs List Comprehensions
Use generator expressions (with parentheses) instead of list comprehensions (with brackets) when you do not need all values at once. The memory savings can be enormous.
# List comprehension - creates the entire list in memory
squares_list = [x**2 for x in range(1000000)]
# Generator expression - produces on demand
squares_gen = (x**2 for x in range(1000000))4. Iterator Chaining
You can combine multiple iterators and built-in functions to create elegant and efficient processing pipelines:
data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
pipeline = (
x
for x in data
if x % 2 == 0 # filter evens
)
pipeline = (x * 2 for x in pipeline) # double
pipeline = (f"Number: {x}" for x in pipeline) # format
for item in pipeline:
print(item)
# Number: 4, Number: 8, Number: 12, Number: 16, Number: 20Source: Python Functional Programming HOWTO
Conclusion
Python iterators are a fundamental concept that appears in virtually every Python codebase, often without you even noticing. From simple for loops to complex data processing pipelines, the iteration protocol is the backbone that makes it all possible.
In this guide, you learned:
- What iterators are and how they differ from iterables
- The iteration protocol with
__iter__(),__next__(), andStopIteration - How to build custom iterators using classes
- How to leverage the
itertoolsmodule for advanced iteration - The key differences between iterators and generators
- Best practices for working with iterators efficiently
Mastering iterators will make your Python code more idiomatic, efficient, and elegant. Practice building your own iterators and explore the itertools module — these are tools you will reach for constantly in real-world projects.