Retries with Requests: Backoff, Session, and HTTPAdapter addresses a recurring problem in Python projects: Retry transient failures within bounds without duplicating operations or causing a request storm. This guide explains the mechanism, provides an executable example, and identifies the boundaries that keep an implementation reliable.
Concept and use case
Requests does not retry failed connections by default. HTTPAdapter accepts Retry to define counts, statuses, methods, and increasing delays.
For the related fundamentals, also read the Requests guide. Integration stays simpler when functions receive dependencies and data explicitly instead of relying on global state.
Practical example
import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry
retry = Retry(
total=4,
backoff_factor=0.5,
status_forcelist=[429, 502, 503, 504],
allowed_methods={"GET", "HEAD"},
respect_retry_after_header=True,
)
session = requests.Session()
session.mount("https://", HTTPAdapter(max_retries=retry))
response = session.get("https://api.example.com/items", timeout=(2, 10))
response.raise_for_status()Restrict retries to idempotent operations, set a timeout on every attempt, respect Retry-After, and add jitter when many clients may retry together.
Important decisions
The correct choice depends on the public contract, expected volume, and failure behavior.
Consider concurrency, empty inputs, and partial failures. Document every limit that affects consumers and choose names that express intent.
Common mistakes
A minimal example does not replace bounds, error handling, and observability. Retrying POST without an idempotency key can duplicate charges or orders. Unlimited retry multiplies latency and load during an outage.
Avoid catching exceptions without context or returning partial output as if it were complete. An explicit failure is usually safer than silently incorrect data.
How to validate
Validate behavior, not only the happy path. Simulate 429, 503, timeout, and recovery. Assert attempt count, maximum duration, and fast failure for permanent errors.