Python Celery: Reliable Background Tasks with Redis

Celery is a distributed task queue that moves work out of the request-response path. Instead of making a user wait for an email, report, or image transformation, the application publishes a message and a worker runs the function. The direct rule is simple: use Celery when work can finish later, needs retries, or must run across processes and machines. Do not add it merely to call a short function.

Celery coordinates producers, brokers, queues, and consumers; it is not a database. This guide uses Redis as the message broker and, optionally, the result backend. Read our Python and Redis guide for the underlying data structures, and use the official Celery documentation as the version-specific reference.

Architecture and the first decisions

An HTTP handler validates input, commits required state, and submits a task with .delay() or .apply_async(). The broker stores the message until a worker reserves it. A result backend may record status and return values. This architecture improves API latency but introduces eventual consistency and distributed failure modes. Return a job identifier and expose polling, a webhook, or events when users need progress.

Celery fits work lasting seconds or minutes, scheduled jobs, isolated queues, and recoverable processing. For concurrent I/O inside one process, Python async/await addresses a different need and does not make work durable. Framework-native background hooks are reasonable for disposable, very small work, but they usually disappear with the web process.

Configure Celery with Redis

Install Celery's Redis extras, then run Redis locally or through a managed service. In production, enable authentication, TLS where applicable, persistence chosen for your loss tolerance, and memory monitoring. Never expose Redis directly to the internet. The official Redis documentation covers deployment and persistence choices.

pip install "celery[redis]"
celery -A tasks worker --loglevel=INFO

from celery import Celery
app = Celery(
"myapp",
broker="redis://localhost:6379/0",
backend="redis://localhost:6379/1",
)
app.conf.update(
task_serializer="json",
accept_content=["json"],
result_serializer="json",
timezone="UTC",
task_track_started=True,
task_time_limit=300,
task_soft_time_limit=270,
worker_prefetch_multiplier=1,
)

Load URLs from environment variables rather than source control. JSON avoids arbitrary Python object deserialization. Result storage consumes memory and requires expiration; set task_ignore_result for fire-and-forget tasks. Our Python Docker guide shows how to package web and worker processes reproducibly.

Retries that help recovery

Retry transient failures such as timeouts, temporary unavailability, and rate limits. Invalid input and rejected business rules are permanent failures; retrying them only delays useful jobs. Exponential backoff, jitter, and a retry cap prevent every worker from hitting a recovering dependency at once.

import requests
@app.task(bind=True, max_retries=5)
def notify_customer(self, order_id: int) -> None:
try:
response = requests.post(
"https://api.example.com/notifications",
json={"order_id": order_id},
timeout=(3, 10),
)
except (requests.Timeout, requests.ConnectionError) as error:
delay = min(2 ** self.request.retries, 600)
raise self.retry(exc=error, countdown=delay)
if response.status_code in {429, 500, 502, 503, 504}:
    retry_after = response.headers.get("Retry-After")
    delay = int(retry_after) if retry_after and retry_after.isdigit() else min(2 ** self.request.retries, 600)
    raise self.retry(exc=requests.HTTPError(response=response), countdown=delay)

response.raise_for_status()  # Permanent 4xx responses fail without retry.</code></pre>
Set network timeouts even when the task has a hard time limit. Do not catch every Exception and retry blindly. Log the reason and preserve the original exception. Celery's official retry guide explains manual retry and automatic retry behavior.
Idempotency is mandatory
Task queues normally provide at-least-once delivery. A worker can complete an external effect, crash before acknowledgement, and receive the same message again. Charging a card, issuing a coupon, or incrementing a balance must therefore tolerate duplicate execution.
Derive a stable idempotency key such as charge:order:123. Enforce uniqueness in the database and update state atomically where possible. When an external API supports idempotency keys, pass the same key on every attempt. A Redis lock can reduce overlap, but it does not replace a durable unique constraint.
@app.task(bind=True, acks_late=True)
def issue_invoice(self, order_id: int) -> str:
    invoice = Invoice.get_or_create_for_order(order_id)
    if invoice.status == "issued":
        return invoice.number
number = tax_api.issue(
    order_id=order_id,
    idempotency_key=f"invoice-order-{order_id}",
)
invoice.mark_issued(number)
return number</code></pre>
acks_late=True acknowledges after execution, allowing unfinished work to be recovered when a process dies, but duplicate execution becomes more likely. Enable it only for idempotent tasks. Send IDs and small primitives in messages, not serialized ORM objects that may be stale by execution time.
Queues, concurrency, and backpressure
Separate workloads by behavior, such as emails, reports, and payments. A large report then cannot starve payment confirmations. Route tasks with task_routes and assign dedicated workers. CPU-bound tasks need process capacity; I/O-bound tasks may use greater concurrency only if databases and remote services can absorb it. Our Python threading and multiprocessing comparison explains that distinction.
High prefetch improves throughput for tiny tasks, but a worker may reserve too many long jobs and create unfairness. A multiplier of one is a sensible starting point for slow workloads. Queue length also needs backpressure: reject excessive submissions, batch responsibly, or degrade optional work before Redis memory becomes the limit.
Production observability
Measure queue depth and oldest-message age, task duration, throughput, retry rate, terminal failures, and worker availability. Structured logs should include task ID, task name, attempt, queue, and the business entity ID. Never log credentials or sensitive payloads. Propagate a request correlation ID into task headers so traces connect web and worker activity.
Celery events and Flower are useful for live inspection, but historical metrics and alerts remain necessary. Combine structured Python logging with distributed tracing and an operational path for exhausted jobs. Alert on impact, such as the oldest job exceeding its service objective, rather than alerting whenever a queue is nonempty.
Common mistakes

Publishing before the database transaction commits, so the worker cannot see the referenced row.
Calling remote systems without timeouts or retrying permanent errors forever.
Assuming exactly-once delivery and omitting idempotency controls.
Sending large files in messages instead of storing them and passing a reference.
Mixing fast and heavy jobs in one queue without time limits.
Deploying an incompatible task signature while old messages remain queued.

Publish after commit. If the gap between database commit and broker publication is unacceptable, use a transactional outbox. Version message schemas and keep deployments compatible while queues drain. Unit-test domain functions without Celery, then add integration tests for publication, retry, and duplicates. The pytest guide provides a practical testing foundation.
Frequently asked questions
Does Celery replace async/await?
No. Celery distributes work and can persist messages; async/await schedules concurrent operations inside a process. A system may use both for different layers.
Is Redis enough as a production broker?
It can be when durability, availability, memory, and security match the workload's risk. Consider RabbitMQ when advanced routing or particular messaging semantics are required.
Should every task store its result?
No. Store results only when a consumer reads them. For side effects, business state in the application database is usually the source of truth.
How can retries be tested quickly?
Test domain behavior separately, simulate specific transient exceptions, and assert that retry is requested. Add a small integration suite with a real broker and worker.
Conclusion
A reliable Celery setup is more than calling .delay(). Classify errors, cap retries, make effects idempotent, isolate workloads, control concurrency, and observe delay as well as execution. Start with one bounded task, protected Redis, small messages, and a measurable service objective. Scale workers and routing only after those guarantees are clear.