QueueHandler and QueueListener for Python Logging addresses a recurring problem in Python projects: Send LogRecord objects to a queue and process slow handlers on a dedicated thread. This guide explains the mechanism, provides an executable example, and identifies the boundaries that keep an implementation reliable.
Concept and use case
QueueHandler enqueues records; QueueListener delivers them to handlers outside the thread serving a request.
For the related fundamentals, also read the dictConfig guide. Integration stays simpler when functions receive dependencies and data explicitly instead of relying on global state.
Practical example
import logging
from logging.handlers import QueueHandler, QueueListener
from queue import Queue
queue = Queue(maxsize=10_000)
console = logging.StreamHandler()
listener = QueueListener(queue, console, respect_handler_level=True)
logger = logging.getLogger("app")
logger.addHandler(QueueHandler(queue))
listener.start()
try:
logger.info("request completed")
finally:
listener.stop()Set capacity and a full-queue policy, respect handler levels, and stop the listener during shutdown so queued records drain.
Important decisions
The correct choice depends on the public contract, expected volume, and failure behavior.
Consider concurrency, empty inputs, and partial failures. Document every limit that affects consumers and choose names that express intent.
Common mistakes
A minimal example does not replace bounds, error handling, and observability. An unbounded queue can consume memory. With multiprocessing, sharing a queue with internal multiprocessing logging can recurse or deadlock.
Avoid catching exceptions without context or returning partial output as if it were complete. An explicit failure is usually safer than silently incorrect data.
How to validate
Validate behavior, not only the happy path. Test saturation, exceptions, shutdown, and required ordering. Verify sensitive fields are removed before enqueueing.