Asynchronous programming has revolutionized how we build Python applications. With the introduction of async and await in Python 3.5 (PEP 492), the language gained a powerful tool for writing concurrent code that is both readable and efficient. In this complete guide, you'll master everything from the fundamentals to advanced asynchronous programming techniques.
If you've ever worked with API calls, I/O operations, or needed to process multiple tasks simultaneously, mastering async/await is essential for writing modern, performant Python code.
The Problem with Synchronous Code
In the traditional synchronous paradigm, each operation blocks execution until it completes. This is especially problematic for I/O operations like HTTP requests, file reading, or database queries. While the program waits for a network response, the CPU sits idle, wasting valuable resources.
import time
import requests
def fetch_data(url):
print(f"Fetching {url}")
response = requests.get(url)
return response.status_code
start = time.time()
code1 = fetch_data("https://api.github.com")
code2 = fetch_data("https://api.github.com/users")
code3 = fetch_data("https://api.github.com/repos")
end = time.time()
print(f"Total time: {end - start:.2f}s")
In this example, each request waits for the previous one to finish, resulting in a total time equal to the sum of all latencies. Asynchronous programming solves this by allowing multiple operations to run concurrently.
What Are Coroutines?
A coroutine is a special function that can pause its execution (await) and resume later. Unlike regular functions, coroutines voluntarily yield control at specific points, allowing other tasks to run while waiting.
To define a coroutine, simply use async def instead of def:
async def my_coroutine():
print("Starting...")
await some_operation()
print("Finished!")
The async keyword turns a regular function into a coroutine. When you call a coroutine, it doesn't run immediately; instead, it returns a coroutine object that needs to be scheduled on the event loop. More details on coroutine definitions can be found in the official CPython documentation.
Async and Await in Practice
The real power of async/await lies in executing I/O operations without blocking the main thread. Let's rewrite the previous example using the aiohttp library:
import asyncio
import aiohttp
import time
async def fetch_data(session, url):
print(f"Fetching {url}")
async with session.get(url) as response:
return response.status
async def main():
async with aiohttp.ClientSession() as session:
tasks = [
fetch_data(session, "https://api.github.com"),
fetch_data(session, "https://api.github.com/users"),
fetch_data(session, "https://api.github.com/repos"),
]
results = await asyncio.gather(*tasks)
print(f"Status: {results}")
start = time.time()
asyncio.run(main())
end = time.time()
print(f"Total time: {end - start:.2f}s")
Notice how the total time is now roughly equal to the slowest operation, not the sum of all of them. This is because requests run concurrently. The complete async/await specification is described in PEP 492 — Coroutines with async and await syntax.
The asyncio Module
asyncio is Python's standard library module for asynchronous programming. It provides the event loop, which manages the execution of coroutines, tasks, and callbacks. The event loop is the heart of asynchronous programming, responsible for coordinating when each task should run or pause.
import asyncio
async def greet(name, delay):
await asyncio.sleep(delay)
print(f"Hello, {name}!")
async def main():
task1 = asyncio.create_task(greet("Ana", 2))
task2 = asyncio.create_task(greet("John", 1))
task3 = asyncio.create_task(greet("Maria", 3))
await task1
await task2
await task3
asyncio.run(main())
The asyncio.run() function was introduced in Python 3.7 and drastically simplifies coroutine execution. Before it, you had to manually manage the event loop with loop.run_until_complete(). The full module documentation is available at docs.python.org/3/library/asyncio.html.
Tasks and Futures
A Task is a wrapper around a coroutine that schedules it for execution on the event loop. When you call asyncio.create_task(), the coroutine is automatically scheduled for concurrent execution. A Future, on the other hand, represents a result that will be available in the future — it's a lower-level concept you'll rarely need to use directly.
import asyncio
async def slow_operation(number):
await asyncio.sleep(2)
return number * 2
async def main():
tasks = [
asyncio.create_task(slow_operation(1)),
asyncio.create_task(slow_operation(2)),
asyncio.create_task(slow_operation(3)),
]
results = await asyncio.gather(*tasks)
print(f"Results: {results}")
for i, task in enumerate(tasks):
print(f"Task {i}: done={task.done()}")
asyncio.run(main())
Tasks are fundamental for running multiple asynchronous operations in parallel. The complete Task API can be found in the official asyncio task documentation.
Concurrent Execution with gather and as_completed
asyncio.gather() is the most common tool for running multiple coroutines concurrently. It returns a list of results in the same order as the input coroutines. asyncio.as_completed(), on the other hand, yields results as they finish, regardless of the original order.
import asyncio
async def fetchdata(id, delay):
await asyncio.sleep(delay)
return f"Data from resource {id_}"
async def main():
coros = [
fetch_data(1, 3),
fetch_data(2, 1),
fetch_data(3, 2),
]
results = await asyncio.gather(*coros)
print(f"Gather: {results}")
for coro in asyncio.as_completed(
[fetch_data(4, 3), fetch_data(5, 1), fetch_data(6, 2)]
):
result = await coro
print(f"Completed: {result}")
asyncio.run(main())
For more complex operations, asyncio.wait() lets you control when to return (when all finish, when the first finishes, or when the first fails). These patterns are widely used in real-world applications and are documented in the concurrent task execution section.
Timeouts and Error Handling
In real-world applications, handling operations that may take longer than expected is crucial. asyncio.timeout() (available since Python 3.11) and asyncio.wait_for() let you set time limits on your async operations.
import asyncio
async def slow_operation():
await asyncio.sleep(10)
return "Done!"
async def main():
try:
async with asyncio.timeout(3):
result = await slow_operation()
print(result)
except TimeoutError:
print("Operation exceeded the time limit!")
try:
result = await asyncio.wait_for(
slow_operation(), timeout=2
)
print(result)
except asyncio.TimeoutError:
print("Timeout with wait_for!")
asyncio.run(main())
Error handling in async code follows the same try/except pattern as synchronous Python, but it's important to note that exceptions in tasks must be caught explicitly. PEP 525 — Asynchronous Generators defines how async generators behave with respect to errors.
Async HTTP Requests with aiohttp
One of the most common uses of async/await is making concurrent HTTP requests. The aiohttp library is the standard choice for async HTTP in Python, offering full support for both HTTP clients and servers.
import asyncio
import aiohttp
async def query_api(session, url):
async with session.get(url) as resp:
data = await resp.json()
return data
async def main():
urls = [
"https://api.github.com",
"https://api.github.com/users/python",
"https://api.github.com/repos/python/cpython",
]
async with aiohttp.ClientSession() as session:
tasks = [query_api(session, url) for url in urls]
results = await asyncio.gather(*tasks)
for url, data in zip(urls, results):
print(f"{url}: {len(data)} fields")
asyncio.run(main())
aiohttp also supports persistent sessions, SSL connections, cookies, and authentication. The full documentation is available at docs.aiohttp.org. If you prefer a more modern alternative, httpx also offers async support with an API similar to requests.
Working with Async Files
File read and write operations also benefit from asynchronous programming, especially when dealing with multiple large files. The aiofiles library provides an async API for file I/O operations.
import asyncio
import aiofiles
async def process_file(path):
async with aiofiles.open(path, mode='r') as file:
content = await file.read()
return f"{path}: {len(content)} characters"
async def main():
files = ["data1.txt", "data2.txt", "data3.txt"]
tasks = [process_file(f) for f in files]
results = await asyncio.gather(*tasks)
for result in results:
print(result)
asyncio.run(main())
aiofiles is particularly useful for web applications that need to serve large files or handle uploads without blocking the event loop. More information can be found on the official aiofiles GitHub repository.
Async Generators and Async Comprehensions
Python also supports asynchronous generators (PEP 525) and asynchronous comprehensions (PEP 530), which let you produce and consume sequences asynchronously. These are useful for processing data streams or paginating API results.
import asyncio
async def generate_numbers():
for i in range(5):
await asyncio.sleep(0.5)
yield i
async def main():
async for number in generate_numbers():
print(f"Received: {number}")
results = [x async for x in generate_numbers()]
print(f"Results: {results}")
asyncio.run(main())
Async generators implement the __aiter__ and __anext__ protocols and are fundamental for streaming libraries and real-time data processing. The full specification is available in PEP 530 — Asynchronous Comprehensions.
Best Practices with Async/Await
Writing efficient async code requires attention to a few key principles:
1. Use asyncio.run() instead of managing the event loop manually
asyncio.run() handles event loop creation and teardown automatically, plus it manages resource cleanup.
# Correct
asyncio.run(main())
Incorrect (old style)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()
2. Avoid mixing sync and async code unnecessarily
Calling synchronous functions inside coroutines blocks the event loop. If you need to run sync code, use asyncio.to_thread() to run it in a separate thread.
import asyncio
import requests
async def fetch_data_wrong():
This BLOCKS the event loop!
response = requests.get("https://api.github.com")
return response.json()
async def fetch_data_correct():
This does NOT block the event loop
loop = asyncio.get_running_loop()
response = await loop.run_in_executor(
None, requests.get, "https://api.github.com"
)
return response.json()
3. Always use async context managers for resources
HTTP sessions, database connections, and files should be managed with async with to ensure proper resource cleanup.
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
return await response.json()
4. Watch out for too many concurrent tasks
Creating thousands of tasks simultaneously can overwhelm the event loop. Use asyncio.Semaphore to limit the number of concurrent operations.
import asyncio
async def process_with_semaphore():
semaphore = asyncio.Semaphore(10)
async def limited_task(item):
async with semaphore:
await asyncio.sleep(1)
return item * 2
tasks = [limited_task(i) for i in range(100)]
return await asyncio.gather(*tasks)
To dive deeper into concurrency patterns, check out the Async IO in Python guide from Real Python, which offers an excellent introduction with practical examples.
Additionally, if you want to understand how decorators can help you write cleaner async code, check out our complete guide on Python Decorators.
Async/Await in Web Frameworks
Modern Python web frameworks support async/await natively. FastAPI was built from the ground up with async support, while Django (since version 3.1) and Flask (since version 2.0) also offer async view support. Combining async/await with web frameworks allows you to serve thousands of concurrent requests with minimal resources.
To master generators in Python, another essential tool for streaming data processing, check out our article on Python Generators.
Conclusion
Asynchronous programming with async/await has transformed how we write Python for I/O operations. With the asyncio module, libraries like aiohttp and aiofiles, and the best practices outlined in this guide, you're ready to build fast, efficient, and scalable Python applications.
Remember: async/await is not about true parallelism (multiple threads or processes), but about concurrency managed by a single event loop. For CPU-intensive tasks, consider using multiprocessing or libraries like concurrent.futures.
To continue your studies, explore the official asyncio documentation, practice with real-world projects, and experiment with different async libraries. Python's async ecosystem is growing rapidly, and mastering these concepts is a competitive edge in the development market.