Last Updated on 2024-07-28 by Clay
In Python programming, we often use the requests module for HTTP requests. However, requests can become a bottleneck when connecting frontend and backend services due to its synchronous request handling. Recently, I experienced Kubernetes probe blockages caused by using requests, which led to the unintended deletion of my service container. In such scenarios, httpx might be a more suitable module for asynchronous request handling.
Introduction to httpx
httpx is a HTTP client library that supports both synchronous and asynchronous API calls natively. It can directly work with Python's async and await keywords.
Additionally, it comes with several powerful features such as proxy support, SSL configuration, customizable retry strategies (so you don't have to write them yourself!), and more. In the current development environment, I feel it can almost completely replace the requests module.
Next, we will show how to use httpx, but before that, let's create a mock FastAPI service and the original requests code.
Of course, if you already have existing code you want to modify, you can jump directly to the httpx section below and start modifying your code.
Creating a Mock FastAPI Service
First, install fastapi and uvicorn.
pip install fastapi uvicornThen, we will create a simple service.
from fastapi import FastAPI
from pydantic import BaseModel
from typing import List
app = FastAPI()
class Item(BaseModel):
id: int
name: str
fake_db: List[Item] = []
@app.get("/items/", response_model=List[Item]])
async def read_items():
return fake_db
@app.post("/items/", response_model=Item])
async def create_item(item: Item):
fake_db.append(item)
return itemNext, start the service:
uvicorn app:app --reload
Output:
INFO: Started server process [560791]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
It will start at http://127.0.0.1:8000 by default.
Original requests Syntax
If you haven't installed requests yet (which is unlikely if you're reading this), you can install it using the following command:
pip install requests
After that, you can write Python code to send requests to the APIs.
import requests
# GET
response = requests.get("http://127.0.0.1:8000/api/fake-db/items")
print("GET:", response.json())
# POST
new_item = {"id": 1, "name": "Item 1"}
response = requests.post("http://127.0.0.1:8000/api/fake-db/items", json=new_item)
print("POST:", response.json())
# GET
response = requests.get("http://127.0.0.1:8000/api/fake-db/items")
print("GET:", response.json())
Output:
GET: []
POST: {'id': 1, 'name': 'Item 1'}
GET: [{'id': 1, 'name': 'Item 1'}]
As you can see, everything works fine.
At this point, we have completed setting up the FastAPI service and sending synchronous requests using requests. If we only need to support synchronous requests, we can simply replace all occurrences of requests with httpx, and this will be the synchronous request method for httpx, which is fully compatible with requests syntax. Isn't that great?
However, the strength of httpx lies in its asynchronous requests. Let's now see how to perform asynchronous requests with httpx!
Asynchronous Requests with httpx
The simplest way to make asynchronous requests with httpx involves using the httpx.AsyncClient() class.
import asyncio
import httpx
async def main():
async with httpx.AsyncClient() as client:
# GET
response = await client.get("http://127.0.0.1:8000/api/fake-db/items")
print("GET:", response.json())
# POST
new_item = {"id": 2, "name": "Item 2"}
response = await client.post("http://127.0.0.1:8000/api/fake-db/items", json=new_item)
print("POST:", response.json())
# GET again
response = await client.get("http://127.0.0.1:8000/api/fake-db/items")
print("GET:", response.json())
# Run
asyncio.run(main())
Output:
GET: [{'id': 1, 'name': 'Item 1'}]
POST: {'id': 2, 'name': 'Item 2'}
GET: [{'id': 1, 'name': 'Item 1'}, {'id': 2, 'name': 'Item 2'}]
We can see that the requests have been successfully executed.
Conclusion
In this post, we introduced how to modify synchronous requests using requests to asynchronous requests using httpx, thereby leveraging the advantages of asynchronous programming to improve the performance and efficiency of applications.
If your service requires handling a large number of HTTP requests, using httpx is an appropriate choice. Although httpx can replace requests in most cases and offers more features, it is not 100% identical. There can be behavioral differences in some extreme cases.
So, if your application encounters inexplicable errors, do not completely rule out this possibility. I wish everyone (including myself) can write simpler code with better performance, and achieve significant breakthroughs in research projects!