Skip to content

[Python] Use `httpx` To Replace `requests` For Asynchronous Requests

Last Updated on 2024-07-28 by Clay

In Python programming, we often use the requests module for HTTP requests. However, requests can become a bottleneck when connecting frontend and backend services due to its synchronous request handling. Recently, I experienced Kubernetes probe blockages caused by using requests, which led to the unintended deletion of my service container. In such scenarios, httpx might be a more suitable module for asynchronous request handling.

Introduction to httpx

httpx is a HTTP client library that supports both synchronous and asynchronous API calls natively. It can directly work with Python's async and await keywords.

Additionally, it comes with several powerful features such as proxy support, SSL configuration, customizable retry strategies (so you don't have to write them yourself!), and more. In the current development environment, I feel it can almost completely replace the requests module.

Next, we will show how to use httpx, but before that, let's create a mock FastAPI service and the original requests code.

Of course, if you already have existing code you want to modify, you can jump directly to the httpx section below and start modifying your code.

Creating a Mock FastAPI Service

First, install fastapi and uvicorn.

pip install fastapi uvicorn


Then, we will create a simple service.

from fastapi import FastAPI
from pydantic import BaseModel
from typing import List

app = FastAPI()

class Item(BaseModel):
    id: int
    name: str

fake_db: List[Item] = []

@app.get("/items/", response_model=List[Item]])
async def read_items():
    return fake_db

@app.post("/items/", response_model=Item])
async def create_item(item: Item):
    fake_db.append(item)
    return item


Next, start the service:

uvicorn app:app --reload


Output:

INFO:     Started server process [560791]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)

It will start at http://127.0.0.1:8000 by default.

Original requests Syntax

If you haven't installed requests yet (which is unlikely if you're reading this), you can install it using the following command:

pip install requests


After that, you can write Python code to send requests to the APIs.

import requests

# GET
response = requests.get("http://127.0.0.1:8000/api/fake-db/items")
print("GET:", response.json())

# POST
new_item = {"id": 1, "name": "Item 1"}
response = requests.post("http://127.0.0.1:8000/api/fake-db/items", json=new_item)
print("POST:", response.json())

# GET
response = requests.get("http://127.0.0.1:8000/api/fake-db/items")
print("GET:", response.json())


Output:

GET: []
POST: {'id': 1, 'name': 'Item 1'}
GET: [{'id': 1, 'name': 'Item 1'}]

As you can see, everything works fine.

At this point, we have completed setting up the FastAPI service and sending synchronous requests using requests. If we only need to support synchronous requests, we can simply replace all occurrences of requests with httpx, and this will be the synchronous request method for httpx, which is fully compatible with requests syntax. Isn't that great?

However, the strength of httpx lies in its asynchronous requests. Let's now see how to perform asynchronous requests with httpx!

Asynchronous Requests with httpx

The simplest way to make asynchronous requests with httpx involves using the httpx.AsyncClient() class.

import asyncio
import httpx


async def main():
    async with httpx.AsyncClient() as client:
        # GET
        response = await client.get("http://127.0.0.1:8000/api/fake-db/items")
        print("GET:", response.json())

        # POST
        new_item = {"id": 2, "name": "Item 2"}
        response = await client.post("http://127.0.0.1:8000/api/fake-db/items", json=new_item)
        print("POST:", response.json())

        # GET again
        response = await client.get("http://127.0.0.1:8000/api/fake-db/items")
        print("GET:", response.json())


# Run
asyncio.run(main())


Output:

GET: [{'id': 1, 'name': 'Item 1'}]
POST: {'id': 2, 'name': 'Item 2'}
GET: [{'id': 1, 'name': 'Item 1'}, {'id': 2, 'name': 'Item 2'}]

We can see that the requests have been successfully executed.

Conclusion

In this post, we introduced how to modify synchronous requests using requests to asynchronous requests using httpx, thereby leveraging the advantages of asynchronous programming to improve the performance and efficiency of applications.

If your service requires handling a large number of HTTP requests, using httpx is an appropriate choice. Although httpx can replace requests in most cases and offers more features, it is not 100% identical. There can be behavioral differences in some extreme cases.

So, if your application encounters inexplicable errors, do not completely rule out this possibility. I wish everyone (including myself) can write simpler code with better performance, and achieve significant breakthroughs in research projects!

References

Read More

[Python] Crawler download a file via resume breakpoint
[Solved][Python] requests.exceptions.TooManyRedirects: Exceeded 30 redirects
Tags:

Leave a Reply