Consume REST APIs with httpx

Imagine your Python script needs to pull the latest weather data, submit a user registration form, or fetch tweets from an API. You could use urllib.request or the ubiquitous requests library, but there's a modern contender that's built for the async age: httpx. If you're using requests for everything, you're missing out on cleaner async support, a more consistent interface, and the ability to throw in retries and timeouts without extra libraries. This lesson shows you how to consume REST APIs with httpx — synchronously and asynchronously — so you can make HTTP calls with confidence in 2024.

The problem this lesson solves

Most Python developers reach for requests when they need to call an API. It's a great library, but it has a few pain points that accumulate in real-world projects:

• No built-in async support. You have to switch to aiohttp or httpx if you want non-blocking calls with asyncio.
• Manual retry logic. requests doesn't retry on failure; you have to wrap calls in a loop and handle ConnectionError yourself.
• Inconsistent timeout handling. Timeouts apply to the whole connection, not to individual read or write operations.
• Verbose session management. You often write boilerplate to reuse connections.

httpx solves these by offering:

• A sync API that works exactly like requests for quick scripts.
• An async API using async/await for high-concurrency workloads.
• Built-in support for timeouts, retries, and connection pooling.
• HTTP/1.1 and HTTP/2 out of the box.

The goal of this lesson is to make you productive with httpx so you can consume REST APIs with httpx in your Python projects without friction.

Core concept / mental model

Think of httpx as a Swiss Army knife for HTTP. It's a client library that lets you:

• Send GET, POST, PUT, PATCH, DELETE requests.
• Attach headers, query parameters, JSON bodies, and files.
• Handle responses with status codes, headers, and content.
• Manage connections via a client object (like a browser session).

The key mental model: httpx.Client is your session. It maintains connection pools, default headers, cookies, and authentication across requests. Without a client, each request opens a new connection (which is fine for one-off calls but wasteful for many).

A simple call looks like this:

import httpx

response = httpx.get('https://httpbin.org/json')
print(response.status_code)  # 200
print(response.json()['slideshow']['title'])  # "Sample Slide Show"

Under the hood, httpx performs a synchronous HTTP request, parses the response, and returns a Response object. The entire process follows the standard HTTP lifecycle:

1. Open connection (or reuse from pool).
2. Send request line + headers + body (if any).
3. Receive response status + headers + body.
4. Return Response.

Pro tip: Use with httpx.Client() as client: to automatically close connections and benefit from connection reuse.

How it works step by step

Let's break down a typical httpx GET request into its core components:

1. Import and instantiate

import httpx

httpx exposes top-level functions like .get(), .post(), but it's better to create a Client for production code.

2. Create a client (optionally)

with httpx.Client() as client:
    response = client.get("https://api.example.com/items")

The Client object holds: - base_url — to avoid repeating the host. - headers — for common headers like Authorization. - timeout — default timeout for all requests. - cookies — persists across requests.

3. Send request with parameters

params = {"page": 1, "limit": 10}
response = client.get("/items", params=params)

params are automatically URL-encoded. You can also pass headers, cookies, and content for POST requests.

4. Handle the response

The Response object has many useful attributes:

5. Async variant

For async code, replace httpx.Client with httpx.AsyncClient:

async with httpx.AsyncClient() as client:
    response = await client.get("https://api.example.com/items")
    data = response.json()

Hands-on walkthrough

Let's put this into practice with three complete examples.

Example 1: Fetch and display data from a public API

We'll use the public Dog API to get a random dog image.

import httpx

def get_random_dog_image() -> str:
    """Fetch a random dog image URL from Dog API."""
    url = "https://dog.ceo/api/breeds/image/random"
    response = httpx.get(url)
    if response.status_code == 200:
        data = response.json()
        return data["message"]
    else:
        return f"Error: {response.status_code}"

print(get_random_dog_image())

Expected output:

https://images.dog.ceo/breeds/hound-afghan/nigel_21543.jpg

Example 2: POST data with JSON and authentication

Assume we have a task management API. We'll create a new task using an API token.

import httpx

with httpx.Client(base_url="https://api.example.com", headers={
    "Authorization": "Bearer your_secret_token_here",
    "User-Agent": "MyPythonApp/1.0"
}) as client:
    task_data = {
        "title": "Learn httpx",
        "description": "Consume REST APIs with httpx",
        "due_date": "2025-12-31"
    }
    response = client.post("/tasks", json=task_data)
    if response.is_success:
        new_task = response.json()
        print(f"Created task {new_task.get('id')}: {new_task.get('title')}")
    else:
        print(f"Failed: {response.status_code} - {response.text}")

Expected output (if API returns success):

Created task 42: Learn httpx

Example 3: Async batch fetching with retries

Suppose we need to fetch user profiles for a list of user IDs concurrently. This is where httpx shines.

import asyncio
import httpx

USER_IDS = [1, 2, 3, 4, 5]
API_BASE = "https://jsonplaceholder.typicode.com"

async def fetch_user(client: httpx.AsyncClient, user_id: int) -> dict:
    """Fetch a single user, retrying once on failure."""
    for attempt in range(2):
        try:
            response = await client.get(f"/users/{user_id}")
            response.raise_for_status()
            return response.json()
        except (httpx.TimeoutException, httpx.HTTPStatusError) as exc:
            if attempt == 1:
                raise exc
            await asyncio.sleep(1)

async def main():
    async with httpx.AsyncClient(base_url=API_BASE) as client:
        tasks = [fetch_user(client, uid) for uid in USER_IDS]
        users = await asyncio.gather(*tasks, return_exceptions=True)
        for user_id, user in zip(USER_IDS, users):
            if isinstance(user, Exception):
                print(f"User {user_id}: Failed -> {user}")
            else:
                print(f"User {user_id}: {user['name']} - {user['email']}")

await main()

Expected output (simplified):

User 1: Leanne Graham - Sincere@april.biz
User 2: Ervin Howell - Shanna@melissa.tv
...

This async pattern gives you massive speed improvements when fetching many endpoints, without blocking other operations.

Compare options / when to choose what

When to choose httpx: - You need async support (web scrapers, microservices, real-time dashboards). - You want a consistent API across sync and async code. - You need HTTP/2 for better performance with multiple requests. - You want built-in timeouts and retry logic.

When to stick with requests: - You're writing a quick script that makes one or two calls. - Your team is already deep into requests and doesn't need async. - You're on a very old Python version (httpx requires 3.7+).

Troubleshooting & edge cases

1. httpx.ConnectError: All connection attempts failed

Cause: Server unreachable, DNS failure, or network blocked.

Fix:

try:
    response = httpx.get("https://unreachable.example.com", timeout=5.0)
except httpx.ConnectError as e:
    print(f"Network error: {e}")

2. httpx.TimeoutException: The read operation timed out

Cause: The server is too slow to respond within the default timeout (5s).

Fix: Increase timeout or configure granular timeouts:

client = httpx.Client(timeout=httpx.Timeout(10.0, connect=5.0, read=15.0))

3. typError: JSONDecodeError when using .json()

Cause: The response body is not valid JSON (e.g., HTML error page).

Fix: Always check the status code first, or use .text to inspect:

response = httpx.get("https://api.example.com/data")
if response.status_code != 200:
    print(f"Unexpected response: {response.text[:200]}")
else:
    data = response.json()

4. SSL errors with self-signed certificates

Cause: Internal APIs with custom or self-signed certificates.

Fix:

# For development only (do not use in production)
import httpx
client = httpx.Client(verify=False)
# Or provide a custom certificate bundle
client = httpx.Client(verify="/path/to/custom-ca-bundle.crt")

Pro tip: Use retries with httpx.Client(event_hooks={'request': [some_hook]}) for automatic retry logic. See httpx documentation.

What you learned & what's next

You now understand how to consume REST APIs with httpx in Python:

• ✅ The core concept of httpx.Client as a session manager
• ✅ How to make GET and POST requests with parameters, JSON, and authentication
• ✅ The difference between sync and async APIs
• ✅ How to handle timeouts, errors, and SSL issues
• ✅ When to choose httpx over requests or aiohttp

These skills are essential for integrating third-party services, building web scrapers, or creating API-driven applications.

Next step: In the next lesson, you'll learn how to build a CLI tool that consumes multiple REST APIs using httpx, adding command-line argument parsing and error handling. This will solidify your async API consumption skills in a real-world project.

Ready to level up? Let's go!