Build a Basic Web Scraper

Building a web scraper from scratch can feel like sneaking into someone else's house — you need the right tools and a clear understanding of what's allowed. Without a proper approach, you'll end up with broken HTML, blocked requests, or worse, legal headaches. This lesson walks you through building a basic web scraper with requests and BeautifulSoup, two powerhouse Python libraries that make extracting data from websites both safe and effective.

The problem this lesson solves

Manual data collection is slow and error-prone. Whether you're tracking product prices, aggregating news headlines, or gathering research data, copying and pasting from dozens of web pages wastes hours. Web scraping automates this, but many developers struggle with the first step: getting the HTML and parsing it reliably.

The core challenge is that websites aren't databases — they display data in messy, nested HTML structures. You need a way to:

1. Fetch the raw HTML from a URL (without getting blocked)
2. Parse that HTML into a navigable tree
3. Extract exactly the pieces you need (headlines, prices, links, etc.)
4. Handle errors when the page structure changes or a request fails

Using requests for the fetch and BeautifulSoup for parsing gives you a clean, Pythonic solution that works for 90% of scraping needs.

Pro tip: Always check a website's robots.txt and terms of service before scraping. Respecting robots.txt is both ethical and keeps your IP from being banned.

Core concept / mental model

Think of a web scraper as a three-layer sandwich:

• Layer 1: The Fetcher (requests) — This makes an HTTP GET request to the server and brings back the raw HTML response. It's like dialing a phone number and getting the voicemail recording.
• Layer 2: The Parser (BeautifulSoup) — This turns the raw text into a structured tree of tags, attributes, and text nodes. Think of it as converting a voice recording into a written transcript with timestamps.
• Layer 3: The Extractor (CSS selectors or .find()) — You query the parsed tree for specific elements — all <h2> tags, all links with a certain class, or the third <div> inside a <section>. This is your highlight pen on the transcript.

The magic happens when you chain these layers: requests → BeautifulSoup → .select() or .find_all().

Definitions you need

How it works step by step

Let's build the scraper piece by piece. The flow is always:

1. Install dependencies (one-time setup)
2. Fetch the page and check the response status
3. Parse the HTML into a BeautifulSoup object
4. Locate target elements using tags, classes, or CSS selectors
5. Extract data — text, links, or attributes
6. Handle errors gracefully (timeouts, missing elements, status codes)

Step 1: Install requests and beautifulsoup4

pip install requests beautifulsoup4

Note: The package name for installation is beautifulsoup4, but you import it as bs4.

Step 2: Fetch a webpage

import requests

url = "https://example.com/news"
response = requests.get(url)

# Always check the response status
if response.status_code == 200:
    html_content = response.text
    print("Page fetched successfully")
else:
    print(f"Failed to fetch page: {response.status_code}")

Step 3: Parse with BeautifulSoup

from bs4 import BeautifulSoup

soup = BeautifulSoup(html_content, 'html.parser')

The second argument 'html.parser' tells BeautifulSoup to use Python's built-in HTML parser. You can also use 'lxml' after installing lxml (faster but extra dependency).

Step 4: Extract elements using find_all()

# Find all <h2> tags (often headings)
headings = soup.find_all('h2')
for heading in headings:
    print(heading.text.strip())

heading.text returns the visible text inside the tag. .strip() removes extra whitespace.

Step 5: Use CSS selectors for precision

When pages use specific classes or IDs, CSS selectors are powerful:

# Find all links inside elements with class "article-card"
articles = soup.select('.article-card a')
for link in articles:
    href = link.get('href')
    title = link.text.strip()
    print(f"{title}: {href}")

The .select() method accepts any CSS selector — learn the basics and you can target almost anything.

Hands-on walkthrough

Let's scrape a public, friendly site — Books to Scrape (http://books.toscrape.com), a demo site built for practicing scraping.

Complete scraper: Extract book titles and prices

import requests
from bs4 import BeautifulSoup

url = "http://books.toscrape.com/catalogue/page-1.html"

response = requests.get(url)
if response.status_code != 200:
    print(f"Failed to fetch page: {response.status_code}")
    exit()

soup = BeautifulSoup(response.text, 'html.parser')

# Each book is inside an <article> with class "product_pod"
books = soup.find_all('article', class_='product_pod')

for book in books:
    # Extract title from <h3><a> tag
    title_tag = book.find('h3').find('a')
    title = title_tag.get('title', title_tag.text.strip()) if title_tag else "No title"

    # Extract price from <p> with class "price_color"
    price_tag = book.find('p', class_='price_color')
    price = price_tag.text.strip() if price_tag else "No price"

    print(f"{title}: {price}")

Expected output (truncated):

A Light in the Attic: £51.77
Tipping the Velvet: £53.74
Soumission: £50.10
...

Scraper with error handling and headers

Many modern websites block scrapers with no user-agent. Always send a realistic user-agent header:

import requests
from bs4 import BeautifulSoup
import time

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
}

try:
    response = requests.get("http://books.toscrape.com/", headers=headers, timeout=10)
    response.raise_for_status()  # Raises HTTPError for bad status codes
except requests.exceptions.RequestException as e:
    print(f"Request failed: {e}")
    exit()

soup = BeautifulSoup(response.text, 'html.parser')

# Sleep between requests to be polite
time.sleep(1)

# Extract categories from sidebar
categories = soup.select('.side_categories ul li a')
for cat in categories:
    cat_text = cat.text.strip()
    # Skip empty or "Books" category
    if cat_text and cat_text.lower() != "books":
        print(f"Category: {cat_text}")

Expected output (partial):

Category: Travel
Category: Mystery
Category: Historical Fiction
Category: Sequential Art
...

Critical rule: Always respect robots.txt. For http://books.toscrape.com/robots.txt, you'll see it allows all — perfect for practice.

Compare options / when to choose what

Parser choices for BeautifulSoup

Recommendation: Start with 'html.parser'. Switch to 'lxml' when scraping large sites or when the HTML is malformed.

find_all() vs .select()

.select() is more powerful but slower for simple cases. Use find_all for clarity, .select() for tricky DOM structures.

Requests vs. other HTTP libraries

Stick with requests for this stage — it's the simplest and most widely used.

Troubleshooting & edge cases

1. HTTP 403 Forbidden

Cause: The server sees your default requests user-agent and blocks it. Fix: Add a realistic User-Agent header (as shown in the hands-on section).

2. Empty results from find_all() or .select()

Cause: The element you're looking for is loaded dynamically via JavaScript (e.g., React, Vue). requests only fetches the initial HTML, not JS-generated content. Fix: Check the page source. If the data isn't there, you need a headless browser like Selenium or Playwright (covered in a later lesson).

3. Stale data or broken scraping after a week

Cause: The website updated its HTML structure — class names changed, elements were reorganized. Fix: Build in defensive checks:

if not books:
    print("No books found — structure may have changed")
    # Log the raw HTML for inspection
    with open('debug.html', 'w') as f:
        f.write(response.text)

4. SSL certificate errors

response = requests.get(url, verify=False)  # Not recommended in production

Better approach: install certificates or use verify='/path/to/cert.pem'.

5. Rate limiting

Fix: Add time.sleep(1) between requests or use requests with a retry adapter.

from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

session = requests.Session()
retries = Retry(total=3, backoff_factor=0.5)
session.mount('https://', HTTPAdapter(max_retries=retries))

What you learned & what's next

You now know how to build a basic web scraper with requests and BeautifulSoup:

• Fetching HTML with proper headers and error handling
• Parsing HTML into a navigable BeautifulSoup tree
• Extracting data using find_all() and .select()
• Choosing the right parser and method for the task
• Handling common issues like 403 errors, dynamic content, and stale structures

You've completed Learning Objective 1 (explain the core idea) and Learning Objective 2 (practical exercise).

What's next: The next lesson covers scraping multiple pages and handling pagination — turning your single-page scraper into a data collection machine. You'll learn to follow "Next" links and build a loop that respects rate limits.