Build Your First Web Scraper with BeautifulSoup and Requests

Ever found yourself copy-pasting data from a website for hours? I've been there too. That's exactly why I started learning web scraping. Today, I'll show you how to build a simple but powerful web scraper using Python's BeautifulSoup and Requests libraries.

What You'll Need

Before we dive in, make sure you have Python installed. We're going to work with two fantastic libraries: Requests handles getting the web page, while BeautifulSoup helps us parse and extract the data we want.

Let's install them first:

pip install requests beautifulsoup4

Start with a Simple Example

Imagine you want to grab the latest article titles from PythonSkillset.com (our own site). It's a practical starting point because you're not breaking any rules.

Here's your first scraper:

import requests
from bs4 import BeautifulSoup

url = "https://pythonskillset.com/articles"
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')

# Find all article titles
titles = soup.find_all('h2', class_='article-title')
for title in titles:
    print(title.text.strip())

That's it. Seven lines of code and you've got your first working scraper.

Understanding the Structure

The beauty of BeautifulSoup is how it mimics HTML structure. You can navigate the page tree like you're clicking through folders on your computer:

• find() gives you the first match
• find_all() returns every match as a list
• You can chain these methods together for precision

A More Realistic Example

Let's say you're tracking prices for a hobby project. Here's how you'd build something practical:

import requests
from bs4 import BeautifulSoup
import csv
from datetime import datetime

def scrape_product_data(url):
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
    }

    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.content, 'html.parser')

    products = []
    for item in soup.find_all('div', class_='product-card'):
        name = item.find('h3').text.strip()
        price = item.find('span', class_='price').text.strip()
        link = item.find('a')['href']

        products.append({
            'name': name,
            'price': price,
            'url': link,
            'scraped_at': datetime.now().isoformat()
        })

    return products

# Save to CSV
products = scrape_product_data('https://example-store.com/products')
with open('products.csv', 'w', newline='') as f:
    writer = csv.DictWriter(f, fieldnames=['name', 'price', 'url', 'scraped_at'])
    writer.writeheader()
    writer.writerows(products)

Handling Common Issues

Websites aren't always cooperative. Here are the usual suspects you'll run into:

JavaScript-heavy pages: Many modern sites load content dynamically. If you only see scripts in your output, that's the issue. For those, you'll need Selenium or Playwright instead.

Anti-scraping measures: Some sites block scrapers. Adding a user-agent header (like we did above) helps. Rate limiting your requests is good practice too.

Bad HTML: Real websites have messy code sometimes. Use soup.prettify() to see what BeautifulSoup actually received.

A Word of Caution

Web scraping comes with responsibilities. Always check a site's robots.txt file first (like https://pythonskillset.com/robots.txt). Respect rate limits, and never scrape copyrighted content or personal data without permission.

Next Steps

You now have the foundation. From here, you can:

• Add error handling with try/except blocks
• Schedule your scraper with cron jobs
• Store data in a database instead of CSV
• Build a web interface to display results

Start simple. Scrape a page you know well. Then gradually tackle more complex sites. Before you know it, you'll be automating all sorts of data collection tasks.

The real magic happens when you combine what you've learned here with other Python tools. Maybe that's for another article though.