Use re module in Python

Have you ever needed to extract an email address from a messy string, validate a phone number format, or replace all occurrences of a pattern in a document? Doing this with basic string methods like .find() and .replace() quickly becomes unwieldy and error-prone. Python’s re module gives you the superpower of regular expressions — a compact and powerful language for describing patterns in text. This lesson will turn you from a regex novice into someone who can confidently search, match, and transform strings with a few lines of code.

The problem this lesson solves

Manual string processing with loops and conditionals works for simple, predictable text. But real-world data is rarely that polite. You might need to:

• Validate that a user-provided email address follows a standard format.
• Extract all URLs from a block of HTML or log file.
• Replace inconsistent whitespace (tabs, multiple spaces, newlines) with a single space.
• Check if a password meets complexity rules (uppercase, lowercase, digit, special char).

Using Python’s built-in string methods for these tasks results in fragile, verbose code that is hard to read and harder to maintain. The re module provides a concise, declarative way to define patterns and apply them to strings — saving time and reducing bugs.

Core concept / mental model

Think of a regular expression (regex) as a miniature programming language that describes a pattern of characters. Instead of writing ten lines of Python to check “does this look like an email?”, you write a one-line pattern like r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}'.

The re module acts as the engine that interprets your pattern and applies it to a target string. The key functions are:

• re.match() — checks if the pattern matches from the start of the string.
• re.search() — finds the first occurrence of the pattern anywhere in the string.
• re.findall() — returns all non-overlapping matches as a list.
• re.sub() — replaces occurrences of the pattern with a replacement string.

Raw strings (r"...") are critical: they tell Python not to interpret backslashes (\) as escape sequences, which lets regex use them freely (e.g., \d for digits).

Pro tip: Always use raw strings for regex patterns. Writing r"\d+" instead of "\\d+" prevents a common source of confusion.

How it works step by step

1. Import the module

import re

2. Write a pattern using raw string syntax

pattern = r'\d{3}-\d{3}-\d{4}'   # matches phone numbers like 555-123-4567

3. Call a function on the target string

4. Inspect the result

Match objects have useful methods: * .group() — the matched text * .start() — start index of match * .end() — end index of match * .span() — tuple (start, end)

Hands-on walkthrough

Let’s use re to solve real problems.

Example 1: Validating a simple email format

import re

def is_valid_email(email):
    # Simple pattern: something@domain.tld
    pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
    return re.match(pattern, email) is not None

# Test
print(is_valid_email("hello@example.com"))    # True
print(is_valid_email("user.name+tag@sub.domain.co"))  # True
print(is_valid_email("@example.com"))         # False
print(is_valid_email("hello@"))               # False
print(is_valid_email("hello@example.c"))      # False (only 1 char in TLD)

Example 2: Extracting all phone numbers from text

import re

text = """
Contact us at 555-123-4567 or 555-987-6543.
Support: 555-111-2222.
"""

pattern = r'\d{3}-\d{3}-\d{4}'
phone_numbers = re.findall(pattern, text)
print(phone_numbers)  # ['555-123-4567', '555-987-6543', '555-111-2222']

Example 3: Cleaning up messy whitespace

import re

messy = "This   has   extra   spaces   and\t\ttabs.\nAnd newlines too.   "
# Replace one or more whitespace characters (\s+) with single space
clean = re.sub(r'\s+', ' ', messy).strip()
print(clean)  # 'This has extra spaces and tabs. And newlines too.'

Example 4: Finding log entries (simulated)

import re

log = """
2023-10-05 14:23:10 INFO  User login successful
2023-10-05 14:25:33 ERROR Connection timeout
2023-10-05 14:30:01 WARN  Disk space low
"""

# Extract all ERROR messages with their timestamps
pattern = r'^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) ERROR (.*)$'
errors = re.findall(pattern, log, re.MULTILINE)
for timestamp, message in errors:
    print(f"[{timestamp}] {message}")
# Output:
# [2023-10-05 14:25:33] Connection timeout

Pro tip: The re.MULTILINE flag (or re.M) makes ^ and $ match start/end of each line instead of start/end of the whole string.

Compare options / when to choose what

Sometimes you shouldn't reach for regex. Here’s a quick decision guide:

When to avoid regex: If the pattern is fixed and simple (str methods suffice), or if you’re parsing nested structures (HTML, JSON, math expressions), dedicated parsers are more robust and readable.

Troubleshooting & edge cases

Common pitfalls and how to fix them

1. Pattern matches but returns None — Check if re.match() only matches at the start. Use re.search() instead.

pattern = r'cat'
print(re.match(pattern, 'I have a cat'))   # None (no match at start)
print(re.search(pattern, 'I have a cat'))  # <re.Match object...>

1. Matching too much or too little — By default, quantifiers like * and + are greedy (match as much as possible). Add ? to make them lazy.

text = "<div>first</div><div>second</div>"
greedy = re.findall(r'<div>.*</div>', text)  # ['<div>first</div><div>second</div>']
lazy   = re.findall(r'<div>.*?</div>', text)  # ['<div>first</div>', '<div>second</div>']

1. Special characters in literal text — To match a literal ., , ?, etc., escape them with a backslash: \., \, \?.

# Find sentences ending with a period
print(re.findall(r'[^.?!]*\.', 'Hello world. How are you? Fine.'))  # ['Hello world.', ' Fine.']

1. re functions always return strings or None — For re.findall(), an empty list means no matches. Check for None from re.match() and re.search() before calling .group().

2. Performance with large texts — Use re.compile() if you reuse the same pattern multiple times. The compiled pattern runs faster.

pattern = re.compile(r'\d{3}-\d{4}')
match1 = pattern.search('Call 555-1234')
match2 = pattern.search('Also 999-5678')

What you learned & what's next

You’ve unlocked a powerful tool: the re module for regular expressions in Python. You now know how to:

• Explain the purpose of regex and raw strings.
• Use re.match(), re.search(), re.findall(), and re.sub() for common tasks.
• Write simple patterns for emails, phone numbers, and whitespace cleanup.
• Troubleshoot greedy matching, special characters, and start-of-string vs. anywhere matching.

Next step: In the next lesson, you'll apply regex to parse and transform structured data (like CSV or log files) — combining your regex skills with Python’s file I/O for real-world automation.

Practice now by writing a script that reads a text file, finds all numbers (integers and decimals), and replaces them with [REDACTED]. Use re.findall() first to understand the pattern, then re.sub() to execute the replacement.