Use Sets for Unique Elements

Ever found yourself staring at a list of user IDs with duplicates, trying to figure out how to extract just the unique ones without writing a messy loop? That exact pain — the noise of repeated elements blocking clean data — is what sets solve in Python. By the end of this lesson, you'll wield sets with confidence to remove duplicates, test membership, and perform blazing-fast comparisons.

The problem this lesson solves

When you work with real-world data — log files, API responses, CSV exports — duplicates are everywhere. A list of email addresses might have the same person signing up twice. A list of tags could contain both "Python" and "python". The core challenge: how do you efficiently isolate only the unique elements?

Common approaches fail you:

• Looping through a list and building a second list manually is slow and error-prone.
• Using list.count() is O(n²) — unusable for large datasets.
• Sorting first adds unnecessary complexity.

The missing ingredient is a Python data structure designed from the ground up for uniqueness: the set. Sets guarantee every element appears exactly once, and they use the same mathematical set operations you learned in middle school, backed by lightning-fast hash tables.

Core concept / mental model

Think of a set as a bag of unique items where order doesn't matter. Unlike a list, which is an ordered sequence of items (some repeated), a set is:

• Unordered — elements have no index; you cannot ask for "the third element".
• Mutable — you can add or remove elements after creation (unlike tuples).
• Hashable-only — every element must be immutable (strings, numbers, tuples; no lists or dicts).

Pro tip: A set is conceptually identical to a mathematical set. If you've ever drawn Venn diagrams, you already know union, intersection, and difference. Python gives you those same operations with simple syntax.

Set definition

A set is a collection of distinct objects represented between curly braces {} or via the set() constructor. Internally, Python stores each element in a hash table, making membership tests O(1) average — way faster than a list's O(n).

# Creating a set from literal values
fruits = {"apple", "banana", "cherry", "apple"}
print(fruits)  # Output: {'banana', 'cherry', 'apple'} — note: order is arbitrary!

# Creating a set from any iterable via constructor
numbers = set([1, 2, 3, 2, 1])
print(numbers)  # Output: {1, 2, 3}

How it works step by step

1. Building a set from scratch

To create a set, you can either list elements in braces or pass an iterable (list, tuple, string) to set(). Duplicates disappear automatically.

# Step 1: start with a list with duplicates
emails = ["alice@example.com", "bob@example.com", "alice@example.com"]

# Step 2: convert to set to get unique emails
unique_emails = set(emails)

# Step 3: (optional) convert back to list if you need ordering
unique_email_list = list(unique_emails)
print(unique_email_list)  # Output: ['bob@example.com', 'alice@example.com']

2. Adding and removing elements

Sets are mutable, so they grow and shrink as needed:

• add(element) — add a single new element (no effect if already present)
• remove(element) — delete an element; raises KeyError if missing
• discard(element) — delete if present; does nothing if missing (safe)
• pop() — remove and return an arbitrary element (useful for batch processing)
• clear() — empty the entire set

tags = {"python", "data"}
tags.add("science")
tags.discard("java")  # no error even though 'java' is not in the set
tags.add("python")    # duplicate — set ignores it
print(tags)           # Output (order varies): {'science', 'data', 'python'}

3. Membership tests – the killer feature

Checking if an element exists in a set is orders of magnitude faster than checking a list, especially as the collection grows.

big_list = list(range(100000))
big_set = set(big_list)

import timeit
# List lookup
start = timeit.default_timer()
result = 99999 in big_list
end = timeit.default_timer()
print(f"List lookup: {end - start:.6f} sec")  # ~0.003 sec

# Set lookup
start = timeit.default_timer()
result = 99999 in big_set
end = timeit.default_timer()
print(f"Set lookup: {end - start:.6f} sec")   # ~0.000001 sec

Pro tip: Whenever you only care whether an item exists (not where it is), use a set for membership checks. The speed advantage grows with collection size.

4. Set operations (union, intersection, difference, symmetric difference)

These mathematical operations work on two or more sets:

python_devs = {"alice", "bob", "charlie"}
data_scientists = {"charlie", "diana", "edward"}

# All unique people across both teams
all_team = python_devs | data_scientists
print(all_team)  # Output (order varies): {'alice', 'bob', 'charlie', 'diana', 'edward'}

# People in both teams (intersection)
both = python_devs & data_scientists
print(both)  # Output: {'charlie'}

# People only in python_devs (difference)
only_python = python_devs - data_scientists
print(only_python)  # Output: {'alice', 'bob'}

# People in exactly one team (symmetric difference)
exclusive = python_devs ^ data_scientists
print(exclusive)  # Output: {'alice', 'bob', 'diana', 'edward'}

Hands-on walkthrough

Let's solve a real problem together: Remove duplicate IP addresses from a server log and find which IPs appear in multiple logs.

Step 1: Simulate log data

log1 = ["192.168.1.1", "10.0.0.2", "192.168.1.1", "172.16.0.3"]
log2 = ["10.0.0.2", "192.168.1.10", "172.16.0.3", "10.0.0.2"]

Step 2: Convert each log to a set to remove duplicates

unique_log1 = set(log1)
unique_log2 = set(log2)

print(f"Unique in log1: {unique_log1}")
print(f"Unique in log2: {unique_log2}")
# Output:
# Unique in log1: {'172.16.0.3', '10.0.0.2', '192.168.1.1'}
# Unique in log2: {'10.0.0.2', '172.16.0.3', '192.168.1.10'}

Step 3: Find IPs that appear in both logs

common_ips = unique_log1 & unique_log2
print(f"IPs in both logs: {common_ips}")
# Output: IPs in both logs: {'10.0.0.2', '172.16.0.3'}

Step 4: Find IPs only in log1

log1_only = unique_log1 - unique_log2
print(f"IPs exclusive to log1: {log1_only}")
# Output: IPs exclusive to log1: {'192.168.1.1'}

Notice how you didn't write a single loop. The set operations handled everything cleanly and fast.

Compare options / when to choose what

When to use sets specifically:

• Deduplication: You have a list/iterable and need only unique items.
• Membership checks: You repeatedly test if x in collection.
• Set operations: You need union, intersection, difference — e.g., "which users are in both groups?".
• Removing dups from large data: A single set() conversion is O(n) — far better than nested loops.

When to avoid sets:

• You need to preserve original order of first occurrence → use dict.fromkeys() instead.
• Elements aren't hashable (e.g., lists, dictionaries) → you'll get a TypeError.
• You need random access by index → use a list.

Troubleshooting & edge cases

1. Forgetting sets are unordered

fruits = {"apple", "banana", "cherry"}
print(fruits[0])  # TypeError: 'set' object is not subscriptable

Fix: If you need ordering, convert to a sorted list: sorted(fruits).

2. Adding unhashable types

try:
    my_set = {[1, 2, 3]}  # list is unhashable
except TypeError as e:
    print(e)  # Output: unhashable type: 'list'

Fix: Use tuples (which are immutable) instead of lists in sets.

3. Empty set vs empty dictionary

a = {}  # This is a dictionary, not a set!
print(type(a))  # Output: <class 'dict'>

b = set()  # This is an empty set
print(type(b))  # Output: <class 'set'>

Fix: Always use set() to create an empty set.

What you learned & what's next

You now understand how to use sets for unique element operations — from creation and mutation to powerful set operations like union, intersection, and difference. You know when to reach for a set vs. a list or dict, and you've handled common pitfalls like unhashable types and ordering surprises.

This was a foundational skill in the Python Fundamentals track. Next up, we'll explore dictionaries: mapping keys to values — building on your understanding of hash-based structures.