Python Web Scraping with BeautifulSoup: Tutorial

Python with AI Tutorial · Chapter 36 of 48

Python web scraping means downloading a web page with requests and pulling the data you need out of its HTML with BeautifulSoup. This chapter scrapes book titles, prices and ratings from a practice site, follows pagination, saves the result to CSV and explains the rules (robots.txt, terms of use, rate limits) that keep scraping legal and polite.

Table of Contents

Install the libraries

You need requests to fetch pages and beautifulsoup4 to parse them. Both install with pip. We practise on books.toscrape.com and quotes.toscrape.com, two sites built specifically for learning to scrape.

pip install requests beautifulsoup4

Output:
Successfully installed beautifulsoup4-4.12.3 requests-2.32.3 soupsieve-2.6 ...

Check robots.txt first

Before scraping any site, read its terms of use and its robots.txt file, which lists the paths the owner does not want automated tools to visit. Python can check it for you.

from urllib import robotparser

rp = robotparser.RobotFileParser()
rp.set_url("https://books.toscrape.com/robots.txt")
rp.read()
print(rp.can_fetch("*", "https://books.toscrape.com/catalogue/page-2.html"))

Output:
True

True means the path is allowed. If a site disallows a path, or its terms forbid automated access, stop and look for an official API or a data export instead. Never scrape personal data or content behind a login.

Fetch and parse a page

Download the HTML with requests.get(), then hand the bytes to BeautifulSoup. The parser turns the page into a tree you can search.

import requests
from bs4 import BeautifulSoup

url = "https://books.toscrape.com/"
r = requests.get(url, timeout=10)
r.raise_for_status()
soup = BeautifulSoup(r.content, "html.parser")

print(soup.title.get_text(strip=True))
print(len(soup.select("article.product_pod")), "books on this page")

Output:
All products | Books to Scrape - Sandbox
20 books on this page

Tip: pass r.content (bytes), not r.text, to BeautifulSoup. When a server does not declare its character set, r.text guesses Latin-1 and the pound sign turns into Â£. BeautifulSoup reads the charset from the HTML itself and decodes correctly.

Extract data from one element

Open the page in your browser, right-click a book and choose Inspect. Each book sits in an <article class="product_pod">. CSS selectors then pick out the pieces.

book = soup.select_one("article.product_pod")

title = book.h3.a["title"]
price = book.select_one("p.price_color").get_text(strip=True)
rating = book.select_one("p.star-rating")["class"][1]
stock = book.select_one("p.availability").get_text(strip=True)

print(title, price, rating, stock, sep=" | ")

Output:
A Light in the Attic | £51.77 | Three | In stock

book.h3.a["title"] reads an attribute, get_text() reads the visible text, and ["class"] returns a list because an element can have several classes; the rating word is the second one.

BeautifulSoup selection methods

You will use a handful of methods for almost every scraper.

Method	Returns	Example
`select(css)`	List of all matching elements	`soup.select("article.product_pod")`
`select_one(css)`	First match or `None`	`soup.select_one("li.next a")`
`find(tag, attrs)`	First match by tag and attributes	`soup.find("p", class_="price_color")`
`find_all(tag, attrs)`	All matches by tag	`soup.find_all("a")`
`get_text(strip=True)`	Visible text without surrounding whitespace	`el.get_text(strip=True)`
`el["href"]` / `el.get("href")`	Attribute value (`.get` returns `None` if missing)	`link.get("href")`
`el.parent`, `el.find_next_sibling()`	Navigate the tree	`price.parent`
`soup.prettify()`	Indented HTML for inspection	`print(book.prettify())`

Loop over every book on the page

Turn each card into a dictionary and convert the price to a number so you can sort and total it.

books = []
for card in soup.select("article.product_pod"):
    books.append({
        "Title": card.h3.a["title"],
        "Price": float(card.select_one("p.price_color").get_text(strip=True).lstrip("£")),
        "Rating": card.select_one("p.star-rating")["class"][1],
    })

print(books[0])
most_expensive = max(books, key=lambda b: b["Price"])
print(most_expensive["Title"], most_expensive["Price"])

Output:
{'Title': 'A Light in the Attic', 'Price': 51.77, 'Rating': 'Three'}
Set Me Free 57.25

Try it with AI

Paste a snippet of HTML from the Inspect panel and ask for the selectors.

Here is the HTML of one product card from a page I am allowed to scrape:

<article class="product_pod">
  <p class="star-rating Three"></p>
  <h3><a href="catalogue/a-light-in-the-attic_1000/index.html" title="A Light in the Attic">A Light in the ...</a></h3>
  <div class="product_price"><p class="price_color">£51.77</p><p class="instock availability">In stock</p></div>
</article>

Write a Python function parse_card(card) using BeautifulSoup that returns a dict with keys Title, Url (absolute, base https://books.toscrape.com/), Price as float, Rating as an integer 1-5 and InStock as a boolean. Handle a missing price by returning None for that key.

Follow pagination politely

Real data spans many pages. Find the Next link, build an absolute URL with urljoin(), and pause between requests so you do not hammer the server. Identify yourself with a User-Agent header.

import time
from urllib.parse import urljoin

HEADERS = {"User-Agent": "learning-scraper/1.0 (contact: you@example.com)"}

def scrape_pages(start_url, max_pages=3):
    url = start_url
    rows = []
    for _ in range(max_pages):
        r = requests.get(url, headers=HEADERS, timeout=10)
        r.raise_for_status()
        soup = BeautifulSoup(r.content, "html.parser")
        for card in soup.select("article.product_pod"):
            price = float(card.select_one("p.price_color").get_text(strip=True).lstrip("£"))
            rows.append((card.h3.a["title"], price))
        next_link = soup.select_one("li.next a")
        if not next_link:
            break
        url = urljoin(url, next_link["href"])
        time.sleep(1)          # be polite: one request per second
    return rows

rows = scrape_pages("https://books.toscrape.com/catalogue/page-1.html")
print(len(rows), "books collected")
print(rows[0])

Output:
60 books collected
('A Light in the Attic', 51.77)

The site has 50 pages and 1,000 books. Raise max_pages when you are ready, but keep the sleep in place. A one-second delay is a good default for any site that does not publish a crawl rate.

Save the results to CSV

Scraped data usually ends up in a spreadsheet or a pandas DataFrame. The csv module writes it in two lines.

import csv

with open("books.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.writer(f)
    writer.writerow(["Title", "Price"])
    writer.writerows(rows)

print(open("books.csv", encoding="utf-8").read()[:80])

Output:
Title,Price
A Light in the Attic,51.77
Tipping the Velvet,53.74
Soumission,50.1

A second site: quotes and tags

Selectors change from site to site, but the workflow does not. On quotes.toscrape.com each quote is a div.quote with an author in small.author and tags in a.tag.

r = requests.get("https://quotes.toscrape.com/", headers=HEADERS, timeout=10)
soup = BeautifulSoup(r.content, "html.parser")

for q in soup.select("div.quote")[:3]:
    author = q.select_one("small.author").get_text()
    tags = [t.get_text() for t in q.select("a.tag")]
    print(author, tags)

Output:
Albert Einstein ['change', 'deep-thoughts', 'thinking', 'world']
J.K. Rowling ['abilities', 'choices']
Albert Einstein ['inspirational', 'life', 'live', 'miracle', 'miracles']

Try it with AI

Ask for help when a scraper breaks after a site redesign, giving the old selector and the new HTML.

My BeautifulSoup scraper used soup.select("p.price_color") to read prices from a site I have permission to scrape, but the site was redesigned and now returns an empty list. Here is the new HTML for one product: <div class="card"><span data-testid="price" class="amt">£51.77</span></div>. Give me the new selector, explain why data-testid attributes are more stable than class names, and show how to log a warning instead of crashing when no price is found.

Common mistakes

Scraping without reading robots.txt and the terms of use. Many commercial sites forbid it, and personal data is protected by law.
Sending requests in a tight loop. Add time.sleep(), cache pages while developing and stop on a 429 response.
Using r.text and getting mangled characters such as Â£; pass r.content instead.
Calling .get_text() on the result of select_one() without checking for None, which raises AttributeError when the element is missing.
Expecting JavaScript-rendered content to appear in r.content. requests only sees the raw HTML; for dynamic pages look for the underlying API call or use a browser automation tool.

Exercise

Using the books list from the page-one example, count how many books have each rating (One to Five) and print the counts sorted by rating word. Then print the average price of the page rounded to two decimals.

Show answer

from collections import Counter

counts = Counter(b["Rating"] for b in books)
for rating in ["One", "Two", "Three", "Four", "Five"]:
    print(f"{rating}: {counts.get(rating, 0)}")

average = sum(b["Price"] for b in books) / len(books)
print("Average price:", round(average, 2))

Output:
One: 6
Two: 3
Three: 3
Four: 4
Five: 4
Average price: 38.05

Counts and the average reflect the twenty books on page one at the time of writing.

Related chapters

Python Requests and APIs – prefer an API whenever one exists.
Python CSV – reading and writing the files your scraper produces.
Python pandas – analyse scraped data with DataFrames.
Python with AI course hub – all 48 chapters in order.

FAQ

Is web scraping with Python legal?

Scraping publicly available data is legal in many places, but you must respect the site’s terms of use, its robots.txt file and data protection laws. Never scrape personal data, content behind a login or sites that forbid automated access, and prefer an official API when one exists.

What is the difference between requests and BeautifulSoup?

requests downloads the page and gives you the raw HTML. BeautifulSoup parses that HTML into a tree so you can search it with CSS selectors and read text and attributes. You normally use both together.

Why does my scraper return an empty list?

Usually the selector no longer matches the HTML, or the content is loaded by JavaScript after the page arrives, so it is not in the HTML requests receives. Print soup.prettify() to see what you actually downloaded.

Working with spreadsheets too? Ready-made Excel, Google Sheets and Power BI templates are at NextGenTemplates.com.

Chapter 36 of 48 · Python with AI: all 48 chapters