Tech Guides · Automation

Web scraping with Python: how to collect data from any website

Informatics HubJuly 20267 min read

Every website you visit is made of structured HTML. Web scraping is the practice of reading that HTML programmatically and extracting the specific data you need from it. It is one of the most immediately useful Python skills you can learn because the applications are endless and the barrier to entry is surprisingly low.

Price monitoring, job listing aggregation, research data collection, lead generation, content archiving. All of these are built on the same core skill of being able to read a web page the same way a browser does and pull out exactly what you need.

How it works at a basic level

A web browser loads a page by making an HTTP request to a server, receiving HTML in response, and rendering it visually. Web scraping does the first two parts without the third. You make the same HTTP request with Python, receive the same HTML, and then parse it to find the data you want.

Every piece of text, price, title, or link you can see on a web page exists somewhere in the HTML source code. Web scraping is just reading that source code with a script instead of your eyes.

Web scraping reads the same HTML your browser renders and extracts structured data from it automatically

The two libraries you need

For most scraping tasks, two Python libraries cover everything you will ever need.

requests handles the HTTP part. It makes the GET request to a URL and returns the HTML content of the page as a string.

BeautifulSoup handles the parsing part. It takes that HTML string and gives you a clean way to search through it, find specific elements, and extract their text or attributes.

# Install both libraries first
# pip install requests beautifulsoup4

import requests
from bs4 import BeautifulSoup

# Step 1: Make the HTTP request
url = "https://example.com"
response = requests.get(url)

# Step 2: Parse the HTML
soup = BeautifulSoup(response.text, "html.parser")

# Step 3: Find elements and extract data
title = soup.find("h1").text
all_links = soup.find_all("a")

# Step 4: Loop through results
for link in all_links:
    print(link.get("href"), link.text)

Finding the right HTML elements

The most important skill in web scraping is knowing how to identify the specific element that contains the data you want. The way to do this is to right click on the data you want in your browser and select Inspect or Inspect Element. This opens the developer tools and highlights the exact HTML tag containing that content.

You can search by tag name, by CSS class, by ID, or by any combination of attributes. BeautifulSoup supports all of these through its find and find_all methods.

# Find by CSS class
price = soup.find("span", class_="product-price").text

# Find by ID
header = soup.find(id="main-header").text

# Find all items in a list
items = soup.find_all("li", class_="result-item")

# Get an attribute value
img_url = soup.find("img")["src"]

When scraping does not work

Some websites load their content through JavaScript after the initial page load. When you make a requests call, you get the HTML before the JavaScript runs, which means the data you want might not be there yet. For these cases you need a tool like Playwright or Selenium that actually controls a real browser and waits for JavaScript to execute before reading the page.

A note on scraping ethics

Before scraping any site, check its robots.txt file at domain.com/robots.txt to see what the site owner permits crawlers to access. Respect rate limits by adding delays between requests so you do not overload the server. Never scrape and republish copyrighted content. Most sites are fine with scraping for personal research or analysis but draw the line at commercial use of their data.

Key takeaways

• Web scraping reads HTML programmatically to extract structured data from websites
• The requests library fetches the page and BeautifulSoup parses the HTML
• Use your browser's Inspect tool to find the exact HTML elements containing the data you need
• For JavaScript-heavy sites, Playwright or Selenium can control a real browser instead