Web Scraping with Python: Extracting Web Data (BeautifulSoup + Selenium)

Web scraping is the technique of extracting data from websites automatically. With Python, you can collect prices, news, images, and any information available on the web programmatically.

Imagine needing to collect product prices from 100 different sites. Doing it manually would take hours. With web scraping, you automate this process in minutes!

Web scraping is useful for monitoring competitor prices, collecting data for analysis, aggregating news, and creating datasets for machine learning.

⚖️ Ethics and Legality

Before scraping, always check the robots.txt file and respect the Terms of Use. Do not overload servers with too many requests, and use delays between them. Avoid scraping sites that offer official APIs.

🍲 BeautifulSoup: Scraping Static Sites

BeautifulSoup is perfect for sites where the content is already in the HTML source code. It is fast and easy to use with the requests library.

import requests
from bs4 import BeautifulSoup

url = "https://quotes.toscrape.com"
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')

# Find elements by class
quotes = soup.find_all('span', class_='text')
for quote in quotes:
    print(quote.text)

🚗 Selenium: Scraping Dynamic Sites

Selenium is necessary when a site uses JavaScript to load content. Since BeautifulSoup does not execute JavaScript, it cannot access dynamically loaded data.

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
try:
    driver.get("https://example.com")
    # Wait for dynamic content
    element = driver.find_element(By.TAG_NAME, "h1")
    print(element.text)
finally:
    driver.quit()

🚀 Conclusion

Web scraping is a powerful tool that opens doors to data analysis and automation at scale. For more details on how to store the data you collect, check our guides on File Manipulation and Dictionaries. Use these tools responsibly and ethically! 🎯