Web Scraping
Most data on the internet has no public API. Web scraping is the technique of extracting that data programmatically — loading pages, parsing HTML, and navigating dynamic JavaScript-rendered content. BeautifulSoup handles static pages. Playwright and Selenium handle the dynamic ones.
BeautifulSoup
Parse and extract data from HTML and XML. The simplest and most readable scraping library in Python.
Playwright
Browser automation for JavaScript-heavy sites. Control Chromium, Firefox, and WebKit programmatically.
Selenium
The original browser automation tool. Web scraping, UI testing, and automating any browser interaction.
Scrapy
The industrial-grade Python web scraping framework. Spiders, pipelines, middlewares, and distributed crawling.
Anti-Scraping Bypass
CAPTCHAs, rate limits, JavaScript challenges, and the techniques for scraping sites that try to block you.