Selenium vs Puppeteer: which to use for scraping and browser automation

Should you use Selenium or Puppeteer?

Use Selenium if you want to automate a browser from Python or another non-JavaScript language, or if you must test across several browsers. Use Puppeteer if your project runs on Node.js and mostly targets Chrome, especially when you also need to generate PDFs. For many scraping jobs you need neither, because a plain HTTP request and an HTML parser are enough.

Selenium is a browser automation project that controls real browsers through the W3C WebDriver standard. Puppeteer is a Node.js library from the Chrome team that controls Chrome and Firefox. Both can run a browser with a visible window or in headless mode, where the browser runs without any window on screen.

I have used both in production. The no-code AI bot framework used Selenium, and Puppeteer was part of the stack for the AI-powered financial app.

When is a plain HTTP request enough?

A plain HTTP request is enough when the data you want is already in the HTML the server sends. Open the page, choose "View Page Source" in your browser, and search for the text you need. If it is there, fetch the page with an HTTP client and parse it.

In Python, that means requests or httpx plus Beautiful Soup or lxml. In Node.js, it means fetch plus Cheerio. This approach is much faster, uses far less memory, and is easier to run at scale than a real browser. Also check the browser's network tab: many sites load data from a JSON API, and calling that API directly is simpler than parsing any HTML.

When do you need a real browser?

A real browser is needed when a page builds its content with JavaScript after it loads, as most single-page apps do. It is also needed when you must click buttons, fill forms, log in, scroll to load more items, or take screenshots and PDFs.

Browser automation is heavier. Each browser instance uses a lot of memory and CPU, and pages take seconds to load. Plan for that when you run many jobs at once, and close browsers properly so they do not pile up on your server.

How do Selenium and Puppeteer compare?

The Chrome DevTools Protocol (CDP) is the low-level interface Chrome exposes for debugging tools. WebDriver BiDi is a newer standard that brings two-way, event-based control to all major browsers. Puppeteer uses CDP for Chrome and WebDriver BiDi for Firefox.

What does a Selenium example look like in Python?

This Selenium 4 example opens a page in headless Chrome, waits for a heading to appear, and prints its text. Install it with pip install selenium. Selenium Manager takes care of the driver.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")

driver = webdriver.Chrome(options=options)
try:
    driver.get("https://example.com")
    heading = WebDriverWait(driver, 10).until(
        EC.visibility_of_element_located((By.CSS_SELECTOR, "h1"))
    )
    print(heading.text)
finally:
    driver.quit()

WebDriverWait checks the condition repeatedly for up to 10 seconds and raises TimeoutException if the element never appears. The finally block makes sure the browser closes even when something fails.

What does a Puppeteer example look like?

This Puppeteer example does the same job in Node.js, then saves the page as a PDF. Install it with npm install puppeteer, which also downloads a compatible Chrome build. Save it as scrape.mjs so top-level await works.

import puppeteer from "puppeteer";

const browser = await puppeteer.launch();
try {
  const page = await browser.newPage();
  await page.goto("https://example.com", { waitUntil: "networkidle2" });
  await page.waitForSelector("h1");

  const heading = await page.$eval("h1", (el) => el.textContent.trim());
  console.log(heading);

  await page.pdf({ path: "page.pdf", format: "A4", printBackground: true });
} finally {
  await browser.close();
}

puppeteer.launch() starts Chrome in headless mode by default. page.$eval runs a function inside the page against the first matching element and returns the result to Node.js.

Why Puppeteer is popular for PDF generation

page.pdf() prints any web page to a PDF using Chrome's own print engine. That means you can design invoices, reports or term sheets in plain HTML and CSS, render them with your data, and export them as PDFs. Use printBackground: true to keep colors and backgrounds, and CSS @page rules to control margins and page size.

How should you handle waits and headless mode?

Wait for a specific condition, never a fixed amount of time. A time.sleep(5) is too slow when the page is fast and too short when the page is slow. In Selenium, use WebDriverWait with an expected condition. In Puppeteer, use page.waitForSelector(), page.waitForNetworkIdle() or page.waitForResponse() for the exact thing you need.

Headless mode is the normal choice on servers because there is no screen. Modern Chrome's headless mode renders pages almost exactly like a normal window. When a script fails and you cannot see why, run it with a visible browser or take a screenshot at the failing step.

How do you scrape politely and legally?

Scraping politely protects both the site and your project. Read the site's robots.txt file and respect the paths it disallows. Read the terms of service, because some sites forbid automated access or reuse of their content. If the site offers an official API, use it instead.

Keep your request rate low: add delays between pages, limit how many pages you load at once, and cache results so you never fetch the same page twice. Set an honest user agent that says who you are. Do not collect personal data you do not need, and never try to get around logins, paywalls or CAPTCHAs.

Summary

Start with a plain HTTP request and an HTML parser, and move to a browser only when the page needs JavaScript or interaction. Choose Selenium for Python and cross-browser work, and Puppeteer for Node.js, Chrome-focused scraping and PDF generation. Whichever you choose, use explicit waits and scrape politely. For help building a scraping or automation backend, see Python backend systems.