What you learn
- Automating the browser with Playwright and agent-driven browser tools
- The non-headless manual-login trick for sites behind authentication
- The .har trick: extracting the real API endpoints behind a page
Summary
Most of the web has no public API, but every site is already an interface a human can drive, which means a program can drive it too. Playwright is the tool that does this: it controls a real browser the way a user would, clicking, typing and reading the page. On top of that sit two tricks that turn fragile scraping into reliable automation. The non-headless trick lets you log in by hand once and reuse the session. The .har trick lets you skip the rendered page entirely and call the API the site itself is calling. This lesson teaches all three, plus when to reach for an agent-driven browser and how to stay on the right side of ethics and the law.
What you will learn
You will learn Playwright basics, how agent-driven browsing (Browser Use and similar) differs from scripted automation, the visible-browser manual-login pattern that sidesteps the hardest part of authenticated sites, and how to record a .har file and let an agent extract the underlying API calls. You will also learn where the legal and ethical lines are and when to hand scale off to a provider like Brightdata.
Prerequisites
Courses 1 to 3 and a working Node project. You should be comfortable with async code, how HTTP requests and headers work, and reading your browser DevTools. The Fundamentals page on what an API is helps if request and response are still fuzzy, because the whole .har trick is about finding the API a page hides.
Node.js is a runtime that lets you run JavaScript outside the browser. Learn what it is, why you need it, and how to install it in a few minutes.
An API is a way for two programs to talk to each other. Learn what an API is, how it works, and why it matters for building with AI.
The terminal is a text window where you type commands to control your computer. Learn the handful of commands you actually need to get started.
The problem
You need data or an action from a site that gives you no API. The naive approach is to scrape the rendered HTML with brittle selectors that break the moment the site changes a class name. The harder cases are sites behind a login, with bot detection, or that load everything through JavaScript so the HTML is empty until scripts run. People burn days fighting selectors and login flows when the site is quietly making clean JSON API calls the whole time, visible in DevTools, that they could call directly.
Comments
Loading comments.
Post a comment