Browser Automation and Scraping: Playwright, Browser Use and the .har Trick

Summary

Most of the web has no public API, but every site is already an interface a human can drive, which means a program can drive it too. Playwright is the tool that does this: it controls a real browser the way a user would, clicking, typing and reading the page. On top of that sit two tricks that turn fragile scraping into reliable automation. The non-headless trick lets you log in by hand once and reuse the session. The .har trick lets you skip the rendered page entirely and call the API the site itself is calling. This lesson teaches all three, plus when to reach for an agent-driven browser and how to stay on the right side of ethics and the law.

What you will learn

You will learn Playwright basics, how agent-driven browsing (Browser Use and similar) differs from scripted automation, the visible-browser manual-login pattern that sidesteps the hardest part of authenticated sites, and how to record a .har file and let an agent extract the underlying API calls. You will also learn where the legal and ethical lines are and when to hand scale off to a provider like Brightdata.

Prerequisites

Courses 1 to 3 and a working Node project. You should be comfortable with async code, how HTTP requests and headers work, and reading your browser DevTools. The Fundamentals page on what an API is helps if request and response are still fuzzy, because the whole .har trick is about finding the API a page hides.

The problem

You need data or an action from a site that gives you no API. The naive approach is to scrape the rendered HTML with brittle selectors that break the moment the site changes a class name. The harder cases are sites behind a login, with bot detection, or that load everything through JavaScript so the HTML is empty until scripts run. People burn days fighting selectors and login flows when the site is quietly making clean JSON API calls the whole time, visible in DevTools, that they could call directly.

Playwright basics

Playwright launches and controls Chromium, Firefox or WebKit. You tell it to go to a URL, wait for elements, click, type and read text or attributes. It auto-waits for elements to be ready, which removes most of the flakiness that plagued older tools. The same library powers the end-to-end tests from your quality gates, so the muscle is reusable. Here is the shape of a basic script that loads a page and pulls some data.

import { chromium } from 'playwright'

const browser = await chromium.launch()
const page = await browser.newPage()

await page.goto('https://example.com/products')
// Auto-waits for the selector before reading
await page.waitForSelector('.product-card')

const titles = await page.$$eval('.product-card h2', (nodes) =>
  nodes.map((n) => n.textContent?.trim()),
)

console.log(titles)
await browser.close()

The non-headless manual-login trick

The hardest part of automating an authenticated site is the login: passwords, two-factor codes, captchas and bot detection all fight you, and hard-coding credentials into a script is both fragile and a security risk. The trick is to skip automating login entirely. Launch the browser non-headless (visible), navigate to the login page, and pause the script while you log in by hand. Once you are in, Playwright saves the authenticated session (cookies and storage) to a file, and every future run loads that file and is already logged in. No stored passwords, no captcha-solving, no brittle login flow.

import { chromium } from 'playwright'

// One-time: log in by hand, then save the session.
const browser = await chromium.launch({ headless: false }) // visible
const context = await browser.newContext()
const page = await context.newPage()

await page.goto('https://app.example.com/login')
console.log('Log in manually in the window, then press Enter here...')
// Wait for YOU to finish login before continuing
await page.waitForURL('**/dashboard', { timeout: 0 })

// Save the authenticated session for all future runs
await context.storageState({ path: 'auth.json' })
await browser.close()

// Every later run starts already logged in:
// const context = await browser.newContext({ storageState: 'auth.json' })

Keep auth.json out of Git (it is a live session token) and treat it like a secret. Sessions expire, so re-running the one-time login step occasionally is normal. This pattern is the single biggest reliability upgrade for authenticated automation.

Agent-driven browsing

Scripted Playwright is precise but brittle: change the page and your selectors break. Agent-driven browser tools like Browser Use take the opposite approach. You give an agent a goal in plain language ("find the cheapest flight from Zurich to London next Friday and add it to the cart") and it reads the page, decides what to click, and adapts when the layout differs from what it expected. It is slower and less deterministic, but it survives site changes and handles tasks you cannot fully specify in advance. The honest trade-off: use scripted Playwright for stable, high-volume, repeatable jobs where you control the target, and an agent-driven browser for one-off or fuzzy tasks where adaptability beats speed. Many real systems combine them - an agent to discover the flow once, then a generated script to run it cheaply at scale.

The .har trick

This is the highest-leverage move in the whole lesson. Before you scrape any rendered HTML, open DevTools, go to the Network tab, perform the action you want to automate, then right-click and "Save all as HAR". A .har file is a complete recording of every network request the page made, including the JSON API calls behind the scenes. Most modern sites render from clean internal APIs, which means the data you want is often sitting in a tidy JSON response you can call directly - no selectors, no headless browser, far faster and far more robust. The file is large and noisy, which is exactly where an agent shines: hand it the .har and ask it to extract the relevant endpoints.

## Task
Here is a .har file I recorded while loading the product list page.

Extract the underlying API the page uses to fetch products. I want:
1. The exact request URL and method.
2. Which headers actually matter (auth token, content-type) vs noise.
3. Query params or request body, with what each field appears to mean.
4. The shape of the JSON response (the fields I care about: name, price, id).
5. A minimal fetch() call in TypeScript that reproduces just that request.

Ignore analytics, fonts, images and tracking pixels. Focus only on the
call that returns the product data.

Once the agent gives you the fetch call, you have replaced a fragile browser scrape with a direct API call that is faster, cheaper and far more stable. If the API needs an auth token, you grab it from the non-headless session above. This combination - manual login for the token, .har trick for the endpoint - handles a huge fraction of real-world "this site has no API" problems.

Ethics, legality and scale

Automating the web is powerful and not consequence-free, so be deliberate. This section is guidance, not legal advice - when money or personal data is involved, talk to a lawyer.

• Read the terms of service and robots.txt. Some sites prohibit automated access, and ignoring that can breach a contract or, in some jurisdictions, computer-misuse law.
• Never scrape personal data without a lawful basis. The privacy laws from CLAUDE.md (GDPR, FADP, US state laws) apply to scraped data the same as any other.
• Rate-limit yourself and identify honestly. Hammering a site is abusive and gets you blocked; a polite, slow scraper is a better citizen and more reliable.
• For legitimate scale, use a provider like Brightdata. They handle proxy rotation, geo-distribution and compliance tooling, which is the responsible way to run large jobs rather than building a covert bot army.

Typical mistakes

The classics: scraping brittle HTML when a clean JSON API was sitting in the .har the whole time; hard-coding credentials instead of using the non-headless session trick; committing auth.json or a token to Git; running an agent-driven browser for a stable high-volume job where a cheap script would do; and ignoring terms of service and rate limits until you get blocked or worse. Check the Network tab before you write a single selector.

Business ROI

Browser automation unlocks data and actions that would otherwise require an integration the vendor never built. The .har trick alone can turn a multi-day scraping project into an afternoon, because you skip the rendered page and call the real API. The non-headless trick removes the most fragile part of any authenticated automation. Together they let a small team automate competitor research, internal data entry across legacy systems, and repetitive vendor-portal tasks that have no API - work that otherwise stays manual forever.

Checklist

Confirm you can do each of these before moving on, because the next lesson assumes you can run untrusted code safely.

• Write a basic Playwright script that loads a page and reads data.
• Use the non-headless trick to log in once and reuse the session headlessly.
• Explain when to use agent-driven browsing instead of a fixed script.
• Record a .har file and extract a replayable API call from it with your agent.
• State two legal or ethical limits you will always respect when scraping.

Resources

Keep the Playwright docs bookmarked for selector and auto-wait reference, and the Browser Use docs for agent-driven browsing. Your browser DevTools Network tab is the single most useful tool in this lesson - learn to record and save a HAR from it fluently. For large or compliance-sensitive jobs, Brightdata and similar providers are the documented, supported path.

Your task

Pick a site you use that has no API. Record a .har of the action you care about, hand it to your agent with the prompt above, and see whether you can reproduce the action with a direct fetch call. If the site needs a login, save a session with the non-headless trick first. Write down which approach (script, agent-driven, or direct API from the .har) you would ship.

Next lesson

Running code an agent generated - or that a scrape pulled in - on your own machine is risky. The next lesson covers sandboxes: isolated, disposable environments from E2B, Daytona and similar tools that contain a bad run so it cannot touch your system.