Browser Automation

Browser automation

penguin browser drives the agent browser from your shell: the desktop app's built-in browser (the Browser tab of the dock), or the user's own Chrome through the PenguinHarness Browser extension — whichever the user chose. The commands are the same on both. Its tabs are shared by every conversation and keep the sign-ins made in them, so a page you open is one the user can watch and take over. Every command acts on the active tab unless --tab <id> names another.

You do not choose the browser, and must not try to (through /api/me/prefs or any other route): penguin browser status says which one is in use, backend: builtin or backend: chrome. In the user's Chrome you reach only the tabs in its Penguin tab group, which are the ones you open, and tabs the user hands over with the extension's icon; the user can take a tab back at any time.

Before you start

If the user's message only names this skill without a task, ask what they want done on the web. Then check the browser:

penguin browser status

status: available · backend: … is followed by the open tabs. status: unavailable (<reason>) (exit code 1) is followed by a note: saying what to tell the user. Tell them, and stop there — never answer from guessed page content:

• not_desktop, shell_unsupported, no_window: the built-in browser needs the PenguinHarness desktop app, up to date and open.
• extension_not_paired: no Chrome is paired. Ask the user to install the PenguinHarness Browser extension and pair it: Browser panel → Connect your Chrome….
• extension_disconnected: their Chrome is paired but not connected. Ask them to open Chrome with the extension enabled, or to pair it again in the Browser panel.
• extension_disabled: an admin turned Chrome connections off on this server; the user's Chrome cannot be used here.

A warning: line under memory: means the browser is using too much memory or holds too many tabs. Close the tabs you opened and no longer need (penguin browser close <tab-id>) before opening more, and reuse a tab (open without --new-tab) where you can.

The loop

1. Go: penguin browser open <url> navigates the active tab (and opens one when there is none); --new-tab opens another. It waits for the load and prints tab <id> · <title> · <url>.
2. Look: penguin browser scan --text first — plain text, cheap. penguin browser scan returns simplified HTML when you need selectors. Both print the tab header, the tab list (tabs: *12 Your Orders | 15 Google, the active tab starred), a --- rule, then the page.
3. Act: penguin browser exec runs JavaScript in the page. Put the script in a quoted heredoc and return exactly what you need, as compact JSON for anything structured.
4. Check: exec prints labelled lines — status: and tab:, return:, diff: (how many elements changed, the largest change indented beneath), transients: (messages that appeared and may be gone again, such as "Added to cart"), new tabs:, and a note:. When they do not settle whether it worked, scan again.

Prefer one precise exec over repeated scans: a scan costs thousands of tokens, a targeted exec a few dozen.

exec

penguin browser exec <<'EOF'
const cards = [...document.querySelectorAll('.result')].slice(0, 10);
return JSON.stringify(cards.map((c) => ({
  title: c.querySelector('h2')?.innerText.trim(),
  href: c.querySelector('a')?.href,
})));
EOF

• The value is the script's explicit return, or else its last expression: penguin browser exec 'document.title' prints the title. Top-level await works. Put an explicit return on its own last line: it is the one form that means the same in every script. A last line that is a statement (el.click();) gives return: undefined.
• Quote the heredoc delimiter (<<'EOF') so the shell leaves $, backticks and quotes alone. --file script.js works too, and so does a one-liner argument.
• return: is cut at 8000 characters. For more, add --save out.json: the whole value goes to the file (a string as-is, anything else as JSON) and only its start is printed. Then read the file.
• --no-monitor skips change tracking: faster, for scripts that only read.
• --timeout 60s for a slow script (the default is 15 s). Poll inside the script for content that loads late (see reference/page-recipes.md).
• status: failed with an error: line means the script threw; the exit code is 1.
• page: reloaded in the status line means the page navigated while the script ran, so its JavaScript context is gone. Scan or exec again on the new page.

Rules that save retries

• Navigating and acting on the new page are two calls. A script that sets location.href or clicks a link and then reads the page fails, because the page it was running in is gone. Navigate first (open, or an exec that only navigates), then act in the next exec.
• Never guess selectors. Scan first and take them from the HTML. Big sites generate their class names; prefer ids, name, aria-label, data-* attributes, roles and visible text.
• Scan shortens long lists to three items plus [FAKE ELEMENT] N more items hidden, selector: "…". Query that selector in an exec for the rest.
• Scan leaves out hidden, floating and covered elements (sidebars, overlays, closed menus). If something you expect is missing, look for it with exec (document.body.innerText, querySelectorAll).
• A scan that is empty or incomplete may be a page still rendering: wait a moment and scan again before concluding anything.
• Trusted input. An el.click() from JavaScript is an untrusted event that some sites ignore: buttons that open popups, custom dropdowns, file pickers. Use penguin browser click '<selector>' (--index n for the n-th match) or click --at x,y — a real mouse move, press and release. penguin browser type '<text>' --selector '<css>' --submit types for real and presses Enter.
• Setting a field from JavaScript needs the native value setter and an input event, or React and Vue will not notice (recipe in reference/page-recipes.md).
• Check disabled before clicking a button; a disabled button's click does nothing.
• Popups and target=_blank links open as new tabs: exec and click list them under new tabs:. Continue there with --tab <id> or penguin browser switch <id>.
• Dialogs are answered for you during exec, click and type: an alert is accepted; a confirm, a prompt or a leave-page dialog is dismissed and printed as dialog: confirm "…" → dismissed (rerun with --accept-dialogs to accept). Rerun with --accept-dialogs only when accepting is what the task asks for, never to confirm a purchase, a payment or a deletion the user has not approved.
• File uploads, cross-origin iframes and closed shadow roots: see reference/page-recipes.md (DataTransfer, and raw DevTools Protocol commands through penguin browser cdp).
• When layout matters or text is drawn in a canvas or an image, penguin browser screenshot -o shot.png (--full-page for the whole page), then look at the file.
• Verify figures on the detail page rather than a summary or a list: a list's total can differ from the order's.
• error: tab_crashed means the tab's page crashed (often from running out of memory). Retrying in that tab will not work: close it (penguin browser close <tab-id>) and open the page again with penguin browser open <url> --new-tab.
• error: tab_released (Chrome) means the user took that tab back from you in their Chrome. Never retry it: open a new tab (penguin browser open <url> --new-tab), or ask the user to hand the tab over again with the extension's toolbar icon.