Browser Search

2. smart-extract — URL content extraction (default)

Goal: extract content from every useful URL returned by SearXNG.

smart-extract.mjs is the default tool for extracting content from any URL. It has 3 operation modes, to be used in this order:

1. Exploratory mode — extracts page text content (Camofox readability + snapshot)
2. Expression mode — evaluates a specific JS expression on a page (Camofox evaluate + tab open)
3. Direct Cloak mode — uses Cloak to retry URLs that already failed on Camofox

Workflow:

Always start with exploratory mode to read the page content. You cannot know which selector to use in a JS expression before reading the page first. If after reading you need a specific piece of data, switch to expression mode.

[!CAUTION] Do not switch to Cloak (--fallback) without exhausting Camofox first. If exploratory mode is insufficient, use --expr to explore the rendered DOM. Cloak is extremely slow — use it only as a last resort.

Base command (1, 2, or n URLs):

exec node <skill_dir>/scripts/smart-extract.mjs "<url1>" ["<url2>" ...]

Examples (modes 1 and 2):

# 1. Exploratory: read page content
exec node <skill_dir>/scripts/smart-extract.mjs "https://example.com"

# 2. Expression: extract specific data (only after reading the page)
exec node <skill_dir>/scripts/smart-extract.mjs "https://example.com" --expr "document.querySelector('h1')?.textContent"

Options:

Behavior (exploratory or expression mode):

• --expr "<js>" — evaluate a JS expression on the page
• --min-chars <n> — minimum character threshold for a valid result (default: 300, does not apply with --expr)

Cloak tuning (automatic escalation or --fallback):

• --wait <ms> — extra wait for JS rendering (default: 2000). Passed to Cloak
• --timeout <ms> — general timeout (default: 60000, Cloak receives 2x)
• --verbose — diagnostic logs on stderr. Useful for Cloak debugging

What happens in practice (modes 1 and 2):

• The script tries Camofox on all URLs. If at least one succeeds, Cloak is not used.
• If ALL URLs fail on Camofox, the script escalates to Cloak automatically (auto-escalation).
• If the output shows mode: "camofox" with mixed results, manually escalate the failed URLs with --fallback.
• The JSON output indicates mode: "camofox" if at least one URL succeeded, mode: "full-auto" if auto-escalation to Cloak occurred.

Last resort — Direct Cloak mode:

When Camofox failed and auto-escalation did not trigger (mixed mode). Accepts one or more URLs:

exec node <skill_dir>/scripts/smart-extract.mjs --fallback "https://url1" "https://url2" "https://url3"

--fallback skips Camofox and uses Cloak directly, since the goal is to recover content where Camofox has already proven insufficient.

Output: JSON on stdout with version, mode, results[] (one entry per URL with ok, content, chars, source, steps).

After reviewing the output, decide: do you have enough information to answer the user's request exhaustively? If yes, proceed. If not, go back to SearXNG with new search queries to fill the gaps.

3. Advanced operations (direct scripts)

For operations not covered by smart-extract.mjs, use the underlying tools directly. See <skill_dir>/data/advanced-browsing-reference.md for the full command reference.

Technical reference — Prerequisites & Docker containers

CloakBrowser prerequisite: after npx skills add you must run npm install inside the skill directory (<skill_dir>) and node -e "import('cloakbrowser').then(c=>c.ensureBinary())" — otherwise cloak-fetch.mjs and smart-extract.mjs --fallback / full-auto will fail with ERR_MODULE_NOT_FOUND: Cannot find package 'cloakbrowser'. Verify with bash <skill_dir>/scripts/check.sh (checks cloakbrowser npm package installed) or bash <skill_dir>/scripts/setup.sh.

Initial setup and diagnostics for SearXNG and Camofox. See <skill_dir>/docker/setup.md when needed.