Web Autopilot

Web Autopilot

Record once in any web app, let AI handle it from now on.

Overview

Record → Analyze → Confirm Fields → Generate → Test → Register as Tool

🎬 Record         User performs the workflow once in a real browser (after login)
🔍 Analyze        AI analyzes network traffic, classifies fixed/dynamic/session fields
✅ Confirm Fields  [Required for submit tasks] User confirms field classifications
📝 Generate       Generates reusable TS script + field mapping
🧪 Test           Iterative test loop, up to 5 rounds of auto-fix
🔧 Register       Register as an OpenClaw tool for direct invocation

Task Types

📊 Query / Export

Data extraction and report generation. Scripts run and output results automatically — no manual intervention needed. Examples: pull sales reports, extract project data, export revenue details

📝 Submit

Submit forms such as expense reports, travel requests, payment requests, etc. Each run requires dynamic parameters. Examples: submit travel request, submit expense report, submit payment request

The key challenge for submit tasks: correctly distinguishing which fields are fixed vs. which change every time, and confirming with the user before generating the script.

Skill Directory

~/.openclaw/rpa/
├── recordings/<task-name>/recording.json
├── tasks/<task-name>/
│   ├── task-meta.json
│   ├── run.ts
│   └── field-mapping.json
└── sessions/<domain>.session.json

Skill scripts: /opt/homebrew/lib/node_modules/openclaw/skills/web-autopilot/scripts/

Commands

1. record — Record a workflow

Ask user: task name, login URL or app URL.

cd /opt/homebrew/lib/node_modules/openclaw/skills/web-autopilot

# Option A: Start from login page (SSO, OAuth, username/password, etc.)
npx ts-node scripts/record.ts --name "my-task" --sso-url "https://login.example.com"

# Option B: Start directly from app (if already logged in or no login needed)
npx ts-node scripts/record.ts --name "my-task" --app-url "https://app.example.com"

Run in PTY mode (pty: true, background: true). User operates browser, types "done" when finished.

Note: --sso-url is a legacy parameter name; it works for any login URL (SSO, OAuth, plain login page, etc.).

2. analyze — Analyze the recording (AI does this)

Read recording.json, separate login traffic from business traffic, identify core APIs.

Key steps:

1. Read ~/.openclaw/rpa/recordings/<task>/summary.txt for overview
2. Parse recording.json to extract all API calls to app domain
3. For each POST/PUT/PATCH with meaningful body:
   • Classify fields: FIXED / DYNAMIC / SESSION / RELATIONAL
   • Detect protocol: rest-json / graphql / form-urlencoded / multipart
4. Map the complete API sequence (prerequisites → main operation → follow-ups)
5. Analyze ALL response fields and create field-mapping.json with human-readable labels
6. Create task-meta.json
7. [Submit tasks] After analysis, present the field classification confirmation table to the user (see below)

Field Classification

Field Analysis Rules (MANDATORY)

Every field must have a human-readable label. Including system-generated field names.

Inference priority:

1. Data value type: timestamp (10^12-13) / monetary amount (contextual) / enum (fixed values) / URL / JSON object
2. Field name pattern: *time/date/_at → datetime | *amount/*price/*cost → monetary | id/_key → ID | *status/*state → status
3. Business context: infer from related fields, API endpoint names
4. If uncertain → annotate as (unknown meaning: sample value)

Field Confirmation Step (MANDATORY for Submit tasks)

After analysis, you must present the following confirmation table to the user and wait for confirmation before generating the script:

📋 Field Classification Confirmation — <task name>

✅ FIXED (hardcoded):
  - approvalFlowId: "xxx"  → Approval Flow ID
  - companyId: "yyy"       → Company Entity
  - currency: "CNY"        → Currency

🔄 DYNAMIC (passed as parameters each run):
  - amount          → Amount (example: --amount 1500)
  - startDate       → Start Date (example: --startDate 2026-03-10)
  - endDate         → End Date (example: --endDate 2026-03-12)
  - destination     → Destination (example: --destination "New York")
  - reason          → Reason (example: --reason "Client visit")
  - attachments     → Attachment path (example: --attachments ~/Desktop/receipt.jpg)

🔗 RELATIONAL (auto-queried):
  - projectId       → Project ID (auto-looked up by project name, --projectName "Project X")

❓ Needs confirmation (AI uncertain):
  - field_abc123    → Unknown meaning (recorded value: "0"), suggest: FIXED("0") or DYNAMIC?

Please confirm the above classification or indicate any fields that need adjustment.

Only proceed to the Generate step after user confirmation.

CSV Export Rules (MANDATORY)

• Keep ALL fields, including hidden fields, dynamic fields, system fields — never crop
• Field order: preserve original order from data, never sort (sorting causes column misalignment)
• JSON/object fields → convert to JSON string for storage
• Use csv.writer + proper quoting to handle JSON fields containing commas

3. generate — Generate the task script

Pre-generation checklist (Query/Export tasks):

• ✅ All fields are in field-mapping.json
• ✅ All fields have human-readable labels
• ✅ CSV export uses field-mapping.json for column headers
• ✅ Field order preserves original order

Pre-generation checklist (Submit tasks):

• ✅ User has confirmed field classification (FIXED / DYNAMIC / RELATIONAL)
• ✅ All DYNAMIC fields converted to CLI parameters (with type, example value, required/optional)
• ✅ RELATIONAL fields have auto-query logic or corresponding parameters
• ✅ Script has --dry-run mode (prints request body without submitting, for testing)
• ✅ Script outputs submission result (success/failure + document number/link)

Submit task invocation example (written to task-meta.json usage field after generation):

# Preview (no actual submission)
npx ts-node run.ts --dry-run --amount 1500 --startDate 2026-03-10 ...

# Submit for real
npx ts-node run.ts --amount 1500 --startDate 2026-03-10 --destination "New York" --reason "Client visit"

4. test — Iterative test loop (max 5 rounds)

Run script → check output → if error: diagnose → fix → repeat.

5. run — Execute a registered task

npx ts-node ~/.openclaw/rpa/tasks/<task>/run.ts --param1 value1

6. list — List all tasks

npx ts-node /opt/homebrew/lib/node_modules/openclaw/skills/web-autopilot/scripts/run-task.ts --list

Session & Credential Management

Session (Cookie/Token Storage)

Sessions are cookie-based and work with any login method:

• SSO (OIDC, SAML, CAS, etc.)
• OAuth / OAuth2
• Username + password forms
• Any browser-based authentication

Session files: ~/.openclaw/rpa/sessions/<domain>.session.json

Credentials (Encrypted Storage)

Login credentials are stored encrypted (AES-256-GCM) in a separate file — never stored in plaintext.

File: ~/.openclaw/rpa/credentials.enc

• Encryption key = machine identity (hostname+username) + optional RPA_CREDENTIAL_KEY env var
• File permissions: 0600 (owner only)
• Supports automatic extraction and encrypted storage from recording.json

# Manage credentials
npx ts-node scripts/utils/credentials.ts list                    # List saved domains + usernames
npx ts-node scripts/utils/credentials.ts save <domain> <user> <pass>  # Save manually
npx ts-node scripts/utils/credentials.ts delete <domain>         # Delete
npx ts-node scripts/utils/credentials.ts extract <recording.json> # Extract from recording

Auto-Login Flow

When a session expires, the auto-login flow kicks in:

1. Read encrypted credentials for the target domain from credentials.enc
2. Select login strategy based on loginFlow.type
3. Launch browser (headless if credentials exist, headed if not)
4. Execute login steps → follow redirects → reach target app
5. Capture cookies/tokens → save new session
6. If all else fails → open headed browser for manual login (fallback)

Login Flow Types

When generating scripts, you must identify the login type from the recording and write it to the loginFlow field in task-meta.json: