Google Maps Scraper

Google Maps Scraper

Drive the local Google Maps scraper API to turn a business-type + location into clean, structured rows.

Mental model

The scraper runs as a local Docker container exposing a REST API at http://localhost:8080 (no auth — localhost only). A "scrape" is an async job: you create it, poll until it's done, then download a CSV. One job can run many keywords. Each result has up to 34 fields.

Step 0 — Make sure it's running

curl -s http://localhost:8080/api/v1/jobs >/dev/null 2>&1 && echo UP || echo DOWN

If DOWN: docker compose up -d (from the kit root), wait ~10s, retry. If Docker isn't installed, point the user to SETUP.md.

Step 1 — Create a job (POST /api/v1/jobs)

Required fields — the API returns 422 without them:

• keywords — array of search strings. Bake the location into each term: "plumbers in Denver CO".
• lat, lon — strings, the city's coordinates: "39.7392", "-104.9903".
• max_time — integer seconds (max wall-clock for the job), e.g. 300. (Sent as seconds; the API stores it as nanoseconds internally — just send seconds.)

Recommended fields:

• depth (default 10) — how far to scroll → roughly how many listings per keyword. Start at 5.
• lang "en", zoom 15 (city level), radius 10000 (meters), fast_mode false.
• email true — ON by default in this kit. Emails are the #1 lead field; the scraper visits each business website to find them (a bit slower). Only set false for a deliberately fast, no-email run.

curl -s -X POST http://localhost:8080/api/v1/jobs \
  -H "Content-Type: application/json" \
  -d '{"name":"job","keywords":["coffee shops in Austin TX"],"lang":"en","zoom":15,
       "lat":"30.2672","lon":"-97.7431","fast_mode":false,"radius":10000,
       "depth":5,"email":true,"max_time":300}'
# → {"id":"<uuid>"}   (HTTP 201; note: lowercase "id")

Two things to handle on every scrape:

1. Emails: on by default (email:true). Don't turn them off unless the user wants a fast run.
2. Socials: ASK first. Before creating the job, ask the user once whether they also want Instagram/Facebook/LinkedIn (see "Social profiles" below). Don't silently skip it.

Step 2 — Poll until done (GET /api/v1/jobs/{id})

The response field is "Status" (capital S): working → ok (success) or failed.

⚠️ Claude harness rule: the Bash tool blocks foreground sleep. Run the poll loop as a background Bash command (run_in_background: true) and read its output file when notified. Do NOT poll with a foreground sleep.

Background poll snippet (parses the value safely — match up to the closing quote, don't anchor on $):

ID="<uuid>"
for i in $(seq 1 40); do
  S=$(curl -s "http://localhost:8080/api/v1/jobs/$ID" | grep -oE '"Status":"[^"]*"' | head -1 | cut -d'"' -f4)
  echo "status=$S"
  [ "$S" = ok ] && { echo DONE; break; }
  [ "$S" = failed ] && { echo FAILED; break; }
  sleep 15
done

Step 3 — Download + parse (GET /api/v1/jobs/{id}/download)

curl -s "http://localhost:8080/api/v1/jobs/$ID/download" -o results.csv

CSV columns (34): input_id, link, title, category, address, open_hours, popular_times, website, phone, plus_code, review_count, review_rating, reviews_per_rating, latitude, longitude, cid, status, descriptions, reviews_link, thumbnail, timezone, price_range, data_id, place_id, images, reservations, order_online, menu, owner, complete_address, about, user_reviews, user_reviews_extended, emails.

⭐ Output ONLY money-useful lead fields (default)

The raw CSV has 34 columns and most are noise. By default, return ONLY these lead fields — the data you actually use to contact and qualify a lead — and drop everything else:

title (name), phone, emails, website, category, address, review_rating, review_count

DROP by default (do not show these unless the user explicitly asks): latitude/longitude (no use for outreach), link, plus_code, cid, data_id, place_id, open_hours, popular_times, reviews_per_rating, reviews_link, thumbnail, images, timezone, price_range, status, input_id, complete_address, reservations, order_online, menu, owner, about, descriptions, user_reviews, user_reviews_extended.

scripts/scrape.py already returns exactly this lead set (use --full to keep all columns, or --fields "a,b,c" to customize) and saves a CSV file by default (results-<id>.csv; pass --json for JSON). If you call the API directly, strip to the lead fields yourself before presenting — never dump the full 34-column row at the user.

Social profiles — ALWAYS ASK the user (Instagram / Facebook / LinkedIn)

Google Maps has no social links, so this is an enrichment: visit each business's website and regex out its IG/FB/LinkedIn URLs. Before scraping, ask the user once whether they want socials too (unless they already said). If yes → use the script: python3 scripts/scrape.py … --socials.

• Token cost — say this to the user when they ask for socials: the extraction itself is 0 LLM tokens (pure HTTP + regex in the script). It only adds ~40–50 tokens per business to your context if you load the rows into chat — e.g. ~+2k tokens for 50 leads. Negligible if you keep the file on disk and show a sample. Save the file; show a few rows.
• DO NOT fetch each website yourself with WebFetch to find socials — that reads every page into your context and costs thousands of tokens. The script does it for free. Always prefer --socials.
• It's opt-in/slower (one HTTP fetch per business) and coverage is partial (~40–70%: only businesses that link socials on their site; no website → no socials). Mention this if the user expects 100%.

Shortcuts (prefer these for common cases):

• One keyword: scripts/scrape.sh "<keyword>" <lat> <lon> [depth] (bash) — create→poll→download in one go.
• Auto-geocode (no coords): python3 scripts/scrape.py "<keyword>" --city "<City, ST>" [--depth N] — resolves lat/lon via OpenStreetMap Nominatim. You can also geocode the city yourself and pass coords. Emails come back by default (use --no-email to skip); add --socials if the user asked for socials.
• Batch (many keywords, ONE job): python3 scripts/scrape.py --keywords-file <file> --city "<City, ST>". The API takes a keywords array, so put all terms in a single job rather than firing many jobs. Do the manual curl flow only for custom job bodies (e.g. setting proxies).

Other endpoints

• GET /api/v1/jobs — list jobs. DELETE /api/v1/jobs/{id} — delete a job + free disk.
• Browser UI + OpenAPI docs: http://localhost:8080 and http://localhost:8080/api/docs.