Add Scraper

Add Scraper for a New Website

You are helping the user add support for scraping property listings from a new website. This is a multi-step process that requires creating a mapping file, capturing a test fixture, and wiring everything up.

Inputs

The user will provide $ARGUMENTS which should be a URL of a sample listing page from the target website, or just the website name. If no URL is provided, ask for one.

Step-by-step workflow

Phase 1: Gather information

1. Get a sample URL if the user didn't provide one
2. Determine the scraper name — derive from the hostname (e.g. www.example-realty.com → example_realty). Use lowercase, underscores for separators, keep it short. Ask the user to confirm the name.
3. Get the HTML — Ask the user how they want to provide the HTML:
   • If the site is static (no JS rendering needed), we can fetch it directly
   • If the site is JS-heavy (React/Vue/Angular SPA), the user should save the page from their browser ("Save As" → "Web Page, HTML Only") and provide the file path
   • They can also pipe it: curl ... | npm run capture-fixture -- --stdin --url <url>

Phase 2: Analyze the HTML and create the mapping

4. Capture the fixture using the capture-fixture utility:

   cd astro-app
   # For URL fetch:
   npm run capture-fixture -- <url> --name <scraper_name>
   # For local file:
   npm run capture-fixture -- --file <path> --url <url> --name <scraper_name> --no-extract

5. Analyze the saved HTML fixture at astro-app/test/fixtures/<name>.html. Look for:

   • Title: Usually in <h1>, <title>, or og:title meta tag
   • Price: Look for price elements, currency symbols, itemprop="price" microdata
   • Address/Location: Address strings, postal codes, og: meta tags, or structured data
   • Coordinates: Often in <script> tags as JSON (lat/lng), or in meta tags
   • Bedrooms/Bathrooms: Usually near icons or labels like "bed", "bath", "BR"
   • Images: <img> tags in gallery sections, or og:image meta tags
   • Reference/ID: Property ID in URL path, hidden inputs, or data- attributes
   • For sale/rent: Status indicators, URL patterns (/to-rent/, /for-sale/)
   • Description: meta[name=description] or content sections
   • Area/Size: Square footage/meters, usually near bed/bath counts

6. Create the mapping JSON at config/scraper_mappings/<name>.json. Follow the format described in the reference file at .claude/skills/add-scraper/reference.md.

   Key rules:

   • Each field must appear in exactly ONE section. If the same field appears in multiple sections, the last one processed wins (processing order: defaultValues → images → features → intFields → floatFields → textFields → booleanFields)
   • Always add cssCountId: "0" when a CSS selector might match multiple elements
   • Use cssAttr to read attribute values (e.g. content from meta tags)
   • Use scriptRegEx for data embedded in <script> tags
   • Use urlPathPart to extract segments from the URL path
   • Prefer class-based or ID-based selectors over nth-child chains