Cashclaw Data Scraper

{
  "$schema": "extraction-schema-v1",
  "source": "{source_url}",
  "description": "{what this dataset contains}",
  "fields": [
    {
      "name": "company_name",
      "type": "string",
      "required": true,
      "description": "Legal company name"
    },
    {
      "name": "website",
      "type": "url",
      "required": true,
      "description": "Company website URL"
    },
    {
      "name": "industry",
      "type": "string",
      "required": false,
      "description": "Primary industry category"
    },
    {
      "name": "employee_count",
      "type": "integer",
      "required": false,
      "description": "Approximate employee count"
    },
    {
      "name": "location",
      "type": "string",
      "required": false,
      "description": "Headquarters city, state/country"
    }
  ],
  "dedup_key": "website",
  "sort_by": "company_name",
  "filters": {
    "industry": "{filter_value}",
    "min_employees": 10
  }
}

# If the source has a public API
curl -s "https://api.example.com/v1/companies?industry=saas&limit=50" \
  -H "Accept: application/json" | jq '.data[]' > raw-data.json

# Fetch the page
curl -sL "https://example.com/directory?page=1" -o page.html

# Parse with node script
node scripts/scraper.js --url "https://example.com/directory" --pages 5 --output raw-data.json

# Extract JSON-LD, microdata, or Open Graph from pages
node scripts/extract-structured.js --url "https://example.com" --format jsonld

Pagination Config:
  Pattern: "{query_param | path | cursor | link_header}"
  Base URL: "{url}"
  Page Param: "page={n}"
  Records Per Page: 20
  Total Pages Needed: 3
  Delay Between Requests: 1500ms
  Stop Condition: "empty results OR target count reached"

Cleaning Pipeline:
  1. Remove Duplicates:
     - Deduplicate on primary key (e.g., website domain)
     - If two records share the same key, keep the more complete one

  2. Normalize Fields:
     - URLs: Add https:// if missing, remove trailing slashes
     - Phone: Standardize to E.164 format (+1XXXXXXXXXX)
     - Email: Lowercase, trim whitespace
     - Company Names: Trim, normalize casing (Title Case)
     - Locations: Standardize to "City, State, Country" format

  3. Validate Data Types:
     - URLs: Must start with http:// or https://
     - Emails: Must match RFC 5322 pattern
     - Numbers: Must be numeric (remove currency symbols, commas)
     - Dates: Normalize to ISO 8601

  4. Handle Missing Data:
     - Required fields missing: Flag record for review or discard
     - Optional fields missing: Set to null, not empty string
     - Never fabricate data to fill gaps

  5. Quality Score:
     - Calculate completeness percentage per record
     - Flag records below 60% completeness for review

Enrichment Sources:
  Company Data:
    - Employee count from LinkedIn company page
    - Industry classification from website metadata
    - Tech stack from BuiltWith or Wappalyzer signals
    - Social media profiles from website footer links

  Contact Data:
    - Email pattern detection (first@, first.last@, firstl@)
    - LinkedIn profile URLs from company team page
    - Phone from website contact page

  Business Signals:
    - Recent funding (Crunchbase, press releases)
    - Job openings count (careers page, job boards)
    - Website traffic estimate (if observable)
    - Social media activity level

{
  "company_name": "Acme Corp",
  "website": "https://acme.com",
  "enriched": {
    "employee_count": {
      "value": 85,
      "source": "linkedin",
      "confidence": "high",
      "date": "2026-03-15"
    },
    "tech_stack": {
      "value": ["React", "Node.js", "AWS"],
      "source": "website_analysis",
      "confidence": "medium",
      "date": "2026-03-15"
    }
  }
}