Programmatic SEO Spy

Programmatic SEO Competitor Spy

Before building your own programmatic SEO pages, know what your competitors are already doing. This skill crawls competitor sitemaps, detects which URL patterns are programmatic, estimates which ones actually drive traffic, assesses template quality, and finds the gaps they're missing.

Core principle: The best pSEO strategy starts with competitive intelligence. If 3 competitors have /vs/ pages and none have /for-{industry}/ pages, that's a signal. This skill turns competitor site structures into strategic intelligence.

When to Use

• "What programmatic SEO are competitors doing?"
• "Reverse-engineer competitor SEO pages"
• "Which pSEO patterns actually work in our space?"
• "Find pSEO gaps our competitors are missing"
• "Analyze competitor URL structure for SEO patterns"

Tool Enhancement (Optional)

This skill works with existing capabilities but is dramatically better with domain analytics data that shows actual traffic per URL pattern.

Agent Prompt to User

"I can analyze competitor site structures with our existing crawling tools. For the most accurate results — especially knowing which patterns actually drive traffic vs. just existing — I'd recommend connecting a domain analytics API."

Recommended: DataForSEO (pay-per-use, ~$0.02-0.05 per domain analysis)

• Sign up at dataforseo.com → get API login + password
• Set DATAFORSEO_LOGIN and DATAFORSEO_PASSWORD env vars

Alternatives that also work:

• SEMrush API (if you have a subscription) → set SEMRUSH_API_KEY
• Ahrefs API (if you have a subscription) → set AHREFS_API_TOKEN
• SimilarWeb API (if you have access) → set SIMILARWEB_API_KEY

"Want to use one of these, or should I proceed with baseline mode? Baseline will detect patterns and assess template quality, but traffic estimates will be inferred rather than measured."

Mode Selection

• Enhanced mode — Domain analytics API provides organic keywords and estimated traffic per URL. Enables confident "this pattern drives X traffic" conclusions. Can show which specific pages within a pattern rank well.
• Baseline mode — Uses site-content-catalog for URL crawling, seo-domain-analyzer for domain-level metrics, web_search for spot-checking rankings. Pattern detection is equally good. Traffic attribution is directional, based on page count × domain authority × keyword indicators.

Phase 0: Intake

1. Competitors — 1-5 competitor URLs to analyze
2. Your product URL — So we can compare coverage
3. Category — What market/category are these competitors in?
4. Specific interest? — Looking for anything specific (vs/ pages, integrations, etc.) or full analysis?
5. Tool preference — Enhanced mode with domain analytics API, or baseline? (see Tool Enhancement above)

Phase 1: Sitemap Crawl

For each competitor, run site-content-catalog:

python3 skills/site-content-catalog/scripts/catalog_content.py \
  --url "<competitor_url>" \
  --output json

Collect:

• All URLs from sitemap and internal crawl
• Page titles
• URL structure/hierarchy
• Last modified dates (if available)

Phase 2: Pattern Detection

2A: URL Clustering

Group URLs by structural patterns using regex matching:

For each detected pattern:

• Page count — How many pages follow this pattern?
• Data axis — What varies per page? (competitor name, tool name, industry, etc.)
• URL consistency — Are URLs cleanly structured or messy?
• Coverage completeness — Have they covered the obvious variations?

2B: Programmatic vs. Editorial Classification

Not every URL pattern is programmatic. Classify each cluster:

• Programmatic — Clearly templated: consistent URL structure, similar page titles following a pattern, high page count, structured content
• Semi-programmatic — Template with heavy manual customization
• Editorial — Individually written, no template pattern
• Auto-generated — Thin/low-quality programmatic (tag pages, archive pages)

Focus analysis on Programmatic and Semi-programmatic clusters only.

Phase 3: Traffic & Performance Analysis

Enhanced Mode (DataForSEO / SEMrush / Ahrefs)

For each competitor domain, pull organic keywords data:

# DataForSEO example
POST /v3/dataforseo_labs/google/ranked_keywords/live
{
  "target": "competitor.com",
  "filters": [
    ["ranked_serp_element.serp_item.url", "contains", "/vs/"]
  ]
}

Per pattern cluster:

• Total ranking keywords across all pages in the pattern
• Estimated monthly organic traffic to the pattern
• Average ranking position for pages in this pattern
• Top-performing pages within the pattern (which variations rank best?)
• Keyword difficulty distribution (are they targeting easy or hard keywords?)