Blog Cannibalization

Blog Cannibalization - Keyword Overlap Detection

Detect when multiple blog posts compete for the same search keywords. Two modes: local-only analysis (default) and DataForSEO API mode for SERP-level data.

Two Modes

Local mode works without any API keys. API mode requires DataForSEO credentials set as environment variables: DATAFORSEO_LOGIN and DATAFORSEO_PASSWORD.

Local Mode Workflow

Step 1: Scan Blog Files

Use Glob to find all content files in the target directory:

• Patterns: **/*.md, **/*.mdx, **/*.html
• Skip files in node_modules/, .git/, drafts/

Step 2: Extract Primary Keywords

For each file, read and extract keyword signals from:

• Title tag or H1 heading (highest weight)
• H2 headings (medium weight)
• First paragraph (supporting signal)
• Meta description if present in frontmatter

Primary keyword extraction method:

1. Tokenize title, H1, H2s, meta description, and first paragraph into 1-gram, 2-gram, and 3-gram phrases.
2. Normalize deterministically: lowercase, remove locale-aware stop words, lemmatize or stem consistently, preserve product names, and keep intent modifiers such as "best", "pricing", "vs", "review", "template", and year.
3. Score sections separately: title/H1 highest, meta description and H2s medium, first paragraph supporting.
4. Select the top-scoring 2-3 word phrase as the primary keyword and record secondary keywords from H2 headings.

Step 3: Cluster by Similarity

Group posts into clusters using these matching rules (in priority order):

1. Exact match - identical primary keyword across 2+ posts
2. Stem match - same root word (e.g., "optimize" vs "optimization")
3. Semantic overlap - Assign explicit intent labels such as informational, commercial, transactional, comparison, or troubleshooting. Include confidence and a one-sentence rationale, or use an embeddings workflow with a documented threshold.
4. Subset match - one keyword contains another (e.g., "email marketing" vs "email marketing for startups")

Step 4: Score and Flag

For each cluster with 2+ posts, assess severity and generate a recommendation.

Step 5: Output Report

Display the results table and per-cluster recommendations.

API Mode Workflow (DataForSEO)

Requires the --api flag and a dedicated local CLI wrapper that reads DATAFORSEO_LOGIN and DATAFORSEO_PASSWORD from the environment and emits JSON. Do not use WebFetch for DataForSEO POST calls and never expose Basic auth headers, login, password, or encoded credentials in prompts or reports. If no wrapper exists in the project, report SKIPPED: DataForSEO wrapper unavailable and run local mode.

Endpoints Used

Page Intersection - find keywords where multiple URLs rank:

POST https://api.dataforseo.com/v3/dataforseo_labs/google/page_intersection/live

{
  "pages": {
    "1": "https://example.com/post-a",
    "2": "https://example.com/post-b"
  },
  "language_code": "en",
  "location_code": 2840
}

Cost: ~$0.01 per call. Returns overlapping keywords with position, volume, CPC.

Ranked Keywords - get all keywords a single URL ranks for:

POST https://api.dataforseo.com/v3/dataforseo_labs/google/ranked_keywords/live

{
  "target": "https://example.com/post-a",
  "language_code": "en",
  "location_code": 2840
}

The wrapper sends DataForSEO auth headers from environment variables and never prints them.

API Analysis Steps

1. Collect all published URLs from the user (or sitemap)
2. Run Ranked Keywords for each URL to build keyword profiles
3. Run Page Intersection for URL pairs that share keyword clusters
4. Calculate severity using the formula below
5. Output enriched report with search volume and position data

Severity Scoring

Four severity levels based on overlap signals:

Severity Formula (API Mode)

severity_score = overlap_count x avg_search_volume x (1 / position_gap)

Where:

• overlap_count = number of shared ranking keywords
• avg_search_volume = mean monthly volume of shared keywords
• position_gap = absolute difference in average ranking position (min 1)

Higher score = more urgent cannibalization problem.

Severity Heuristic (Local Mode)

Without SERP data, use a simplified scoring:

• Critical: Exact primary keyword match between posts
• High: Stem match on primary keyword, or 3+ shared H2 keywords
• Medium: Semantic overlap on primary keyword
• Low: Subset match only, or shared secondary keywords