No Outlinks Audit

SKILL: Dead-End Pages — No Outgoing Internal Links Audit

DESCRIPTION

Finds every blog/content page on a site that has zero outgoing links to any URL on the same domain in its body content. These pages absorb link equity but pass none on — they are structural dead-ends that silently suppress topical authority and hurt crawl efficiency across the site.

For each dead-end page the skill generates 3 suggestions: pages on the same site that the dead-end page SHOULD link out to, including the exact anchor text (the target page's top keyword), where in the dead-end page to place the link, and a ready-to-paste context copy sentence with the link already embedded.

This skill is the structural inverse of the Orphan Pages skill:

• Orphan Pages: pages with no incoming internal links (nobody links TO them)
• Dead-End Pages: pages with no outgoing internal links (they link out to nobody)

TOOLS

• Step 1 uses curl via bash_tool for sitemap discovery
• Step 2 uses curl via bash_tool for framework detection
• Step 3 uses curl via bash_tool for outgoing link detection (method varies by framework)
• Step 5 uses Ahrefs MCP (site-explorer-top-pages) for keyword research

WORKFLOW

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ STEP 1 — DISCOVER ALL PAGES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

WHY THIS STEP EXISTS You need a complete list of all blog/content pages to check. Curl-based sitemap discovery is fast, free, and reflects the live site — not a stale crawl cache.

SUBSTEP 1A — FIND THE SITEMAP

Run in order until one returns valid XML:

curl -sL --max-time 15 "https://www.DOMAIN.com/sitemap.xml" | head -50
curl -sL --max-time 15 "https://www.DOMAIN.com/sitemap_index.xml" | head -50
curl -sL --max-time 15 "https://www.DOMAIN.com/robots.txt" | grep -i sitemap

If none work, crawl the blog index directly:

curl -sL "https://www.DOMAIN.com/blog/" | grep -oP 'href="[^"]*"' | grep '/blog/'

SUBSTEP 1B — EXTRACT ALL CONTENT URLS

curl -sL "https://www.DOMAIN.com/sitemap.xml" \
  | grep -oP '(?<=<loc>)[^<]+' \
  | grep '/blog/' \
  | grep -v '/blog/$' \
  | grep -v '/page/' \
  | sort -u > /tmp/blog_urls.txt

wc -l /tmp/blog_urls.txt

Call this LIST_ALL. Record TOTAL_PAGES = length of LIST_ALL.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ STEP 2 — DETECT SITE FRAMEWORK ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

WHY THIS STEP EXISTS The detection method for outgoing links depends entirely on how the site renders HTML. Running the wrong method gives completely wrong results — this step prevents that.

Run both checks:

# Check response headers
curl -sI "https://www.DOMAIN.com" | grep -i "x-powered-by\|x-nextjs\|x-vercel\|server\|generator"

# Check HTML source for framework signals
curl -sL "https://www.DOMAIN.com" \
  | grep -o 'name="generator"[^>]*\|__NEXT_DATA__\|_next\|__nuxt\|ng-version\|gatsby'

DECISION TABLE:

IMPORTANT: Always test the detection method on 3 sample pages before running the full list. Verify that a page you know has links returns links, and a page you know is sparse returns few/none. If results look wrong, re-read this step.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ STEP 3 — DETECT OUTGOING INTERNAL LINKS PER PAGE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Run the method determined in Step 2.

──────────────────────────────────────── STEP 3A — NEXT.JS SITES (RSC HEADER METHOD) ────────────────────────────────────────

HOW IT WORKS Next.js with React Server Components exposes a server-rendered payload endpoint. By sending RSC: 1 as a request header, the server returns the full rendered page data as a serialised React component tree — including all links — without needing a headless browser. This works on all Next.js sites using the App Router (Next.js 13+).

This is the correct method for any site running on Next.js / Vercel, regardless of whether it uses ISR, SSG, or SSR. The RSC payload always contains the rendered content with all internal links visible as plain text paths.

HOW TO VALIDATE THE FETCH WORKED A valid RSC response:

• Is always larger than 10KB (most blog posts return 50KB–250KB)
• Starts with React markers like $Sreact.fragment or J: or 1:"$
• If a page returns less than 10KB or has no $ in the first 500 chars, the fetch failed — flag that page for manual review, do NOT mark it as a dead-end

import subprocess, re, time

domain = "DOMAIN.com"        # without www, e.g. "infrasity.com"
blog_prefix = "/blog/"       # content prefix, e.g. "/blog/"

with open('/tmp/blog_urls.txt') as f:
    urls = [u.strip() for u in f if u.strip()]

# Asset extensions to exclude — these are not outgoing content links
SKIP_EXT = (
    '.png','.webp','.jpg','.jpeg','.gif','.svg','.ico',
    '.woff2','.woff','.ttf','.eot','.otf',
    '.css','.js','.jsx','.ts','.tsx',
    '.xml','.txt','.pdf','.zip','.mp4','.mp3','.webm'
)

# Path prefixes to exclude — assets, CDN, internal Next.js paths
SKIP_PREFIX = (
    '/PostImages/', '/images/', '/fonts/', '/favicon',
    '/_next/', '/_vercel/', '/static/',
    '/api/', '//cdn.', '//assets.'
)

def get_outgoing_links(url):
    result = subprocess.run([
        'curl', '-sL', '--max-time', '10',
        '-H', 'RSC: 1',
        '-H', 'Next-Router-State-Tree: %5B%22%22%2C%7B%22children%22%3A%5B%22__PAGE__%22%2C%7B%7D%5D%7D%2Cnull%2Cnull%2Ctrue%5D',
        url
    ], capture_output=True, text=True, timeout=15)

    data = result.stdout

    # Validate RSC fetch succeeded
    if len(data) < 10000 or not any(m in data[:500] for m in ['$S', 'J:', '1:"']):
        return None  # Fetch failed — flag for manual review

    # Extract all paths pointing to the same domain
    # Extract both absolute domain links and relative paths starting with /
    # (excluding protocol-relative links starting with //)
    raw = re.findall(
        rf'(?:(?:https?://)?(?:www\\.)?{re.escape(domain)}|(?<!/))/(?!/)([^\\s\"\\\'\\\\),\\]]+)',
        data
    )
    raw = ['/' + r for r in raw]

    clean = set()
    for l in raw:
        # Normalise: strip trailing junk
        l = l.split('\\')[0].split('"')[0].split("'")[0].rstrip('),.]\\/')
        if len(l) < 2:
            continue
        # Exclude assets
        if any(l.lower().endswith(e) for e in SKIP_EXT):
            continue
        if any(l.startswith(p) for p in SKIP_PREFIX):
            continue
        # Exclude self-link
        slug = url.rstrip('/').split('/')[-1]
        if l.rstrip('/') == blog_prefix.rstrip('/') + '/' + slug:
            continue
        clean.add(l)

    return clean

dead_ends = []
has_outlinks = []
failed = []

for i, url in enumerate(urls):
    slug = url.split('/')[-1]
    try:
        links = get_outgoing_links(url)
        if links is None:
            failed.append(url)
            print(f"[{i+1:3d}] FAILED   {slug}")
        elif len(links) == 0:
            dead_ends.append(url)
            print(f"[{i+1:3d}] DEAD-END {slug}")
        else:
            has_outlinks.append(url)
            print(f"[{i+1:3d}] LINKS({len(links):2d}) {slug}")
    except Exception as e:
        failed.append(url)
        print(f"[{i+1:3d}] ERROR    {slug}: {e}")
    time.sleep(0.15)

with open('/tmp/dead_end_urls.txt', 'w') as f:
    f.write('\n'.join(dead_ends))

print(f"\nTotal:         {len(urls)}")
print(f"Dead-ends:     {len(dead_ends)}")
print(f"Has outlinks:  {len(has_outlinks)}")
print(f"Failed:        {len(failed)} (verify manually)")
print(f"Dead-end rate: {round(len(dead_ends)/len(urls)*100)}%")

──────────────────────────────────────── STEP 3B — STATIC / WORDPRESS / TRADITIONAL CMS (STANDARD CURL) ────────────────────────────────────────

HOW IT WORKS For sites that serve fully-rendered HTML directly (static files, WordPress, Gatsby, traditional CMS), standard curl fetches the complete page including all links. Parse only the main content area to exclude nav/footer/sidebar links.

import subprocess, re, time
from urllib.parse import urljoin

domain = "DOMAIN.com"
base = "https://www." + domain

with open('/tmp/blog_urls.txt') as f:
    urls = [u.strip() for u in f if u.strip()]

SKIP_EXT = (
    '.png','.webp','.jpg','.jpeg','.gif','.svg','.ico',
    '.woff2','.css','.js','.xml','.pdf','.zip'
)

def get_outgoing_links(url):
    result = subprocess.run([
        'curl', '-sL', '--max-time', '10',
        '-A', 'Mozilla/5.0 (compatible; InternalLinkBot/1.0)',
        url
    ], capture_output=True, text=True, timeout=15)

    html = result.stdout

    # Extract main content area only
    content = ''
    for tag in ['article', 'main', 'body']:
        m = re.search(rf'<{tag}[^>]*>(.*?)</{tag}>', html, re.DOTALL | re.IGNORECASE)
        if m:
            content = m.group(1)
            break
    if not content:
        content = html

    hrefs = re.findall(r'<a\\s+[^>]*href\\s*=\\s*["\']([^"\'\\ ]+)["\']', content, re.IGNORECASE)

    clean = set()
    for href in hrefs:
        href = href.strip()
        if not href or href.startswith('#') or href.startswith('mailto:') or href.startswith('tel:'):
            continue
        abs_url = urljoin(base, href)
        if domain not in abs_url:
            continue
        path = '/' + abs_url.split(domain, 1)[-1].lstrip('/')
        if any(path.lower().endswith(e) for e in SKIP_EXT):
            continue
        if abs_url.rstrip('/') == url.rstrip('/'):
            continue
        clean.add(path)
    return clean

dead_ends = []
has_outlinks = []

for i, url in enumerate(urls):
    slug = url.split('/')[-1]
    links = get_outgoing_links(url)
    if links:
        has_outlinks.append(url)
        print(f"[{i+1:3d}] LINKS({len(links):2d}) {slug}")
    else:
        dead_ends.append(url)
        print(f"[{i+1:3d}] DEAD-END {slug}")
    time.sleep(0.2)

with open('/tmp/dead_end_urls.txt', 'w') as f:
    f.write('\n'.join(dead_ends))

print(f"\nTotal:         {len(urls)}")
print(f"Dead-ends:     {len(dead_ends)}")
print(f"Has outlinks:  {len(has_outlinks)}")
print(f"Dead-end rate: {round(len(dead_ends)/len(urls)*100)}%")

──────────────────────────────────────── STEP 3C — UNKNOWN / PURE SPA — FLAG TO USER ────────────────────────────────────────

If framework detection is inconclusive (e.g. Nuxt.js, Angular, Vue SPA) or if curl returns fewer than 10 <a> tags across 3 sample pages, stop and tell the user:

"This site appears to use client-side rendering that cannot be detected through standard curl requests. To run this audit accurately, one of the following is needed:

1. Set up an Ahrefs Site Audit project for this domain (uses a headless browser that renders JS). Share the project ID once the first crawl completes.
2. Export a crawl from Screaming Frog with JavaScript rendering enabled and provide the CSV of internal links.
3. Confirm the exact framework being used so the right detection header can be applied.

Do not proceed to generate a report until the detection method is confirmed working on at least 5 sample pages."

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ STEP 4 — VALIDATE RESULTS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

WHY THIS STEP EXISTS Detection bugs are silent. A misconfigured asset filter or a failed fetch can silently produce a wrong dead-end list. This step catches that before suggestions are generated.

Do ALL of the following before proceeding:

1. Pick 3 random pages from LIST_DEAD_ENDS. Open each in a browser and manually confirm the page genuinely has no outgoing links to the domain. If any DO have links, re-check the detection script's asset filter and domain pattern.

2. Pick 3 pages from LIST_HAS_OUTLINKS. Verify they have visible links in the browser to other pages on the same domain.

3. If DEAD_END_RATE > 80%, something is likely wrong — the fetch may be returning only asset links or the domain pattern may not be matching. Debug before continuing.

4. If DEAD_END_RATE is 0%, the filter may be too loose (counting assets or self-links). Check what links are being detected on known-sparse pages.

5. For Step 3A (Next.js): if the failed list is non-empty, manually check those pages. Some may genuinely be dead-ends; others may have errored. Re-run failed pages individually before finalising the list.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ STEP 5 — KEYWORD RESEARCH ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

WHY THIS STEP EXISTS To generate relevant outgoing link suggestions you need to know what each dead-end page AND each potential target page is about. One Ahrefs call covers the full blog.

TOOL: Ahrefs:site-explorer-top-pages

PARAMETERS:

• target: blog prefix (e.g. www.DOMAIN.com/blog/)
• mode: prefix
• date: today's date YYYY-MM-DD
• select: url,top_keyword,sum_traffic
• limit: 500
• order_by: sum_traffic:desc

RESULT: MAP_KEYWORDS — a keyword and traffic map for all pages.

For any dead-end page not in MAP_KEYWORDS, derive keyword from URL slug: replace hyphens with spaces. NEVER fabricate traffic numbers — use null.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ STEP 6 — CLUSTER AND GENERATE SUGGESTIONS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

WHY THIS STEP EXISTS Outgoing link suggestions must be topically relevant to the dead-end page. Clustering first, then matching within and across clusters, produces natural and useful suggestions.

HOW TO CLUSTER Group all dead-end pages by primary topic using their keyword and title. Create 5–15 clusters depending on the site's content breadth. Adapt cluster names to the actual content — do not use generic names like "Misc" or "Other" unless truly necessary.

HOW TO GENERATE SUGGESTIONS For each dead-end page, identify 3 pages from LIST_ALL (any page on the site, not only other dead-ends) that:

1. Are topically related — same or adjacent subject matter
2. Would be a natural next step for a reader of the dead-end page
3. Are ideally already well-linked pages — so the dead-end connects into the broader internal link graph

For each of the 3 suggestions, provide:

ANCHOR TEXT The top keyword of the TARGET page (the page being linked TO). Rules:

• Use the kw field from MAP_KEYWORDS for the target page
• If target has no keyword data, derive from its URL slug
• Must read naturally as hyperlinked text in a sentence
• No em dashes (—) in or around the anchor text

WHERE TO PLACE A specific section-level description of where in the DEAD-END page to insert the outgoing link. Rules:

• Section-level specific, not just "somewhere in the article"
• Contextually logical — where a reader would naturally want to go deeper
• Examples: "In the intro when defining the core concept" "After the comparison table when suggesting next steps" "In the Tools and Resources section at the end" "When first mentioning [adjacent topic] mid-article"

SUGGESTED CONTEXT COPY A ready-to-paste sentence (1–2 sentences max) to drop directly into the dead-end page at the placement location. Rules: