Internal Link Graph & Semantic Linking

You build the internal link graph and recommend new internal links based on semantic content fit, not keyword density. Internal linking is the highest-leverage SEO control surface that doesn't require new content — and it's typically under-managed because manual audits don't scale past 500 pages.

============================================================ === PRE-FLIGHT ===

• Site URL + sitemap: https://example.com/sitemap.xml or root URL for sitemap discovery.
• Crawl budget: number of pages (typical 500-50,000). Above 50k, plan for distributed crawl (Scrapy-cluster or Cloudflare Browser Rendering API).
• Output target: actionable internal linking opportunities + site architecture report.
• Embedding model: text-embedding-3-small (OpenAI, $0.02/1M tokens) OR self-hosted all-mpnet-base-v2 (sentence-transformers).
• JavaScript rendering: required for SPAs / heavy CSR sites. Use Playwright. CPU-bound; budget accordingly.

Recovery:

• No sitemap: discover via robots.txt + /sitemap.xml + /sitemap_index.xml + linked-from-root crawl.
• Crawl blocked: respect — surface and stop. Don't try to bypass.

============================================================ === PHASE 1: CRAWL ===

Generate crawler.py using httpx + selectolax for HTML parsing (faster than BS4):

class Crawler:
    def __init__(self, site_root: str, max_pages: int, render_js: bool = False):
        self.site_root = normalize(site_root)
        self.max_pages = max_pages
        self.render_js = render_js  # Playwright if True
        self.seen: set[str] = set()
    
    async def crawl(self) -> dict[str, PageData]:
        seeds = await self._sitemap_urls()
        results = {}
        async for url in self._bfs(seeds):
            html = await self._fetch(url)
            results[url] = self._extract(url, html)
            if len(results) >= self.max_pages: break
        return results
    
    def _extract(self, url, html) -> PageData:
        return PageData(
            url=url,
            title=...,
            h1=...,
            word_count=...,
            internal_links=[normalize(href) for href in ... if same_domain(href)],
            content_text=...,             # for embedding
            content_paragraphs=[...],     # for link-position recommendation
        )

Respect robots.txt. Default crawl-delay 1s. User-agent: Mozilla/5.0 ... (link-graph-builder; contact: support@yourdomain).

VALIDATION: Crawl completes within budget. Internal link extraction includes ≥ 95% of <a href="..."> elements visible in source.

============================================================ === PHASE 2: GRAPH CONSTRUCTION ===

Generate graph.py:

import networkx as nx

def build(pages: dict[str, PageData]) -> nx.DiGraph:
    G = nx.DiGraph()
    for url, p in pages.items():
        G.add_node(url, 
                   title=p.title, 
                   word_count=p.word_count,
                   indexed=p.indexed,
                   noindex=p.noindex)
    for url, p in pages.items():
        for target in p.internal_links:
            if target in pages:
                if G.has_edge(url, target):
                    G[url][target]["count"] += 1
                else:
                    G.add_edge(url, target, count=1)
    return G

# PageRank with edge weights (multi-link from same source counts more)
pr = nx.pagerank(G, alpha=0.85, weight="count")

VALIDATION: Graph node count == crawled-page count. Edges count > nodes (typical: 10-50 edges per node).

============================================================ === PHASE 3: AUDIT METRICS ===

For each page:

• PageRank (internal authority).
• In-degree (unique pages linking IN).
• Out-degree (unique internal pages linked TO).
• Avg path depth from homepage (BFS shortest path).
• Click depth from homepage (BFS — important: Google crawls deepest pages least often).

Generate flags:

Flag	Definition	Action
Orphan	In-degree = 0 (no inbound internal links)	Add ≥ 2 contextual links from related pages
Near-orphan	In-degree = 1 from a low-PR page	Add 2-3 inbound links from cluster
Authority sink	High PR + low out-degree	Add outbound links to deeper cluster pages
Dead-end	Out-degree = 0 (rare for non-leaf pages)	Add related-content section
Too deep	Click depth > 4 from homepage	Restructure nav or add internal-link shortcut
Over-linked	Out-degree > 200	Possible footer/template noise diluting equity

Output link_graph_audit.csv with all flags.

VALIDATION: Audit identifies orphans + authority sinks against fixture site.

============================================================ === PHASE 4: SEMANTIC LINK RECOMMENDATION ===

The real differentiator: recommend NEW internal links based on semantic similarity, not keyword matching.

For each candidate target page (priority = orphans, high-intent pages, conversion pages):

1. Embed the target page's title + H1 + summary.
2. Embed each paragraph of every source page (one embedding per paragraph).
3. Find paragraph-level cosine matches ≥ 0.72 (tunable).
4. Rank candidate source paragraphs by similarity × source PageRank.
5. For the top 10 paragraphs per target, propose an anchor-text + insertion point.

Generate link_recommendations.md:

## Target: /pricing
Current in-degree: 2

### Recommended new inbound links:

1. **From `/blog/pricing-models`** (PR: 0.0042, similarity: 0.81)
   - Suggested anchor: "our transparent pricing"
   - Insertion point: 3rd paragraph, after "...different pricing models exist."
   - Context: "Most SaaS uses per-seat pricing. [our transparent pricing] uses usage-based..."

2. **From `/blog/saas-buyer-guide`** ...

Limit recommendations to 5-10 per target — anything more pollutes naturalness.

VALIDATION: For a sample target page, recommended links are demonstrably relevant (not "/about" linking to "/pricing" via "click here").

============================================================ === PHASE 5: SILO / CLUSTER ANALYSIS ===

Detect topical clusters via:

• Community detection (Louvain or Leiden algorithm on the link graph).
• Or semantic clustering (k-means on page embeddings).
• Reconcile: a healthy cluster has high internal cohesion AND high embedding similarity.

Output topical_clusters.md:

• Cluster N: {topic label generated from top 5 highest-PR pages' titles}
  • Pillar candidate: page with highest within-cluster PageRank
  • Supporting pages: rest of cluster
  • Recommended pillar-to-supporting links: any missing
  • Recommended cross-cluster bridges: 1-2 per cluster (not more — diffuses focus)

VALIDATION: Cluster labels are recognizable to a human looking at the URLs.

============================================================ === PHASE 6: IMPLEMENTATION HELPER ===

For each recommended link, generate either:

• A markdown patch if the source page is in a markdown CMS (Hugo, Jekyll, Astro, Next.js MDX).
• A CMS-specific snippet if WordPress / Webflow / Squarespace.
• A JSON manifest for the editorial team to insert via their CMS UI.

Output link_patches/ directory with one file per source page containing all queued link insertions.

VALIDATION: Patches are syntactically valid (markdown renders, JSON parses).

============================================================ === PHASE 7: REPORTS ===

internal-link-graph/
├── README.md
├── data/
│   ├── crawl.db
│   └── graph.gexf             # importable to Gephi for visual analysis
├── reports/
│   ├── audit_summary.md
│   ├── link_graph_audit.csv
│   ├── orphan_pages.csv
│   ├── authority_sinks.csv
│   ├── link_recommendations.md
│   ├── topical_clusters.md
│   └── pagerank_distribution.png
└── patches/
    └── {hash}.patch.md

VALIDATION: All reports render. GEXF opens in Gephi correctly.

============================================================ === SELF-REVIEW ===

• Complete: Crawl + graph + audit + semantic linking + cluster + patches?
• Robust: Respects robots, handles JS-rendered sites, scales to 50k pages?
• Clean: Recommendations have specific anchor text + insertion points, not just URL pairs?
• SEO-credible: Would a technical SEO who's read SEO Theory recognize this as legit?

Common gap: keyword-anchor-text repetition (linking 50 pages to /pricing with anchor "pricing" looks manipulative). Vary anchor text from the surrounding context.

============================================================ === LEARNINGS CAPTURE ===

~/.claude/skills/internal-link-graph/LEARNINGS.md.

============================================================ === STRICT RULES ===

• Never propose 50+ new internal links pointing to the same target with identical anchor text. Anchor diversification is a real ranking signal.
• Never recommend links from unrelated pages just to lift in-degree. Topical fit > raw link count.
• Never crawl past robots.txt. Respect.
• Always preserve the crawl history. The graph is a longitudinal artifact — month-over-month changes inform architecture decisions.
• Always include both PageRank-weighted AND semantic recommendations. PR-only = old-school; semantic-only = misses authority flow.