Visualize an Internal Link Graph With Python and NetworkX

Visualize internal links in Python: turn a crawl export into a NetworkX graph, find hubs, orphans and dead ends, compute internal PageRank and plot the result.

· BugViso Team

16 min read

To visualize an internal link graph, export your crawl as source-to-target link pairs, load them into a NetworkX directed graph, compute in-degree, PageRank and betweenness, and draw the result with PyVis or Matplotlib.

Orphan pages and clusters then stand out at once. This guide gives the full Python script.

A website is a directed graph ($G = (V, E)$). Each unique URL is a vertex (node), and each internal hyperlink is a directed edge from source to destination.

Model your site this way and graph theory algorithms can find anomalies that tabular audits miss: isolated orphan clusters, high-betweenness bottlenecks, and unbalanced authority distribution.

This guide provides an end-to-end engineering blueprint for building, analyzing, and visualizing your website's internal link graph using Python, networkx, and pyvis.


The Graph Theory Toolkit for Technical Architecture

Before writing code, we must understand the core network metrics that describe the health of an internal link topology:

Network Graph Metrics Map

MetricArchitectural significance
In-Degree CentralityTotal incoming internal links
Out-Degree CentralityTotal outbound links on a page
Betweenness CentralityCritical informational bridge
PageRank CentralityStationary authority weight
Strongly Connected (SCC)Mutual reachability groups
Weakly Connected (WCC)Structural islands and orphans

In-Degree vs. Out-Degree

  • In-Degree ($k_{in}$): The count of directed edges pointing to a node. High in-degree indicates that a page is widely referenced across your site (e.g., your homepage, terms page, or top-level category hub).
  • Out-Degree ($k_{out}$): The count of outgoing links originating from a node. Extremely high out-degree (e.g., uncurated footer menus with 250+ links) dilutes the equity transferred along each edge.

Eigenvector Centrality vs. PageRank Centrality

While both metrics quantify node influence by considering the importance of neighboring nodes, their mathematical behavior differs significantly:

  • Eigenvector Centrality: Solves for the principal eigenvector of the adjacency matrix: $\lambda x = A x$. It assumes that connections to high-scoring nodes contribute more to the score of the node. However, on directed acyclic graphs (DAGs) or graphs with dead ends, eigenvector centrality fails to converge or collapses to zero for non-cyclic subgraphs.
  • PageRank Centrality: Resolves this limitation by incorporating the damping factor ($d = 0.85$) and random teleportation matrix $\frac{1 - d}{N} \mathbf{E}$. This guarantees convergence across all directed web graphs, regardless of whether cycles or sinkholes exist.

Betweenness Centrality: Detecting Bottlenecks

Betweenness centrality measures the fraction of all shortest paths between all pairs of nodes in the graph that pass through a specific node $v$:

$$C_B(v) = \sum_{s \neq v \neq t} \frac{\sigma_{st}(v)}{\sigma_{st}}$$

Where $\sigma_{st}$ is the total number of shortest paths from node $s$ to node $t$, and $\sigma_{st}(v)$ is the number of those paths that pass through $v$.

In technical SEO, a page with high betweenness centrality acts as a critical gateway. If this bridge page breaks (returns a 404 or 500 error) or is removed during a CMS migration, entire sections of your website become disconnected from the main crawl graph.

For procedures on detecting broken pathways before they affect crawlers, review our guide on broken link checker tools for large sites.

Louvain Community Detection: Evaluating Topical Silos

To assess whether your internal links respect topical clustering, you can apply the Louvain modularity algorithm. Modularity ($Q$) measures the density of links inside communities compared to links between communities:

$$Q = \frac{1}{2m} \sum_{i, j} \left[ A_{ij} - \frac{k_i k_j}{2m} \right] \delta(c_i, c_j)$$

Where:

  • $A_{ij}$ is the edge weight between nodes $i$ and $j$.
  • $k_i, k_j$ are the degrees of the nodes.
  • $m$ is the total number of edges.
  • $\delta(c_i, c_j)$ is 1 if nodes $i$ and $j$ belong to the same community, 0 otherwise.

When executed on your crawl graph, Louvain clustering should partition your site along clear semantic boundaries (e.g., all database articles cluster together; all frontend articles cluster together). If the algorithm groups unrelated topics into a single community, your site suffers from excessive, unfocused cross-linking.

Connected Components: Finding Structural Islands

  • Strongly Connected Component (SCC): A maximal subgraph where every node is reachable from every other node along directed paths. A healthy topic cluster should form an SCC.
  • Weakly Connected Component (WCC): A subgraph where nodes are connected if edge directions are ignored. If your graph contains multiple disjoint WCCs, you have completely isolated "islands" that search crawlers cannot discover through standard link traversal.

Data Pipeline: Extracting Directed Edges from Crawl Data

To build our graph, we need a dataset containing directed link pairs: a source URL and a target URL.

The following Python script uses BeautifulSoup and requests to crawl an internal domain, extract internal hyperlinks, strip fragments and tracking query parameters, and export a clean edge list to CSV:

python
#!/usr/bin/env python3
"""
crawl_link_edges.py - High-throughput crawler to extract internal link edges.
Saves clean directed edges (source, target) for graph analysis.
"""

import csv
import sys
from urllib.parse import urljoin, urlparse, urldefrag
from collections import deque
import requests
from bs4 import BeautifulSoup

def is_internal_url(url: str, base_domain: str) -> bool:
    parsed = urlparse(url)
    return parsed.netloc == base_domain and parsed.scheme in ["http", "https"]

def clean_url(url: str) -> str:
    # Strip URL fragments (#section) and normalize trailing slashes
    clean, _ = urldefrag(url)
    parsed = urlparse(clean)
    path = parsed.path.rstrip("/") if parsed.path != "/" else "/"
    return f"{parsed.scheme}://{parsed.netloc}{path}"

def crawl_edges(start_url: str, max_pages: int = 500, output_csv: str = "crawl_edges.csv"):
    parsed_start = urlparse(start_url)
    base_domain = parsed_start.netloc
    start_url_clean = clean_url(start_url)
    
    visited = set()
    queue = deque([start_url_clean])
    edges = set()
    
    session = requests.Session()
    session.headers.update({"User-Agent": "GraphCrawler/1.0"})
    
    print(f"[*] Starting crawl on: {start_url_clean}")
    print(f"[*] Target max pages: {max_pages}")
    
    while queue and len(visited) < max_pages:
        current_url = queue.popleft()
        
        if current_url in visited:
            continue
            
        visited.add(current_url)
        print(f"[{len(visited)}/{max_pages}] Crawling: {current_url}")
        
        try:
            resp = session.get(current_url, timeout=5)
            if "text/html" not in resp.headers.get("Content-Type", ""):
                continue
                
            soup = BeautifulSoup(resp.text, "html.parser")
            for a_tag in soup.find_all("a", href=True):
                raw_href = a_tag["href"].strip()
                abs_url = urljoin(current_url, raw_href)
                target_url = clean_url(abs_url)
                
                if is_internal_url(target_url, base_domain):
                    # Record directed edge: source -> target
                    edges.add((current_url, target_url))
                    
                    if target_url not in visited and target_url not in queue:
                        queue.append(target_url)
                        
        except Exception as err:
            print(f"[!] Error fetching {current_url}: {err}")
            
    # Export edges to CSV
    print(f"\n[*] Crawl complete. Exporting {len(edges)} unique edges to {output_csv}...")
    with open(output_csv, "w", newline="", encoding="utf-8") as f:
        writer = csv.writer(f)
        writer.writerow(["source", "target"])
        for src, tgt in sorted(edges):
            writer.writerow([src, tgt])
            
    print(f"[+] Successfully generated edge list: {output_csv}")

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python3 crawl_link_edges.py <https://example.com> [max_pages]")
        sys.exit(1)
        
    pages = int(sys.argv[2]) if len(sys.argv) > 2 else 250
    crawl_edges(sys.argv[1], max_pages=pages)

Run the edge extraction crawler:

bash
python3 scripts/crawl_link_edges.py https://example.com 300

Building the Directed Graph and Computing Metrics with NetworkX

The four metrics this script computes, and what to do with each:

MetricWhat it showsWhat to do
In-degreeHow many pages link to a pageLow in-degree on an important page: add contextual links
PageRankRelative internal authorityCheck that money pages score well, not just the homepage
BetweennessPages that bridge sectionsProtect bridge pages; losing one can cut off a section
Zero in-degreeOrphan pagesLink to them or remove them

Once the edge list is exported, we load the dataset into networkx to instantiate a directed graph (nx.DiGraph()) and compute our topological metrics:

python
#!/usr/bin/env python3
"""
analyze_link_graph.py - Analyzes internal link network topology.
Computes In-Degree, Out-Degree, Betweenness, and PageRank Centrality using NetworkX.
"""

import sys
import pandas as pd
import networkx as nx

def analyze_graph(edge_csv: str):
    print(f"[*] Ingesting edge list from: {edge_csv}")
    df = pd.read_csv(edge_csv)
    
    # Instantiate directed graph
    G = nx.DiGraph()
    for _, row in df.iterrows():
        G.add_edge(row['source'], row['target'])
        
    num_nodes = G.number_of_nodes()
    num_edges = G.number_of_edges()
    
    print(f"\n" + "="*60)
    print(f"NETWORK GRAPH TOPOLOGY OVERVIEW:")
    print(f"="*60)
    print(f"Total Unique Nodes (Pages): {num_nodes}")
    print(f"Total Directed Edges (Links): {num_edges}")
    print(f"Graph Density: {nx.density(G):.5f}")
    
    # 1. Compute PageRank
    print("[*] Computing PageRank centrality (damping=0.85)...")
    pagerank = nx.pagerank(G, alpha=0.85)
    
    # 2. Compute Betweenness Centrality
    print("[*] Computing Betweenness centrality...")
    betweenness = nx.betweenness_centrality(G)
    
    # 3. Degrees
    in_degrees = dict(G.in_degree())
    out_degrees = dict(G.out_degree())
    
    # Compile Node Metrics DataFrame
    metrics = []
    for node in G.nodes():
        metrics.append({
            "url": node,
            "in_degree": in_degrees.get(node, 0),
            "out_degree": out_degrees.get(node, 0),
            "betweenness": betweenness.get(node, 0.0),
            "pagerank": pagerank.get(node, 0.0)
        })
        
    metrics_df = pd.DataFrame(metrics)
    
    # 4. Connected Components Analysis
    print("\n" + "="*60)
    print("CONNECTED COMPONENTS & ORPHAN CLUSTER AUDIT:")
    print("="*60)
    wccs = list(nx.weakly_connected_components(G))
    print(f"Total Weakly Connected Components: {len(wccs)}")
    
    if len(wccs) > 1:
        print("[!] STRUCTURAL ALERT: Disconnected islands detected!")
        for idx, comp in enumerate(wccs[1:], start=2):
            print(f"  Component #{idx} ({len(comp)} nodes):")
            for url in list(comp)[:5]:
                print(f"    - {url}")
            if len(comp) > 5:
                print(f"    ... and {len(comp) - 5} more.")
    else:
        print("[+] SUCCESS: The entire crawled network is weakly connected.")
        
    # Top Authority Nodes
    print("\n" + "="*60)
    print("TOP 10 NODES BY INTERNAL PAGERANK:")
    print("="*60)
    top_pr = metrics_df.sort_values(by="pagerank", ascending=False).head(10)
    for idx, row in top_pr.reset_index().iterrows():
        print(f"{idx+1:02d}. [PR: {row['pagerank']:.5f} | In: {row['in_degree']:3d}] {row['url']}")
        
    # Top Bottleneck Bridges
    print("\n" + "="*60)
    print("TOP 5 STRUCTURAL BRIDGES (BETWEENNESS CENTRALITY):")
    print("="*60)
    top_between = metrics_df.sort_values(by="betweenness", ascending=False).head(5)
    for idx, row in top_between.reset_index().iterrows():
        print(f"{idx+1:02d}. [Betweenness: {row['betweenness']:.5f}] {row['url']}")
        
    metrics_df.to_csv("network_metrics_report.csv", index=False)
    print(f"\n[+] Full metrics export saved to network_metrics_report.csv")
    return G, metrics_df

if __name__ == "__main__":
    csv_input = sys.argv[1] if len(sys.argv) > 1 else "crawl_edges.csv"
    analyze_graph(csv_input)

Run the graph analysis:

bash
python3 scripts/analyze_link_graph.py crawl_edges.csv

While numerical metrics reveal structural statistics, visual representation is necessary to communicate architectural realities to non-technical stakeholders and cross-functional engineering teams.

Using pyvis, we can export an interactive, physics-driven HTML visualization that renders directly in any modern browser. In the script below, we color nodes based on their topic path and scale their physical radius proportionally to their calculated PageRank:

python
#!/usr/bin/env python3
"""
visualize_pyvis.py - Generates an interactive force-directed graph with PyVis.
Nodes are sized by PageRank and colored by path category.
"""

import sys
from urllib.parse import urlparse
import pandas as pd
import networkx as nx
from pyvis.network import Network

def assign_color_by_path(url: str) -> str:
    path = urlparse(url).path
    if path == "" or path == "/":
        return "#FF4444"  # Homepage: Vibrant Red
    elif path.startswith("/blog"):
        return "#3B82F6"  # Blog: Blue
    elif path.startswith("/docs") or path.startswith("/guides"):
        return "#10B981"  # Documentation: Emerald Green
    elif path.startswith("/product") or path.startswith("/features"):
        return "#8B5CF6"  # Commercial: Purple
    else:
        return "#9CA3AF"  # General Utility: Gray

def generate_interactive_visualization(edge_csv: str, output_html: str = "link_graph.html"):
    df = pd.read_csv(edge_csv)
    
    G = nx.DiGraph()
    for _, row in df.iterrows():
        G.add_edge(row['source'], row['target'])
        
    print(f"[*] Calculating PageRank for visual node scaling...")
    pr = nx.pagerank(G, alpha=0.85)
    max_pr = max(pr.values()) if pr else 1.0
    
    # Initialize PyVis Network
    net = Network(
        height="900px",
        width="100%",
        directed=True,
        bgcolor="#111827",
        font_color="#F9FAFB"
    )
    
    # Configure physics simulation engine for clean cluster repulsion
    net.set_options("""
    {
      "nodes": {
        "borderWidth": 1,
        "borderWidthSelected": 3
      },
      "edges": {
        "color": { "inherit": true },
        "smooth": { "type": "continuous" },
        "arrows": { "to": { "enabled": true, "scaleFactor": 0.5 } }
      },
      "physics": {
        "forceAtlas2Based": {
          "gravitationalConstant": -50,
          "centralGravity": 0.01,
          "springLength": 100,
          "springConstant": 0.08
        },
        "maxVelocity": 50,
        "solver": "forceAtlas2Based",
        "timestep": 0.35,
        "stabilization": { "iterations": 150 }
      }
    }
    """)
    
    print(f"[*] Adding {G.number_of_nodes()} nodes and {G.number_of_edges()} edges to PyVis...")
    
    for node in G.nodes():
        node_pr = pr.get(node, 0.0)
        # Normalize radius between 10px and 45px
        node_size = 10 + (node_pr / max_pr) * 35
        node_color = assign_color_by_path(node)
        
        # Tooltip content
        in_deg = G.in_degree(node)
        out_deg = G.out_degree(node)
        title_tooltip = f"<b>{node}</b><br>PageRank: {node_pr:.5f}<br>In-Links: {in_deg}<br>Out-Links: {out_deg}"
        
        parsed = urlparse(node)
        label = parsed.path if parsed.path else "/"
        if len(label) > 25:
            label = label[:22] + "..."
            
        net.add_node(
            node,
            label=label,
            title=title_tooltip,
            value=node_size,
            color=node_color
        )
        
    for src, tgt in G.edges():
        net.add_edge(src, tgt)
        
    print(f"[*] Generating standalone interactive HTML: {output_html}")
    net.write_html(output_html)
    print(f"[+] Interactive visualization ready! Open '{output_html}' in any web browser.")

if __name__ == "__main__":
    input_csv = sys.argv[1] if len(sys.argv) > 1 else "crawl_edges.csv"
    generate_interactive_visualization(input_csv)

Generate the visualization:

bash
python3 scripts/visualize_pyvis.py crawl_edges.csv

When you open link_graph.html in your browser, the physics simulation activates. Dense thematic clusters naturally coalesce into visual constellations, while orphaned nodes drift to the periphery.


Diagnosing Architectural Anti-Patterns Visually

When inspecting your force-directed link graph, look for these four common structural patterns:

Visual Network Topology Archetypes

1. The Star Topology (Centralized Bottleneck)

  • All nodes connect exclusively to a single hub.
  • Risk: Leaf nodes lack lateral peer cross-links.

2. The Barbell Topology (Divided Silos)

  • Two dense clusters connected by a single fragile bridge.
  • Risk: Breaking the bridge cuts off 50% of the site.

3. The Satellite Island (Disconnected Subgraph)

  • A cluster of nodes floating free from the main body.
  • Risk: Search crawlers cannot access the island.

4. The Megamenu Mesh (Over-Densified Graph)

  • Every node links to every other node indiscriminately.
  • Risk: Total dilution of PageRank across low-value pages.

1. The Barbell Topology

If your blog cluster and your documentation cluster are only connected by a single link in the main navigation, your graph forms a barbell structure.

If an editorial change inadvertently alters that navigation link, the entire documentation cluster's incoming link equity collapses.

Architectural Fix: Inject contextual in-content link bridges between relevant blog articles and technical documentation guides, thickening the connection between clusters.

2. Satellite Islands (Weakly Connected Components)

If your visualization displays a detached cluster floating outside the gravitational pull of the main network, you have discovered an orphaned sub-domain or abandoned landing page campaign.

These pages are accessible only if a crawler already knows the URL (e.g., via XML sitemaps), but receive zero internal authority from the domain's primary backlink profile.

3. The Megamenu Mesh

If your graph appears as an impenetrable, uniform sphere where every node connects to every other node, your site suffers from boilerplate over-linking.

When every page contains 300 identical links in the header and footer, the network graph lacks topological differentiation. Search engines cannot discern which pages represent core topical hubs versus secondary utility endpoints.

For a structured review of sitewide link health, consult our how to audit a website for SEO.


A standard Python requests script only inspects static HTML returned by the initial HTTP response. If your website is a single-page application (SPA) that renders links dynamically via client-side JavaScript, a static scraper will miss critical edges, producing a fragmented, inaccurate graph.

To capture all edges, do two things:

  1. Pre-render HTML Server-Side: Guarantee that all <a href="..."> elements exist in the raw server-rendered response before client hydration.
  2. Execute Headless Crawls: Use headless browser frameworks (such as Playwright or Puppeteer) to allow JavaScript execution before extracting the DOM.

To understand how Googlebot parses dynamic links, consult our guide on JavaScript links crawl discovery and Googlebot onclick issues.

The graph is most useful for finding what a crawl alone hides: start with how to find orphan pages, then act on the result with the step-by-step on-page SEO checklist. To generate links automatically from the same data, see automated internal linking.


Frequently Asked Questions

Export your crawl as a list of source-to-target link pairs, load the pairs into a NetworkX directed graph in Python, and draw it with PyVis or Matplotlib. PyVis produces an interactive HTML file you can open in any browser. Sizing nodes by PageRank and colouring them by topic makes clusters, hubs and orphan pages easy to spot.

What does betweenness centrality mean for SEO?

Betweenness centrality measures how many shortest paths between other pages pass through a given page. A page with high betweenness is a gateway: if it breaks, returns a 404 or is removed in a migration, whole sections of the site can lose their link path from the rest of the graph. Find these pages and add alternative links around them.

How do you find orphan pages with NetworkX?

Look for disconnected parts of the graph. Pages with an in-degree of zero have no internal links pointing to them, and separate weakly connected components show clusters that are cut off from the main site entirely. Crawlers can reach those pages only if they already know the URLs, for example from the XML sitemap.

Your graph is probably missing pages because the crawler only read static HTML. A requests and BeautifulSoup crawler sees the initial server response, so links that a single-page application adds with JavaScript never appear as edges. Make sure links exist in the server-rendered HTML, or crawl with a headless browser such as Playwright before extracting links.

A healthy internal link graph shows clear topic clusters that are densely linked inside and connected to each other by several links. Warning signs are a star where every page hangs off one hub, a barbell where two clusters share a single bridge, detached islands floating away from the main graph, and a uniform mesh where every page links to every other page.


Building custom Python crawl scripts and NetworkX pipelines provides deep insights, but requires significant maintenance as site scale increases. Maintaining scripts, managing proxy rotations, and rendering complex single-page applications at scale can quickly strain engineering resources.

The BugViso auditing platform automates the same analysis for the pages it crawls (up to 300 per scan, depending on plan). It renders pages in a real browser and:

  1. Renders and Maps Every Dynamic Edge: Executes complete client-side JavaScript to capture dynamic DOM links, pagination triggers, and interactive navigational elements.
  2. Scores Link Equity: Counts each page's inbound and outbound internal links and computes an internal PageRank, shown as an equity score and tier in the link graph. It does not compute betweenness; use the NetworkX script above for that.
  3. Pinpoints Structural Islands and Orphan Traps: Flags disconnected subgraphs, weakly connected components, and buried pages residing at Depth 4+.
  4. Detects Equity Leaks: Correlates link graph edges with HTTP status codes to catch redirect chains and 404 dead ends that bleed link authority. To trace redirect hops, review our guide on redirect chain audits and link equity drains.

More detail is on the technical SEO audit feature page.


Transforming your site's internal links into an interactive directed graph replaces intuitive guesswork with empirical mathematical architecture, revealing precisely how authority flows through your digital ecosystem.

Generate your website's complete internal link graph and diagnose structural bottlenecks by launching a full crawl with BugViso.

Written by

· BugViso Team

Kazi writes BugViso’s guides on website audits, Core Web Vitals, technical SEO, accessibility and AI search. The data in them comes from audits BugViso runs in a real Chromium browser, including our 219-site study of website health.

Found this useful? Share it.

Summarise with AI

Open this guide in your assistant.

Follow BugViso

New guides and audit data as we publish them.

AI visibility tracker

Is your brand in AI answers?

See how often AI assistants mention you, cite your site and rank you against your competitors.

  • ChatGPT
  • Gemini
  • Perplexity
  • Claude
  • AI Overviews
See how it works

See where your site stands

Run a free BugViso audit for SEO, speed, accessibility and AI search readiness — with fixes you can ship today.