Doc Link Rot Detector

What this skill does

Walks every markdown file in a repo, extracts external HTTP(S) links, and probes each one with a HEAD request. Reports broken (4xx/5xx), redirected, and timeout-prone links so docs maintainers can fix them in a batch.

Inputs

repo_dir: repo root containing markdown docs.
Optional concurrency: max in-flight requests (default 8).
Optional timeout_ms: per-request timeout (default 5000).
Optional allowlist: regex of hosts to skip (e.g., intranet URLs).

Steps

Find candidate files: rg -l --glob '*.md' '' <repo_dir>.
Extract URLs from each file: rg -oN 'https?://[^\s)\]]+' <file>. Strip trailing punctuation.
Deduplicate URLs across the repo, retaining one file:line reference per unique URL for citation.
For each URL, request HEAD with curl -sIL -o /dev/null -w '%{http_code} %{url_effective} %{redirect_count}\n' --max-time <timeout_ms_seconds>. If HEAD returns 405 or 501, retry with GET range 0-0.
Run requests with a worker pool (concurrency concurrency).
Classify each result: 200 ok, 301/302 redirect (note new URL), 404/410 broken, 5xx server-error, timeout/DNS error.
Group results by classification and sort by file:line.
For redirects, suggest the canonical URL (effective URL after redirect chain) as the replacement.
Emit link-rot-report.md with sections per classification.

Output

link-rot-report.md containing a summary header (counts per status), one section per classification with file:line — URL — note rows, and a final "fixed redirect suggestions" patch block (a unified diff over the markdown files that maintainers can apply).

Verification

Re-run a sample of 10 URLs from the broken section after the report is written; the second probe must agree with the first to rule out transient failures. Sample three redirect entries by visiting them in a browser and confirming the suggested canonical URL matches. If more than 30% of URLs returned timeout, increase concurrency limit suggestion in the report header — the host may be aggressively rate-limiting.

Edge cases

Sites that block HEAD (some CDN-fronted): retry with GET; if still failing, mark as "method-not-allowed" rather than broken.
Anchor-only URLs (#section): skip; can't be checked over network.
Localhost or private-CIDR URLs: skip with a "non-public" classification.
Markdown reference-style links ([label][ref]): resolve via the bottom-of-file definitions before extracting URLs.

amitte/doc-link-rot-detector