crawlie

The fast, free, open-source technical SEO + GEO crawler — built for humans and agents.

Crawl any site for broken links, redirects, missing metadata, and 40+ SEO & Generative-Engine checks — with plain-English guidance on every fix. Runs locally, ships a CLI and an MCP server, and costs nothing.

Use cases:

• Pre-launch QA — catch broken links, redirects, 4xx/5xx, and missing metadata before you ship.
• GEO optimization — make pages citable by AI search: structured data, semantic HTML, answer-ready content, authorship/E-E-A-T.
• Agent workflows — let a marketing/SEO agent audit a site and propose fixes autonomously via MCP.
• CI/CD gating — crawlie crawl … --fail-on error in a pipeline to block regressions.
• Client reporting — generate a polished, shareable HTML report in one command.
• Auditing AI-generated sites — verify that the site your agent just built is actually built for search.

What it checks:

46 rules and counting.

• Technical SEO — broken links · 4xx/5xx · redirects & chains · titles & meta descriptions (missing / duplicate / length) · H1s · canonicals · noindex / nofollow / X-Robots-Tag · robots.txt blocking · images missing alt · thin & duplicate content · orphan & deep pages
• Performance & security — slow responses · large pages · missing compression · HTTPS · mixed content · HSTS
• Mobile, international & social — viewport · lang · hreflang · Open Graph · Twitter cards · structured data
• GEO — Generative Engine Optimization — structured data, semantic HTML, answer-readiness, authorship/E-E-A-T, dated content, question-style headings, and extractable blocks, rolled into a per-page GEO score.

Every finding links to plain-English guidance: why it matters, how to fix it, and what happens if you ignore it.