Scrawly SEO Crawler

The free, open-source SEO crawler built for Google and AI search. What it checks, how it works, how to install it, and how agencies can use it for technical SEO and GEO audits.

Agency clients now ask me a question classic crawlers cannot answer: "We rank on Google. Why does ChatGPT never mention us?"

Often the cause is technical, and invisible. robots.txt allows every AI crawler, but a CDN or firewall turns those crawlers away. robots.txt says yes. The server says no. The crawlers I paid for did not check both.

That gap is why I built Scrawly: a free, open-source desktop SEO crawler and site audit tool for macOS, Windows and Linux. It audits a website for the technical issues that affect Google rankings and visibility in AI search tools like ChatGPT, Perplexity and Google AI Overviews.

Version 1.0.0 went public on October 6, 2026, under the MIT License. There is no account, no licence key and no page cap. This article covers everything: what Scrawly checks, how it scores, how to install it, how to run your first audit, and how agencies can use it.

What is Scrawly? Scrawly is a local-first, open-source technical SEO crawler and audit platform engineered without artificial URL limits or licensing paywalls. Built from the ground up as a robust free screaming frog alternative software, Scrawly provides search professionals and technical architects with enterprise-grade crawl diagnostics, JavaScript rendering evaluation, and site architecture mapping right on their local machine. Beyond replacing traditional desktop crawlers, it functions as a modern, free Sitebulb alternative, pairing comprehensive technical issue categorization with AI-readiness auditing and native Model Context Protocol (MCP) integration. For teams seeking a transparent, community-driven screaming frog alternative, Scrawly delivers full crawl data sovereignty without the enterprise subscription tag.

How to install Scrawly You need Python 3.11 or newer (3.11, 3.12 and 3.13 are supported), git, and about 150 MB for a one-time download of the crawler browser. Node.js is not needed. If git or Python is missing, the installer adds them where it safely can, using Homebrew on macOS or winget on Windows.