Apps with 'Crawler' feature

A free in-depth website analyzer providing audits of security, performance, SEO, accessibility and other technical aspects. Available as a desktop application for Windows/macOS/Linux and as a CLI tool for advanced users and CI/CD processes. It also includes an offline web page exporter.

import.io is a web-based platform that puts the power of the machine readable web in your hands. Using our tools you can create an API or crawl an entire website in a fraction of the time of traditional methods, no coding required.

Netpeak Spider is an SEO crawler for a day-to-day SEO audit, fast issue check, comprehensive analysis, and website scraping.

Prying Deep crawls dark/clear net platforms and extracts as much intelligence as possible effectively.

Crawlbase, formerly ProxyCrawl, helps you stay anonymous while crawling the web, web crawling protection the way it should be.

Pulno is a website audit tool which checks for SEO related issues and sends comprehensible tips to improve on-page SEO with page speed optimizer (CSS and image file optimization), meta tags and unique text analyzer.

OpenRPA automates repetitive tasks performed by humans to cut costs, eliminate keying errors, speed up processes, and easily link applications together. These tasks can be converted into a business process and then automated.

A network agnostic DHT crawler and monitor. The crawler connects to DHT bootstrappers and then recursively follows all entries in their k-buckets until all peers have been visited.

Open-Source Python Based SEO Web Crawler Free SEO Analysis Tool Made by SEOs for Digital Marketing Experts

Analyse and improve your onsite SEO Audit and identify any server issues Extract any on-page information you need.

Octoparse is a no-code web scraping tool. It provides both free pre-made scraper templates and custom scraping features with which people without coding knowledge can extract various web data with simple point-and-click.

A web-based crawler with real-time crawl feedback.

Advanced, fast & flexible SEO website crawler that can help identify technical or architectural issues with any site.

Common Crawl builds and maintains an open repository of web crawl data that can be accessed and analyzed by anyone

Extracty can extract any web data and create an API to the webpage's information.

A1 Website Analyzer can crawl websites and collect data to be used for finding site and SEO problems. Filter results to see which URLs are blocked by robots.txt, which are noindex, nofollow, canonical to self, canonical to other, response codes etc.

Web scraping including cloud.