Apps with 'Crawler' feature

A free in-depth website analyzer providing audits of security, performance, SEO, accessibility and other technical aspects. Available as a desktop application for Windows/macOS/Linux and as a CLI tool for advanced users and CI/CD processes. It also includes an offline web page exporter.

import.io is a web-based platform that puts the power of the machine readable web in your hands. Using our tools you can create an API or crawl an entire website in a fraction of the time of traditional methods, no coding required.

Netpeak Spider is an SEO crawler for a day-to-day SEO audit, fast issue check, comprehensive analysis, and website scraping.

Prying Deep crawls dark/clear net platforms and extracts as much intelligence as possible effectively.

Scrapling is an adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl.

A flexible web crawler and scraping tool using Playwright, supporting both BFS and DFS crawling strategies with screenshot capture and structured output. Installable via npm and usable both as a CLI and programmatically.

A self-hosted BitTorrent indexer, DHT crawler, content classifier and torrent search engine with web UI, GraphQL API and Servarr stack integration.

Crawlbase, formerly ProxyCrawl, helps you stay anonymous while crawling the web, web crawling protection the way it should be.

A network agnostic DHT crawler and monitor. The crawler connects to DHT bootstrappers and then recursively follows all entries in their k-buckets until all peers have been visited.

OpenRPA automates repetitive tasks performed by humans to cut costs, eliminate keying errors, speed up processes, and easily link applications together. These tasks can be converted into a business process and then automated.

Indexes Telegram channel connections with a visual relations map, multilingual support, channel statistics, and community-focused, ad-free analytics tools.

API for website crawling, scraping, and structured data extraction. Converts web content to markdown or data for AI apps, automates multi-page analysis, and supports custom output formats.

Pulno is a website audit tool which checks for SEO related issues and sends comprehensible tips to improve on-page SEO with page speed optimizer (CSS and image file optimization), meta tags and unique text analyzer.

👾 Fast and simple video download library and CLI tool written in Go.

File system crawler, storage search engine and storage analytics software powered by Elasticsearch to help visualize and manage your disk space usage.

Octoparse is a no-code web scraping tool. It provides both free pre-made scraper templates and custom scraping features with which people without coding knowledge can extract various web data with simple point-and-click.

A web-based crawler with real-time crawl feedback.

Advanced, fast & flexible SEO website crawler that can help identify technical or architectural issues with any site.

Common Crawl builds and maintains an open repository of web crawl data that can be accessed and analyzed by anyone

Open-Source Python Based SEO Web Crawler Free SEO Analysis Tool Made by SEOs for Digital Marketing Experts

Analyse and improve your onsite SEO Audit and identify any server issues Extract any on-page information you need.

Extracty can extract any web data and create an API to the webpage's information.

A1 Website Analyzer can crawl websites and collect data to be used for finding site and SEO problems. Filter results to see which URLs are blocked by robots.txt, which are noindex, nofollow, canonical to self, canonical to other, response codes etc.

Web scraping including cloud.

Generate visual sitemaps + screenshots for live & staging websites. Run Audits & Plan Content & Create Flows for SEO, UX, and Competitor Analysis.

Web scraping API with a flexible usage-based pricing that offers real browsers, rotating mobile proxies, and JavaScript rendering.

A web scraping service to collect data from websites, without any programming or DIY tools.