Apps with 'Web Crawler' feature

Free software that downloads entire sites to local storage for offline use, maintaining the original link structure for seamless browsing.

This is an independent DIY search engine that focuses on non-commercial content, and attempts to show you sites you perhaps weren't aware of in favor of the sort of sites you probably already knew existed.

Portia is an open source visual scraping tool, allows you to scrape websites without any programming knowledge required! Simply annotate pages you're interested in, and Portia will create a spider to extract data from similar pages.

Apify is a web scraping and automation platform - it extracts data from websites, crawls lists of URLs and automates workflows on the web. Turn any website into an API!.

Minexa.ai is a next-generation tool that makes web scraping faster and more affordable with an AI-powered solution no other alternative has. Unlike others that require constant tweaking, struggle under heavy loads, or charge extra for natural language processing, Minexa adapts...

Netpeak Spider is an SEO crawler for a day-to-day SEO audit, fast issue check, comprehensive analysis, and website scraping.

grab-site is a crawler for archiving websites to WARC files. It includes a dashboard for monitoring multiple crawls, and supports changing URL ignore patterns during the crawl.

Seobility allows you to quickly audit and review a website from an onsite SEO perspective. You will get valuable insights to improve your search engine optimization. It is free for small websites with up to 1.000 pages.

Oxylabs is the leading global provider of premium proxies and data scraping solutions for large-scale web intelligence collection.

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

Pulno is a website audit tool which checks for SEO related issues and sends comprehensible tips to improve on-page SEO with page speed optimizer (CSS and image file optimization), meta tags and unique text analyzer.

A browser extension that uses AI to detect listings type data which can be easily scraped into CSV or Excel file, no coding required. Can automatically click next button to continue to the next page. The extension runs completely in user’s browser.

This project is a java web spider (web crawler) with the ability to download (and resume) files. It is also highly customizable with regular expressions and download templates.

Extract information from web sites with a visual point-and-click toolkit. Turn websites into useful data. Automate data workflows on the web, process, and transform data at any scale.

Listly, a web extension, simplifies web scraping without coding. This helps you collect and export enormous volumes of data into either Excel or Google Sheets.