Scrapy is a free and open-source web-crawling framework written in Python. Originally designed for web scraping, it can also be used to extract data using APIs or as a general-purpose web crawler. It was developed and is maintained by Zyte formerly Scrapinghub, a web-scraping...
Cloud platform enabling automated data extraction, web crawling, and browser workflow automation using reusable acts, headless browser support, advanced scheduling, persistent storage, API integrations, modular design and scalable resource management.
Czechia
EU
Crawlbase, formerly ProxyCrawl, helps you stay anonymous while crawling the web, web crawling protection the way it should be.
grab-site is a crawler for archiving websites to WARC files. It includes a dashboard for monitoring multiple crawls, and supports changing URL ignore patterns during the crawl.
Lookyloo is a web interface that allows users to capture a website page and then display a tree of domains that call each other.
Switzerland
Flyscrape is a standalone and scriptable web scraper, combining the speed of Go with the flexibility of JavaScript. — Focus on data extraction rather than request juggling.
GermanyEU
Scraperr is a self-hosted web application that allows users to scrape data from web pages by specifying elements via XPath. Users can submit URLs and the corresponding elements to be scraped, and the results will be displayed in a table.
United States
The most popular web scraping extension. Start scraping in minutes. Automate your tasks with our Cloud Scraper. No software to download, no coding needed.
Common Crawl builds and maintains an open repository of web crawl data that can be accessed and analyzed by anyone
Apache Nutch is a highly extensible and scalable open source web crawler software project.
StormCrawler is an open source SDK for building distributed web crawlers with Apache Storm. The project is under Apache license v2 and consists of a collection of reusable resources and components, written mostly in Java.
Data Miner is a Google Chrome extension that helps you scrape data from web pages and into a CSV file or Excel spreadsheet.
United Kingdom
