Apps with 'website-crawler' feature

  1. HTTrack is a free (GPL, libre/free software) and easy-to-use offline browser utility.
    show more
  2. Netpeak Spider is a desktop tool which crawls your website like a search engine robot and detects key SEO issues that influence the website’s visibility in SERP. Netpeak Spider is available on Windows only. This tool allows you to:
    show more


  3. A web-based crawler with real-time crawl feedback. Advanced, fast & flexible SEO website crawler that can help identify technical or architectural issues with any site.
    No screenshots yet
  4. Mixnode is a fast, flexible, massively scalable platform to extract and analyze data from the web.
    No screenshots yet
  5. Boost your SEO, speed and security: Checkbot for Chrome crawls your site to test you’re following over 50+ best practices. Test 1,000s of pages in minutes for broken links, duplicate content, invalid HTML, insecure forms, redirect chains and more.
  6. Extracty can extract any web data and create an API to the webpage's information.

    Discontinued

    The company do not host an online backend anymore, so it's not possible to execute any scripts online. The website has a functional front end.

  7. Common Crawl builds and maintains an open repository of web crawl data that can be accessed and analyzed by anyone
  8. Seobility allows you to quickly audit and review a website from an onsite SEO perspective. You will get valuable insights to improve your search engine optimization. It is free for small websites with up to 1.000 pages.