Generates verified robots.txt, sitemap.xml, llms.txt and llms-full.txt from a website crawl, checking crawler rules, links, hostnames and hidden text.
Platforms
- Online
Canada

Generates verified robots.txt, sitemap.xml, llms.txt and llms-full.txt from a website crawl, checking crawler rules, links, hostnames and hidden text.
