PageSift

PageSift is a self-hosted web content engine written in Rust that turns URLs, PDFs and Office files into clean Markdown and structured data for AI agents and LLM pipelines. It offers a CLI, a Firecrawl-compatible API, Jina Reader-compatible endpoints, an MCP server and Python/Node.js SDKs, with built-in self-developed OCR for scanned PDFs, self-healing selector extraction and change monitoring. Version 1.2 adds opt-in ITNA access-barrier recovery ([itna] enabled = true): a blocked or challenge HTTP result may get one bounded browser retry on the same egress, while rate limits, login walls and explicit HTTP-only requests do not escalate. PDF → Markdown at 2.4 ms/page; wide crawls at 3,500–4,400 pages/s in ~300 MB. Open source (AGPL-3.0) with commercial licensing.