@harshvz/crawler

crawler is a Playwright-based web crawler designed to turn websites into reusable knowledge artifacts.

Unlike traditional scrapers that focus on extracting isolated data fields, this tool focuses on capturing meaning-bearing content from real, JavaScript-rendered pages and preserving it in a form suitable for documentation, internal knowledge bases, and AI/LLM workflows.

The crawler navigates websites using BFS or DFS strategies, renders each page in a real browser, and extracts core semantic elements such as metadata, headings (H1–H6), paragraphs, and inline text. The extracted content is stored as Markdown files, alongside full-page screenshots, providing both textual knowledge and visual ground truth for every crawled page.

The project is intentionally opinionated and minimal:

It prioritizes content understanding over raw scraping speed

It captures human-readable, context-preserving text

It produces outputs that are immediately usable by humans and machines

At its core, crawler is built as a knowledge ingestion layer — a foundation for turning websites into structured documentation, searchable knowledge bases, or LLM-ready corpora, while remaining fully local, open-source, and developer-controlled.

As the project evolves, the focus is on making extraction more controllable and deterministic, allowing users to define what content is captured and how it is organized — without introducing black-box behavior or external dependencies.