ArchiveBox

Because modern websites are complicated and often rely on dynamic content, ArchiveBox archives the sites in several different formats beyond what public archiving services like Archive.org and Archive.is are capable of saving.

ArchiveBox imports a list of URLs from stdin, remote url, or file, then adds the pages to a local archive folder using wget to create a browsable html clone, youtube-dl to extract media, and a full instance of Chrome headless for PDF, Screenshot, and DOM dumps, and more...

Using multiple methods and the market-dominant browser to execute JS ensures we can save even the most complex, finicky websites in at least a few high-quality, long-term data formats.

Can import links from:

• Pocket, Pinboard, Instapaper
• RSS, XML, JSON, or plain text lists
• Browser history or bookmarks (Chrome, Firefox, Safari, IE, Opera, and more)
• Shaarli, Delicious, Reddit Saved Posts, Wallabag, Unmark.it, and any other text with links in it!

Can save these things for each site:

• favicon.ico favicon of the site
• example.com/page-name.html wget clone of the site, with .html appended if not present
• output.pdf Printed PDF of site using headless chrome
• screenshot.png 1440x900 screenshot of site using headless chrome
• output.html DOM Dump of the HTML after rendering using headless chrome
• archive.org.txt A link to the saved site on archive.org
• warc/ for the html + gzipped warc file <timestamp>.gz
• media/ any mp4, mp3, subtitles, and metadata found using youtube-dl
• git/ clone of any repository for github, bitbucket, or gitlab links
• index.html & index.json HTML and JSON index files containing metadata and details

The archiving is additive, so you can schedule ./archive to run regularly and pull new links into the index. All the saved content is static and indexed with JSON files, so it lives forever & is easily parseable, it requires no always-running backend.