Web Archiving
186 items from iipc/awesome-web-archiving ★2,661
-
HTTPreserve linkstat github.com
Command line implementation of httpreserve.info to describe the status of a web page. Can be easily scripted and provides JSON output to enable querying through tools like JQ. HTTPreserve Linkstat describes current status, and earliest and latest links on archive.org. (Golang)…
-
Tweet Archvies Unleashed Toolkit github.com
An open-source toolkit for analyzing line-oriented JSON Twitter archives with Apache Spark. (In Development)
-
warcrefs github.com
Web archive deduplication tools. (Stable)
-
HadoopConcatGz github.com
A Splitable Hadoop InputFormat for Concatenated GZIP Files (and *.warc.gz). (Stable)
-
PastPage github.com
Browser extension for Chrome and Firefox that recovers broken or changed pages by querying the Wayback Machine and other web archives in parallel. (Stable)
-
wasapi-downloader github.com
Java command line application to download crawls from WASAPI. (Stable)
-
duckdb_warc github.com
DuckDB extension to query WARC files. (In Development)
-
Jwat-Tools github.com
Tools for reading/writing/validating WARC/ARC/GZIP files (Java). (Stable)
-
Jwat github.com
Libraries for reading/writing/validating WARC/ARC/GZIP files (Java). (Stable)
-
-
-
node-cdxj github.com
CDXJ file parser (Node.js). (Stable)
-
Dead-Web Index github.com
Reachability labels (alive / blocked / dead) for the top 10 million domains, two probe arms (polite HTTP and a browser TLS fingerprint), 2026. CC BY 4.0, JSONL.
-
bagnabit2warc github.com
Convert a bag-nabit dataset stored in a ZIP into a full-content WARC.
-
WarcPartitioner github.com
Partition (W)ARC Files by MIME Type and Year. (Stable)
