warcrefs

github.comtool

Web archive deduplication tools. (Stable)

from

Web Archiving

added

2026-10-10

likes

0

similar

1. ▲
   warcdedupe gitlab.com

   site category #web-archiving link ↗

   WARC deduplication tool (and WARC library) written in Rust. (In Development)

   
2. ▲
   webarchive-discovery github.com

   tool category #web-archiving link ↗

   WARC and ARC full-text indexing and discovery tools, with a number of associated tools capable of using the index shown below. (Stable)

   
3. ▲
   UNT Web Archiving Course github.com

   tool category #web-archiving link ↗
   
4. ▲
   webarchive-indexing github.com

   tool category #web-archiving link ↗

   Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.

   
5. ▲
   webarchive github.com

   tool category #web-archiving link ↗

   Golang readers for ARC and WARC webarchive formats (Golang).

   
6. ▲
   Web2Warc github.com

   tool category #web-archiving link ↗

   An easy-to-use and highly customizable crawler that enables anyone to create their own little Web archives (WARC/CDX). (Stable)

   

Web Archiving › Tools & Software > Utilities: “Web archive deduplication tools. (Stable)”