warcdedupe

gitlab.comsite

WARC deduplication tool (and WARC library) written in Rust. (In Development)

from

Web Archiving

added

2026-10-10

likes

0

similar

1. ▲
   warcrefs github.com

   tool category #web-archiving link ↗

   Web archive deduplication tools. (Stable)

   
2. ▲
   warc github.com

   tool category #web-archiving link ↗

   A Rust library for reading and writing WARC files. (Stable)

3. ▲
   Web2Warc github.com

   tool category #web-archiving link ↗

   An easy-to-use and highly customizable crawler that enables anyone to create their own little Web archives (WARC/CDX). (Stable)

   
4. ▲
   warctools github.com

   tool category #web-archiving link ↗

   Library to work with ARC and WARC files (Python).

   
5. ▲
   FastWARC github.com

   tool category #web-archiving link ↗

   A high-performance WARC parsing library (Python).

   
6. ▲
   duckdb_warc github.com

   tool category #web-archiving link ↗

   DuckDB extension to query WARC files. (In Development)

Web Archiving › Tools & Software > Utilities: “WARC deduplication tool (and WARC library) written in Rust. (In Development)”