cdx-toolkit

pypi.orgsite

Library and CLI to consult cdx indexes and create WARC extractions of subsets. Abstracts away Common Crawl's unusual crawl structure. (Stable)

from

Web Archiving

added

2026-10-10

likes

0

similar

1. ▲
   Common Crawl CDX API index.commoncrawl.org

   site category #web-archiving link ↗

   Search Common Crawl's CDX URL index.

2. ▲
   duckdb-web-archive-cdx github.com

   tool category #web-archiving link ↗

   DuckDB extension to query the Internet Archive and CommonCrawl CDX APIs directly from SQL. (In Development)

3. ▲
   Common Crawl Columnar Index commoncrawl.org

   site category #web-archiving link ↗

   SQL-queryable index, with CDX info plus language classification. (Stable)

4. ▲
   Common Crawl files data.commoncrawl.org

   site category #web-archiving link ↗

   WARCs, CDX files, parquet url index, parquet host index, etc.

5. ▲
   ArchiveTools github.com

   tool category #web-archiving link ↗

   Collection of tools to extract and interact with WARC files (Python).

   
6. ▲
   Web2Warc github.com

   tool category #web-archiving link ↗

   An easy-to-use and highly customizable crawler that enables anyone to create their own little Web archives (WARC/CDX). (Stable)

   

Web Archiving › Tools & Software > Utilities: “Library and CLI to consult cdx indexes and create WARC extractions of subsets. Abstracts away Common Crawl's unusual crawl structure. (Stable)”