duckdb-web-archive-cdx

github.comtool

DuckDB extension to query the Internet Archive and CommonCrawl CDX APIs directly from SQL. (In Development)

from

Web Archiving

added

2026-10-10

likes

0

similar

1. ▲
   duckdb_warc github.com

   tool category #web-archiving link ↗

   DuckDB extension to query WARC files. (In Development)

2. ▲
   cdx-toolkit pypi.org

   site category #web-archiving link ↗

   Library and CLI to consult cdx indexes and create WARC extractions of subsets. Abstracts away Common Crawl's unusual crawl structure. (Stable)

3. ▲
   OutbackCDX github.com

   tool category #web-archiving link ↗

   RocksDB-based capture index (CDX) server supporting incremental updates and compression. Can be used as backend for OpenWayback, PyWb and Heritrix. (Stable)

4. ▲
   Common Crawl CDX API index.commoncrawl.org

   site category #web-archiving link ↗

   Search Common Crawl's CDX URL index.

5. ▲
   Common Crawl Columnar Index commoncrawl.org

   site category #web-archiving link ↗

   SQL-queryable index, with CDX info plus language classification. (Stable)

6. ▲
   A Whirlwind Tour of Common Crawl's Datasets using Java github.com

   tool category #web-archiving link ↗
   

Web Archiving › Tools & Software > Utilities: “DuckDB extension to query the Internet Archive and CommonCrawl CDX APIs directly from SQL. (In Development)”