Common Crawl Columnar Index

commoncrawl.orgsite

SQL-queryable index, with CDX info plus language classification. (Stable)

from

Web Archiving

added

2026-10-10

likes

0

similar

1. ▲
   Common Crawl CDX API index.commoncrawl.org

   site category #web-archiving link ↗

   Search Common Crawl's CDX URL index.

2. ▲
   Common Crawl commoncrawl.org

   site category #data-engineering link ↗

   Open source repository of web crawl data.

3. ▲
   Common Crawl Foundation Blog commoncrawl.org

   site category #web-archiving link ↗

   rss

4. ▲
   Common Crawl files data.commoncrawl.org

   site category #web-archiving link ↗

   WARCs, CDX files, parquet url index, parquet host index, etc.

5. ▲
   Common Crawl groups.google.com

   site category #web-archiving link ↗
6. ▲
   cdx-toolkit pypi.org

   site category #web-archiving link ↗

   Library and CLI to consult cdx indexes and create WARC extractions of subsets. Abstracts away Common Crawl's unusual crawl structure. (Stable)

Web Archiving › Tools & Software > Analysis: “SQL-queryable index, with CDX info plus language classification. (Stable)”