StormCrawler

stormcrawler.netsite

A collection of resources for building low-latency, scalable web crawlers on Apache Storm. (Stable)

from

Web Archiving

added

2026-10-10

likes

0

similar

1. ▲

   grab-site github.com

   The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns. (Stable)

   tool link ↗
   tool
2. ▲

   Web2Warc github.com

   An easy-to-use and highly customizable crawler that enables anyone to create their own little Web archives (WARC/CDX). (Stable)

   tool link ↗
   
3. ▲

   Storm storm.apache.org

   A free and open source distributed realtime computation system.

   site topic programming link ↗
   
4. ▲

   Common Crawl Foundation Blog commoncrawl.org

   rss

   site link ↗
   site
5. ▲

   webarchive-discovery github.com

   WARC and ARC full-text indexing and discovery tools, with a number of associated tools capable of using the index shown below. (Stable)

   tool link ↗
   
6. ▲

   StackStorm stackstorm.com

   site link ↗
   site

Web Archiving › Tools & Software > Acquisition: “A collection of resources for building low-latency, scalable web crawlers on Apache Storm. (Stable)”