cobweb

rubygems.orgsite

web crawler that can use resque to cluster crawls to quickly crawl extremely large sites.

from

Selenium

added

2026-10-10

likes

0

similar

1. ▲

   Crawler github.com

   A high performance web crawler in Elixir.

   tool topic programming link ↗
   
2. ▲

   Crawly github.com

   high-level web crawling & scraping framework for Elixir.

   tool topic programming link ↗
   tool
3. ▲

   crawley github.com

   Web scraper/crawler for cli.

   tool link ↗
   tool
4. ▲

   WhatWeb github.com

   tool link ↗
   
5. ▲

   webpush github.com

   tool link ↗
   
6. ▲

   Crawl git.autistici.org

   A simple web crawler in Golang. (Stable)

   site topic programming link ↗
   site

Selenium › Resources > Tools > Ruby: “web crawler that can use resque to cluster crawls to quickly crawl extremely large sites.”