Web Archiving
186 items from iipc/awesome-web-archiving ★2,661
-
Squidwarc github.com
An open source, high-fidelity, page interacting archival crawler that uses Chrome or Chrome Headless directly. (In Development)
-
warctools github.com
Library to work with ARC and WARC files (Python).
-
Warcat github.com
Tool and library for handling Web ARChive (WARC) files (Python). (Stable)
-
ArchiveSpark github.com
An Apache Spark framework (not only) for Web Archives that enables easy data processing, extraction as well as derivation. (Stable)
-
FastWARC github.com
A high-performance WARC parsing library (Python).
-
SolrWayback github.com
A backend Java and frontend VUE JS project with freetext search and a build in playback engine. Require Warc files has been index with the Warc-Indexer. The web application also has a wide range of data visualization tools and data export tools that can be used on the whole…
-
webarchive-discovery github.com
WARC and ARC full-text indexing and discovery tools, with a number of associated tools capable of using the index shown below. (Stable)
-
node-warc github.com
Parse WARC files or create WARC files using either Electron or chrome-remote-interface (Node.js). (Stable)
-
Chronicler github.com
Web browser with record and replay functionality. (In Development)
-
MemGator github.com
A Memento Aggregator CLI and Server (Golang). (Stable)
-
ArchiveTools github.com
Collection of tools to extract and interact with WARC files (Python).
-
F(b)arc github.com
A commandline tool and Python library for archiving data from Facebook using the Graph API. (Stable)
-
Common Crawl Jupyter notebooks github.com
A collection of notebooks using Common Crawl's various datasets. (Stable)
-
crau github.com
A lightweight command-line tool for archiving the Web and playing archives: you just need a list of URLs. The name "crau" stems from the Brazilian pronounciation of "crawl". (Stable)
-
jwarc github.com
Read and write WARC files with a type safe API (Java).
-
Warcworker github.com
An open source, dockerized, queued, high fidelity web archiver based on Squidwarc with a simple web GUI. (Stable)
-
warc github.com
A Rust library for reading and writing WARC files. (Stable)
-
Mink github.com
A Google Chrome extension for querying Memento aggregators while browsing and integrating live-archived web navigation. (Stable)
-
warc2html github.com
Converts WARC files to static HTML suitable for browsing offline or rehosting.
-
har2warc github.com
Convert HTTP Archive (HAR) -> Web Archive (WARC) format (Python).
-
Cairn github.com
A npm package and CLI tool for saving webpages as single HTML files. It does not produce WARC or WACZ archives. (Stable)
-
HTTrack httrack.com
Useful tool for downloading a whole website and offline browsing.
-
Warclight github.com
A Project Blacklight based Rails engine that supports the discovery of web archives held in the WARC and ARC formats. (In Development)
-
OutbackCDX github.com
RocksDB-based capture index (CDX) server supporting incremental updates and compression. Can be used as backend for OpenWayback, PyWb and Heritrix. (Stable)
-
webarchive-indexing github.com
Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
-
-
Shine github.com
A prototype web archives exploration UI, developed with researchers as part of the Big UK Domain Data for the Arts and Humanities project. (Stable)
-
crocoite github.com
Crawl websites using headless Google Chrome/Chromium and save resources, static DOM snapshot and page screenshots to WARC files. (In Development)
-
httrack2warc github.com
Convert HTTrack archives to WARC format (Java).
-
Warcat-rs github.com
Command-line tool and Rust library for handling Web ARChive (WARC) files. (In Development)
-
Wasp github.com
A fully functional prototype of a personal web archive and search system. (In Development)
-
Web2Warc github.com
An easy-to-use and highly customizable crawler that enables anyone to create their own little Web archives (WARC/CDX). (Stable)
-
warcbench github.com
A tool for exploring, analyzing, transforming, recombining, and extracting data from WARC (Web ARChive) files.
-
Archives Unleashed Notebooks github.com
Notebooks for working with web archives with the Archives Unleashed Toolkit, and derivatives generated by the Archives Unleashed Toolkit. (Stable)
-
UNT Web Archiving Course github.com
-
html2warc github.com
A simple script to convert offline data into a single WARC file. (Stable)
-
Wget-lua github.com
Wget with Lua extension. (Stable)
-
duckdb-web-archive-cdx github.com
DuckDB extension to query the Internet Archive and CommonCrawl CDX APIs directly from SQL. (In Development)
-
Zotero Robust Links Extension github.com
A Zotero extension that submits to and reads from web archives. Supercedes leonkt/zotero-memento.
-
Archives Research Compute Hub github.com
Web application for distributed compute analysis of Archive-It web archive collections. (Stable)
-
webarchive github.com
Golang readers for ARC and WARC webarchive formats (Golang).
-
gowarcserver github.com
BadgerDB-based capture index (CDX) and WARC record server, used to index and serve WARC files (Go).
-
warc-safe github.com
Automatic detection of viruses and NSFW content in WARC files.
-
Sparkling github.com
Internet Archive's Sparkling Data Processing Library. (Stable)
-
py-wasapi-client github.com
Command line application to download crawls from WASAPI (Python). (Stable)
-
PANDORÆ github.com
A desktop research software to be plugged on a Solr endpoint to query, retrieve, normalize and visually explore web archives. (Stable)
-
Unwarcit github.com
Command line interface to unzip WARC and WACZ files (Python).
-
playback github.com
A toolkit for searching archived webpages from Internet Archive, archive.today, Memento and beyond. (In Development)
-
MementoMap github.com
A Tool to Summarize Web Archive Holdings (Python). (In Development)
-
tikalinkextract github.com
Extract hyperlinks as a seed for web archiving from folders of document types that can be parsed by Apache Tika (Golang, Apache Tika Server). (In Development)
- next page of items loading…
