Web Archiving
186 items from iipc/awesome-web-archiving ★2,661
-
Heritrix Walkthrough github.com
(In Development)
-
HTTPreserve linkstat github.com
Command line implementation of httpreserve.info to describe the status of a web page. Can be easily scripted and provides JSON output to enable querying through tools like JQ. HTTPreserve Linkstat describes current status, and earliest and latest links on archive.org. (Golang)…
-
Tweet Archvies Unleashed Toolkit github.com
An open-source toolkit for analyzing line-oriented JSON Twitter archives with Apache Spark. (In Development)
-
warcrefs github.com
Web archive deduplication tools. (Stable)
-
HadoopConcatGz github.com
A Splitable Hadoop InputFormat for Concatenated GZIP Files (and *.warc.gz). (Stable)
-
PastPage github.com
Browser extension for Chrome and Firefox that recovers broken or changed pages by querying the Wayback Machine and other web archives in parallel. (Stable)
-
Social Feed Manager gwu-libraries.github.io
Open source software that harvests social media data and web resources from Twitter, Tumblr, Flickr, and Sina Weibo.
-
SecurityTrails securitytrails.com
Web based archive for WHOIS and DNS records. REST API available free of charge.
-
The Unarchiver theunarchiver.com
Unarchive many different kinds of archive files.
-
wasapi-downloader github.com
Java command line application to download crawls from WASAPI. (Stable)
-
duckdb_warc github.com
DuckDB extension to query WARC files. (In Development)
-
Jwat-Tools github.com
Tools for reading/writing/validating WARC/ARC/GZIP files (Java). (Stable)
-
Jwat github.com
Libraries for reading/writing/validating WARC/ARC/GZIP files (Java). (Stable)
-
-
-
node-cdxj github.com
CDXJ file parser (Node.js). (Stable)
-
Dead-Web Index github.com
Reachability labels (alive / blocked / dead) for the top 10 million domains, two probe arms (polite HTTP and a browser TLS fingerprint), 2026. CC BY 4.0, JSONL.
-
bagnabit2warc github.com
Convert a bag-nabit dataset stored in a ZIP into a full-content WARC.
-
WarcPartitioner github.com
Partition (W)ARC Files by MIME Type and Year. (Stable)
-
What is a web archive? youtube.com
A video from the UK Web Archive YouTube Channel
-
WS-DL Blog ws-dl.blogspot.com
Web Science and Digital Libraries Research Group blogs about various Web archiving related topics, scholarly work, and academic trip reports.
-
WineBottler winebottler.kronenberg.org
For running Xenu and Notepad++ on macOS.
-
ReplayWeb.page webrecorder.net
A browser-based, fully client-side replay engine for both local and remote WARC & WACZ files. Also available as an Electron based desktop application. (Stable)
-
Browsertrix webrecorder.net
From Webrecorder, source available at https://github.com/webrecorder/browsertrix.
-
ArchiveWeb.Page webrecorder.net
A plugin for Chrome and other Chromium based browsers that lets you interactively archive web pages, replay them, and export them as WARC & WACZ files. Also available as an Electron based desktop application.
-
Web Data Commons webdatacommons.org
Structured data extracted from Common Crawl. (Stable)
-
WARC Explorer webdata.nb.no
Browser-based inspector for WARC files and records (client-side).
-
Web Curator Tool webcuratortool.org
Open-source workflow management for selective web archiving. (Stable)
-
Web Archiving Roundtable webarchivingrt.wordpress.com
Unofficial blog of the Web Archiving Roundtable of the Society of American Archivists maintained by the members of the Web Archiving Roundtable.
-
Internet Archive Wayback web.archive.org
Base URL for IA's Wayback Machine.
-
The Web as History uclpress.co.uk
An open-source book that provides a conceptual overview to web archiving research, as well as several case studies.
-
#WebArchiving twitter.com
-
#WebArchiveWednesday twitter.com
-
@commoncrawl twitter.com
Official Common Crawl Foundation handle.
-
@WebSciDL twitter.com
ODU Web Science and Digital Libraries Research Group.
-
@NetPreserve twitter.com
Official IIPC handle.
-
Tempas v2 tempas.l3s.de
Temporal web archive search based on links and anchor texts extracted from the German web from 1996 to 2013 (results are not limited to German pages, e.g., Obama@2005-2009 in Tempas). (Stable)
-
Tempas v1 tempas.l3s.de
Temporal web archive search based on Delicious tags. (Stable)
-
Windows Snipping Tool support.microsoft.com
Windows built-in for partial screen capture and annotation. On macOS you can use Command + Shift + 4 (keyboard shortcut for taking partial screen capture).
-
Glossary of Archive-It and Web Archiving Terms support.archive-it.org
-
StormCrawler stormcrawler.net
A collection of resources for building low-latency, scalable web crawlers on Apache Storm. (Stable)
-
Tutorial for Humanities researchers about how to explore Arquivo.pt sobre.arquivo.pt
-
Smarsh smarsh.com
-
cdx-toolkit pypi.org
Library and CLI to consult cdx indexes and create WARC extractions of subsets. Abstracts away Common Crawl's unusual crawl structure. (Stable)
-
PlayOnMac playonmac.com
For running Xenu and Notepad++ on macOS.
-
PlayOnLinux playonlinux.com
For running Xenu and Notepad++ on Ubuntu.
-
PageFreezer pagefreezer.com
-
Reconstructive oduwsdl.github.io
A Service Worker module for client-side reconstruction of composite mementos by rerouting resource requests to corresponding archived copies (JavaScript).
-
Definition of Web Archivability nullhandle.org
This describes the ease with which web content can be preserved. (Archived version from the Stanford Libraries)
-
IIPC Blog netpreserveblog.wordpress.com
- next page of items loading…
