Web Archiving
186 items from iipc/awesome-web-archiving ★2,661
-
-
IIPC netpreserve.org
-
Warchaeology nationallibraryofnorway.github.io
A collection of tools for inspecting, manipulating, deduplicating and validating WARC-files. (Stable)
-
UK Government Web Archive nationalarchives.gov.uk
Main page for the UKGWA.
-
MirrorWeb mirrorweb.com
-
SiteStory mementoweb.github.io
A transactional archive that selectively captures and stores transactions that take place between a web client (browser) and a web server. (Stable)
-
WARCreate matkelly.com
A Google Chrome extension for archiving an individual webpage or website to a WARC file. (Stable)
-
Retrieving and Archiving Information from Websites by Wael Eskandar and Brad Murray kit.exposingtheinvisible.org
-
Common Crawl CDX API index.commoncrawl.org
Search Common Crawl's CDX URL index.
-
IIPC Slack iipc.slack.com
Ask @netpreserve for access.
-
The warc-specifications iipc.github.io
A community HTML version of the official specification and hub for new proposals.
-
httpreserve.info httpreserve.info
Service to return the status of a web page or save it to the Internet Archive. HTTPreserve includes disambiguation of well-known short link services. It returns JSON via the browser or command line via CURL using GET. Describes web sites using earliest and latest dates in the…
-
Xenu home.snafu.de
Desktop link checker for Windows.
-
Hanzo hanzo.co
-
WASAPI groups.google.com
-
OpenWayback groups.google.com
-
Common Crawl groups.google.com
-
Webrecorder US GovArchive govarchive.us
High-fidelity replay.
-
Wget gnu.org
An open source file retrieval utility that of version 1.14 supports writing warcs. (Stable)
-
GLAM Workbench: Web Archives glam-workbench.github.io
See also this related blog post on 'Asking questions with web archives'.
-
warcdedupe gitlab.com
WARC deduplication tool (and WARC library) written in Rust. (In Development)
-
WebMemex github.com
Browser extension for Firefox and Chrome which lets you archive web pages you visit. (In Development)
-
here github.com
.
-
Heritrix github.com
An open source, extensible, web-scale, archival quality web crawler. (Stable)
-
Heritrix Q&A github.com
A discussion forum for asking questions and getting answers about using Heritrix.
-
Comparison of web archiving software github.com
-
Web Archiving Community github.com
-
Crawl git.autistici.org
A simple web crawler in Golang. (Stable)
-
End of Term Archive eotarchive.org
WARCs, CDX files, parquet url index.
-
Wikipedia's List of Web Archiving Initiatives en.wikipedia.org
-
Common Crawl Foundation discord.gg
-
Common Crawl files data.commoncrawl.org
WARCs, CDX files, parquet url index, parquet host index, etc.
-
The Web Crawl section of COPTR coptr.digipres.org
-
Conifer conifer.rhizome.org
From Rhizome, source available at https://github.com/Rhizome-Conifer. The hosted service was discontinued in June 2026; existing collections remain accessible.
-
Community Archive community-archive.org
Open Twitter Database and API with tools and resources for building on archived Twitter data.
-
Common Crawl Columnar Index commoncrawl.org
SQL-queryable index, with CDX info plus language classification. (Stable)
-
Common Crawl Web Graph commoncrawl.org
A host or domain-level graph of the web, with ranking information. (Stable)
-
Common Crawl Foundation Blog commoncrawl.org
rss
-
Chrome Revolver chromewebstore.google.com
Browser extension: switches between browser tabs.
-
Chrome Open Multiple URLs chromewebstore.google.com
Browser extension: opens multiple URLs and also extracts URLs from text.
-
Chrome link checker chromewebstore.google.com
Browser extension: basic link checker.
-
Chrome Check My Links chromewebstore.google.com
Browser extension: a link checker with more options.
-
Chrome link gopher chromewebstore.google.com
Browser extension: link harvester on a page.
-
Continuing Education to Advance Web Archiving (CEDWARC) cedwarc.github.io
-
Common Crawl Foundation Partners ccfpartners.slack.com
(ask greg zat commoncrawl zot org for an invite)
-
UK Web Archive Blog blogs.bl.uk
-
DSHR's Blog blog.dshr.org
David Rosenthal regularly reviews and summarizes work done in the Digital Preservation field.
-
Offical ISO 28500 WARC specification homepage bibnum.bnf.fr
-
Archives Unleashed Toolkit documentation aut.docs.archivesunleashed.org
-
Arkiwera arkiwera.se
- next page of items loading…
