Big Data
1,992 items in 9 lists
-
Twitter Fatcache github.com
key/value cache for flash storage.
-
S3QL github.com
A file system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack.
-
Omnigraph github.com
Typed graph database where agents branch and merge like Git. S3-native, Rust, traversal + vector + BM25 in one runtime.
-
CCM github.com
A script to easily create and destroy an Apache Cassandra cluster on localhost.
-
AthenaX github.com
a streaming analytics platform that enables users to run production-quality, large scale streaming analytics using Structured Query Language (SQL).
-
FlexViz github.com
Python library for interactive, cross-filtered dashboards that stay responsive on 100M+ rows by aggregating with Polars on the server.
-
GraphFrames github.com
Data frame based graph API.
-
isolation-forest github.com
distributed Spark and Scala implementation of isolation forest for unsupervised outlier detection.
-
dna-claude-analysis github.com
Personal genome analysis toolkit with Python scripts analyzing raw DNA data across 17 categories (health risks, ancestry, pharmacogenomics, nutrition, psychology, etc.) and generating a terminal-style single-page HTML visualization.
-
Pinball github.com
DAG based workflow manager. Job flows are defined programmatically in Python. Support output passing between jobs.
-
Bistro github.com
general-purpose data processing engine for both batch and stream analytics. It is based on a novel data model, which represents data via functions and processes data via column operations as opposed to having only set operations in conventional approaches like MapReduce or SQL.
-
graspologic github.com
-
sparklyr github.com
An alternative R backend, using dplyr.
-
411 github.com
an web application for alert management resulting from scheduled searches into Elasticsearch.
-
Livy github.com
REST server with extensive language support (Python, R, Scala), ability to maintain interactive sessions and object sharing.
-
Twitter Twemcache github.com
fork of Memcache.
-
BayesDB github.com
statistic oriented SQL database.
-
mysql_utils github.com
Pinterest MySQL Management Tools.
-
Hermes github.com
asynchronous message broker built on top of Kafka.
-
Snakebite github.com
A pure python HDFS client.
-
Heroic github.com
A scalable time series database based on Cassandra and Elasticsearch, by Spotify.
-
python-deequ github.com
Python API for Deequ.
-
White Elephant github.com
log aggregator and dashboard.
-
Gazette github.com
Distributed streaming infrastructure built on cloud storage which makes it easy to mix and match batch and streaming paradigms.
-
Photon ML github.com
A machine learning library supporting classical Generalized Mixed Model and Generalized Additive Mixed Effect Model.
-
Rakam github.com
open-source real-time custom analytics platform powered by Postgresql, Kinesis and PrestoDB.
-
FirstData github.com
The world's most comprehensive authoritative data source knowledge base. 210+ curated sources from governments, international organizations, and research institutions. MCP integration for AI agents. MIT licensed.
-
chispa github.com
PySpark test helpers with beautiful error messages.
-
Seaweed File System github.com
Fast, Simple and Scalable Distributed File System with O(1) disk seek.
-
spark-daria github.com
A Scala library with essential Spark functions and extensions to make you more productive.
-
sequenceiq/docker-spark github.com
Yarn images from SequenceIQ.
-
GhostDB github.com
a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.
-
Apache Toree github.com
IPython protocol based middleware for interactive applications.
-
Hannibal github.com
Tool to monitor and maintain HBase clusters.
-
Mongo-Spark github.com
Official MongoDB connector.
-
D3.compose github.com
Compose complex, data-driven visualizations from reusable charts and components.
-
quinn github.com
A native PySpark implementation of spark-daria.
-
dash-cytoscape github.com
-
Archives Unleashed Toolkit github.com
Open-source toolkit for analyzing web archives.
-
Gockerize github.com
Package golang service into minimal Docker containers.
-
Flintrock github.com
A command-line tool for launching Spark clusters on EC2.
-
dbmask github.com
Open-source tool that scans SQL databases for sensitive columns, masks them with deterministic fakes, and validates the masked copy row by row against the original.
-
Scikit-network github.com
-
IonDB github.com
A key-value store for microcontroller and IoT applications.
-
MemDB github.com
Distributed Transactional In-Memory Database (based on MongoDB).
-
LinkedIn Cleo github.com
is a flexible software library for enabling rapid development of partial, out-of-order and real-time typeahead search.
-
visNetwork github.com
-
ElephantDB github.com
Distributed database specialized in exporting data from Hadoop.
-
Eventsim github.com
Event data simulator. Generates a stream of pseudo-random events from a set of users, designed to simulate web traffic.
-
Popoto.js github.com
- next page of items loading…
