Big Data
1,992 items in 9 lists
-
VivaGraphJS github.com
-
ScyllaDB github.com
NoSQL data store using the seastar framework, compatible with Apache Cassandra.
-
Tarantool github.com
An in-memory database and application server.
-
Scalding github.com
Scala library for Map Reduce jobs, built on Cascading.
-
Thanos github.com
Thanos is a set of components to create a highly available metric system with unlimited storage capacity using multiple (existing) Prometheus deployments.
-
Druid github.com
Column oriented distributed data store ideal for powering interactive applications
-
Druid github.com
Column oriented distributed data store ideal for powering interactive applications.
-
Little Ball of Fur github.com
A library for sampling graph structured data.
-
chdb github.com
in-process OLAP SQL engine powered by ClickHouse, callable from Python with native pandas/Arrow DataFrame interop.
-
Twemproxy github.com
A fast, light-weight proxy for memcached and redis.
-
Numaflow (⭐2.8k) github.com
Kubernetes-native, serverless platform for massively parallel stream and data processing, with exactly-once semantics and language-agnostic user code.
-
kafka-manager github.com
A tool for managing Apache Kafka.
-
Yelp/mrjob :fire::fire::fire::fire::fire: github.com
-
Confluent Schema Registry (⭐2.4k) github.com
Widely used registry for Kafka, providing a RESTful interface and compatibility checks for Avro, JSON Schema, and Protobuf schemas.
-
PigPen github.com
map-reduce for Clojure which compiles to Apache Pig.
-
Gobblin from LinkedIn github.com
linkedin's universal data ingestion framework.
-
Kreuzberg github.com
Polyglot document intelligence library with a Rust core and bindings for Python, TypeScript, Go, and more. Extracts text, tables, and metadata from 62+ document formats for data pipeline ingestion.
-
Apache Iceberg github.com
Upserts, Deletes And Incremental Processing on Big Data..
-
Desbordante github.com
An open-source data profiler specifically focused on discovery and validation of complex patterns in data.
-
Delta Lake github.com
Storage layer with ACID transactions.
-
Summing Bird github.com
Streaming MapReduce with Scalding and Storm, by Twitter.
-
BigDL github.com
Distributed Deep Learning library.
-
DataFlow github.com
Open-source platform for data preparation, synthetic data generation, and AI/data pipelines. Includes reusable skills for automating workflow steps across data and AI tasks.
-
Secor github.com
Pinterest's Kafka to S3 distributed consumer.
-
Gaffer github.com
A large-scale graph database.
-
Bruin github.com
End-to-end data pipeline tool that combines ingestion, transformation (SQL + Python), and data quality in a single CLI. Connects to BigQuery, Snowflake, PostgreSQL, Redshift, and more. Includes VS Code extension with live previews.
-
Rustworkx github.com
A high performance Python graph library implemented in Rust.
-
kairosdb github.com
Fast scalable time series database.
-
Substation github.com
A cloud native data pipeline and transformation toolkit written in Go.
-
PyHive github.com
Python interface to Hive and Presto.
-
kafka-docker github.com
Kafka in Docker.
-
Data Profiler github.com
The DataProfiler is a Python library designed to make data analysis, monitoring, and sensitive data detection easy.
-
Snappy github.com
A fast compressor/decompressor. Used with Parquet.
-
Weave github.com
Weaving Docker containers into applications.
-
envisionjs github.com
dynamic HTML5 visualization.
-
SouravRoy-ETL/duckle github.com
Visual-first, open-source data studio (ETL/ELT) that runs entirely on DuckDB. Drag sources, transforms and sinks onto a canvas and it compiles to plain DuckDB SQL; 300+ connectors and a built-in MCP server.
-
Apache Hudi github.com
Upserts, Deletes And Incremental Processing on Big Data..
-
HiBench github.com
a Hadoop benchmark suite.
-
Delight github.com
A free & cross platform monitoring tool (Spark UI / Spark History Server alternative).
-
HyperDex github.com
A scalable, searchable key-value store. Deprecated.
-
Cronicle github.com
Distributed, easy to install, NodeJS based, task scheduler
-
kafkacat github.com
Generic command line non-JVM Apache Kafka producer and consumer.
-
lakeFS github.com
An open source platform that delivers resilience and manageability to object-storage based data lakes.
-
YCSB github.com
-
Jitsu github.com
An open-source Customer Data Platform. Captures event data from websites, apps, and servers and streams it into ClickHouse, Snowflake, BigQuery, Redshift, Postgres, and MySQL in real time.
-
OpenTSDB github.com
A scalable, distributed Time Series Database.
-
Elephant Bird github.com
libraries for working with LZOP-compressed data.
-
ZomboDB github.com
PostgreSQL Extension that allows creating an index backed by Elasticsearch.
-
Cromwell github.com
Workflow management system with Spark backend.
-
Hail github.com
Genetic analysis framework.
- next page of items loading…
