Data Engineering
313 items from igorbarinov/awesome-data-engineering ★9,166
-
DynamoDB aws.amazon.com
[dynamodb]
-
MongoDB mongodb.com
Scalable, flexible document database with querying and indexing.
-
Robinhood's Faust github.com
Forever scalable event processing & in-memory durable K/V store as a library with asyncio & static typing.
-
SlothDB github.com
an embedded SQL database that runs everywhere: on your laptop, on a server, and in the browser. [MIT] website
-
Kuiper github.com
An edge lightweight IoT data analytics/streaming software implemented by Golang, and it can be run at all kinds of resource-constrained edge devices.
-
Timely github.com
A time series database application that provides secure access to time series data based on Accumulo and Grafana.
-
Nika github.com
Intent-as-code workflow engine for AI data pipelines: reviewable YAML DAGs statically checked (schema, permits, cost floor) before execution, with tamper-evident run traces.
-
Project Nessie github.com
A Transactional Catalog for Data Lakes with Git-like semantics. Works with Apache Iceberg tables.
-
FiloDB github.com
Distributed. Columnar. Versioned. Streaming. SQL.
-
S3QL github.com
A file system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack.
-
Omnigraph github.com
Typed graph database where agents branch and merge like Git. S3-native, Rust, traversal + vector + BM25 in one runtime.
-
CCM github.com
A script to easily create and destroy an Apache Cassandra cluster on localhost.
-
FlexViz github.com
Python library for interactive, cross-filtered dashboards that stay responsive on 100M+ rows by aggregating with Polars on the server.
-
dna-claude-analysis github.com
Personal genome analysis toolkit with Python scripts analyzing raw DNA data across 17 categories (health risks, ancestry, pharmacogenomics, nutrition, psychology, etc.) and generating a terminal-style single-page HTML visualization.
-
Pinball github.com
DAG based workflow manager. Job flows are defined programmatically in Python. Support output passing between jobs.
-
mysql_utils github.com
Pinterest MySQL Management Tools.
-
Snakebite github.com
A pure python HDFS client.
-
Heroic github.com
A scalable time series database based on Cassandra and Elasticsearch, by Spotify.
-
FirstData github.com
The world's most comprehensive authoritative data source knowledge base. 210+ curated sources from governments, international organizations, and research institutions. MCP integration for AI agents. MIT licensed.
-
Seaweed File System github.com
Fast, Simple and Scalable Distributed File System with O(1) disk seek.
-
Gockerize github.com
Package golang service into minimal Docker containers.
-
dbmask github.com
Open-source tool that scans SQL databases for sensitive columns, masks them with deterministic fakes, and validates the masked copy row by row against the original.
-
IonDB github.com
A key-value store for microcontroller and IoT applications.
-
MemDB github.com
Distributed Transactional In-Memory Database (based on MongoDB).
-
Eventsim github.com
Event data simulator. Generates a stream of pseudo-random events from a set of users, designed to simulate web traffic.
-
dvt github.com
Data Validation Tool compares data from source and target tables to ensure that they match. It provides column validation, row validation, schema validation, custom query validation, and ad hoc SQL exploration.
-
kafkat github.com
Simplified command-line administration for Kafka brokers.
-
Dekart github.com
Open-source SQL to map platform for BigQuery, Snowflake, and PostGIS.
-
Rocker-compose github.com
Docker composition tool with idempotency features for deploying apps composed of multiple containers. Deprecated.
-
ReductStore github.com
High-performance blob and time-series storage, with edge deployment, selective replication, and efficient querying of multimodal data.
-
Kroxylicious github.com
A Kafka Proxy, solving problems like encrypting your Kafka data at rest.
-
RethinkDB rethinkdb.com
The open-source database for the realtime web.
-
Crate.IO crate.io
Scalable SQL database with the NOSQL goodies.
-
Apache Spark spark.apache.org
A multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters.
-
ElasticSearch elastic.co
Search & Analyze Data in Real Time.
-
Apache Avro avro.apache.org
Apache Avro™ is a data serialization system.
-
Storm storm.apache.org
A free and open source distributed realtime computation system.
-
Flink flink.apache.org
A streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams.
-
ZingChart zingchart.com
Built to render large data sets other libraries can't handle. Super customizable styling and new features added often. Everything in one modular library.
-
Datomic datomic.com
The fully transactional, cloud-ready, distributed database.
-
parquet parquet.apache.org
Columnar storage format that uses the record shredding and assembly algorithm described in the Dremel paper.
-
Grai github.com
A data catalog tool that integrates into your CI system exposing downstream impact testing of data changes. These tests prevent data changes which might break data pipelines or BI dashboards from making it to production.
-
Hivemall github.com
Scalable machine learning library for Hive/Hadoop.
-
Kyoto Tycoon github.com
A lightweight network server on top of the Kyoto Cabinet key-value database, built for high-performance and concurrency.
-
docker-logstash github.com
A highly configurable Logstash (1.4.4) - Docker image running Elasticsearch (1.7.0) - and Kibana (3.1.2).
-
Zodiac github.com
A lightweight tool for easy deployment and rollback of dockerized applications.
-
Deep Spark github.com
Connecting Apache Spark with different data stores. Deprecated.
-
DQOps github.com
An open-source data quality platform for the whole data platform lifecycle from profiling new data sources to applying full automation of data quality monitoring.
-
Scriptella ETL github.com
Open-source, Java-based ETL and script execution tool for transferring and transforming data between databases, files, and other sources.
-
pg-kafka github.com
A PostgreSQL extension to produce messages to Apache Kafka.
- next page of items loading…
