Data Engineering
313 items from igorbarinov/awesome-data-engineering ★9,166
-
Druid github.com
Column oriented distributed data store ideal for powering interactive applications.
-
kafka-manager github.com
A tool for managing Apache Kafka.
-
Kreuzberg github.com
Polyglot document intelligence library with a Rust core and bindings for Python, TypeScript, Go, and more. Extracts text, tables, and metadata from 62+ document formats for data pipeline ingestion.
-
Desbordante github.com
An open-source data profiler specifically focused on discovery and validation of complex patterns in data.
-
DataFlow github.com
Open-source platform for data preparation, synthetic data generation, and AI/data pipelines. Includes reusable skills for automating workflow steps across data and AI tasks.
-
Secor github.com
Pinterest's Kafka to S3 distributed consumer.
-
Gaffer github.com
A large-scale graph database.
-
Bruin github.com
End-to-end data pipeline tool that combines ingestion, transformation (SQL + Python), and data quality in a single CLI. Connects to BigQuery, Snowflake, PostgreSQL, Redshift, and more. Includes VS Code extension with live previews.
-
kairosdb github.com
Fast scalable time series database.
-
Substation github.com
A cloud native data pipeline and transformation toolkit written in Go.
-
PyHive github.com
Python interface to Hive and Presto.
-
kafka-docker github.com
Kafka in Docker.
-
Data Profiler github.com
The DataProfiler is a Python library designed to make data analysis, monitoring, and sensitive data detection easy.
-
Snappy github.com
A fast compressor/decompressor. Used with Parquet.
-
Weave github.com
Weaving Docker containers into applications.
-
SouravRoy-ETL/duckle github.com
Visual-first, open-source data studio (ETL/ELT) that runs entirely on DuckDB. Drag sources, transforms and sinks onto a canvas and it compiles to plain DuckDB SQL; 300+ connectors and a built-in MCP server.
-
Delight github.com
A free & cross platform monitoring tool (Spark UI / Spark History Server alternative).
-
HyperDex github.com
A scalable, searchable key-value store. Deprecated.
-
kafkacat github.com
Generic command line non-JVM Apache Kafka producer and consumer.
-
lakeFS github.com
An open source platform that delivers resilience and manageability to object-storage based data lakes.
-
Jitsu github.com
An open-source Customer Data Platform. Captures event data from websites, apps, and servers and streams it into ClickHouse, Snowflake, BigQuery, Redshift, Postgres, and MySQL in real time.
-
OpenTSDB github.com
A scalable, distributed Time Series Database.
-
ZomboDB github.com
PostgreSQL Extension that allows creating an index backed by Elasticsearch.
-
librdkafka github.com
Apache Kafka client library for C and C++. [BSD-2-Clause]
-
Snappydata github.com
OLTP + OLAP Database built on Apache Spark.
-
Actionbase github.com
A database for user interactions (likes, views, follows) represented as graphs, with precomputed reads served in real-time.
-
Akumuli github.com
A numeric time-series database. It can be used to capture, store and process time-series data in real-time. The word "akumuli" can be translated from esperanto as "accumulate".
-
smart_open github.com
Utils for streaming large files (S3, HDFS, gzip, bz2).
-
Flocker github.com
Easily manage Docker containers & their data.
-
FlockDB github.com
A distributed, fault-tolerant graph database by Twitter. Deprecated.
-
Gravitino github.com
An open-source, unified metadata management for data lakes, data warehouses, and external catalogs.
-
HStreamDB github.com
The streaming database built for IoT data storage and real-time processing.
-
Dalmatiner DB github.com
Fast distributed metrics database.
-
elasticsearch-jdbc github.com
JDBC importer for Elasticsearch.
-
PipelineDB github.com
The Streaming SQL Database.
-
HAProxy Exporter github.com
Simple server that scrapes HAProxy stats and exports them via HTTP for Prometheus consumption.
-
kafka-node github.com
Node.js client for Apache Kafka 0.8.
-
Blueflood github.com
A distributed system designed to ingest and process time series data.
-
Gobblin github.com
Universal data ingestion framework for Hadoop from LinkedIn.
-
PyXley github.com
Python helpers for building dashboards using Flask and React.
-
Robinhood's Faust github.com
Forever scalable event processing & in-memory durable K/V store as a library with asyncio & static typing.
-
SlothDB github.com
an embedded SQL database that runs everywhere: on your laptop, on a server, and in the browser. [MIT] website
-
Kuiper github.com
An edge lightweight IoT data analytics/streaming software implemented by Golang, and it can be run at all kinds of resource-constrained edge devices.
-
Timely github.com
A time series database application that provides secure access to time series data based on Accumulo and Grafana.
-
Nika github.com
Intent-as-code workflow engine for AI data pipelines: reviewable YAML DAGs statically checked (schema, permits, cost floor) before execution, with tamper-evident run traces.
-
Project Nessie github.com
A Transactional Catalog for Data Lakes with Git-like semantics. Works with Apache Iceberg tables.
-
FiloDB github.com
Distributed. Columnar. Versioned. Streaming. SQL.
-
S3QL github.com
A file system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack.
-
Omnigraph github.com
Typed graph database where agents branch and merge like Git. S3-native, Rust, traversal + vector + BM25 in one runtime.
-
CCM github.com
A script to easily create and destroy an Apache Cassandra cluster on localhost.
- next page of items loading…
