Data Engineering
313 items from igorbarinov/awesome-data-engineering ★9,166
-
Druid github.com
Column oriented distributed data store ideal for powering interactive applications.
-
kafka-manager github.com
A tool for managing Apache Kafka.
-
Kreuzberg github.com
Polyglot document intelligence library with a Rust core and bindings for Python, TypeScript, Go, and more. Extracts text, tables, and metadata from 62+ document formats for data pipeline ingestion.
-
Desbordante github.com
An open-source data profiler specifically focused on discovery and validation of complex patterns in data.
-
DataFlow github.com
Open-source platform for data preparation, synthetic data generation, and AI/data pipelines. Includes reusable skills for automating workflow steps across data and AI tasks.
-
Secor github.com
Pinterest's Kafka to S3 distributed consumer.
-
Gaffer github.com
A large-scale graph database.
-
Bruin github.com
End-to-end data pipeline tool that combines ingestion, transformation (SQL + Python), and data quality in a single CLI. Connects to BigQuery, Snowflake, PostgreSQL, Redshift, and more. Includes VS Code extension with live previews.
-
kairosdb github.com
Fast scalable time series database.
-
Substation github.com
A cloud native data pipeline and transformation toolkit written in Go.
-
PyHive github.com
Python interface to Hive and Presto.
-
kafka-docker github.com
Kafka in Docker.
-
Data Profiler github.com
The DataProfiler is a Python library designed to make data analysis, monitoring, and sensitive data detection easy.
-
Snappy github.com
A fast compressor/decompressor. Used with Parquet.
-
Weave github.com
Weaving Docker containers into applications.
-
SouravRoy-ETL/duckle github.com
Visual-first, open-source data studio (ETL/ELT) that runs entirely on DuckDB. Drag sources, transforms and sinks onto a canvas and it compiles to plain DuckDB SQL; 300+ connectors and a built-in MCP server.
-
Delight github.com
A free & cross platform monitoring tool (Spark UI / Spark History Server alternative).
-
HyperDex github.com
A scalable, searchable key-value store. Deprecated.
-
kafkacat github.com
Generic command line non-JVM Apache Kafka producer and consumer.
-
lakeFS github.com
An open source platform that delivers resilience and manageability to object-storage based data lakes.
-
Jitsu github.com
An open-source Customer Data Platform. Captures event data from websites, apps, and servers and streams it into ClickHouse, Snowflake, BigQuery, Redshift, Postgres, and MySQL in real time.
-
OpenTSDB github.com
A scalable, distributed Time Series Database.
-
ZomboDB github.com
PostgreSQL Extension that allows creating an index backed by Elasticsearch.
-
librdkafka github.com
Apache Kafka client library for C and C++. [BSD-2-Clause]
-
Snappydata github.com
OLTP + OLAP Database built on Apache Spark.
-
Actionbase github.com
A database for user interactions (likes, views, follows) represented as graphs, with precomputed reads served in real-time.
-
Akumuli github.com
A numeric time-series database. It can be used to capture, store and process time-series data in real-time. The word "akumuli" can be translated from esperanto as "accumulate".
-
smart_open github.com
Utils for streaming large files (S3, HDFS, gzip, bz2).
-
Flocker github.com
Easily manage Docker containers & their data.
-
FlockDB github.com
A distributed, fault-tolerant graph database by Twitter. Deprecated.
-
Gravitino github.com
An open-source, unified metadata management for data lakes, data warehouses, and external catalogs.
-
HStreamDB github.com
The streaming database built for IoT data storage and real-time processing.
-
Dalmatiner DB github.com
Fast distributed metrics database.
-
elasticsearch-jdbc github.com
JDBC importer for Elasticsearch.
-
PipelineDB github.com
The Streaming SQL Database.
-
HAProxy Exporter github.com
Simple server that scrapes HAProxy stats and exports them via HTTP for Prometheus consumption.
-
kafka-node github.com
Node.js client for Apache Kafka 0.8.
-
Blueflood github.com
A distributed system designed to ingest and process time series data.
-
Gobblin github.com
Universal data ingestion framework for Hadoop from LinkedIn.
-
PyXley github.com
Python helpers for building dashboards using Flask and React.
-
Apache Kafka kafka.apache.org
Publish-subscribe messaging rethought as a distributed commit log.
-
Apache Cassandra cassandra.apache.org
The right choice when you need scalability and high availability without compromising performance.
-
Redash redash.io
Make Your Company Data Driven. Connect to any data source, easily visualize and share your data.
-
Seaborn seaborn.pydata.org
A Python visualization library based on matplotlib. It provides a high-level interface for drawing attractive statistical graphics.
-
AI Dev Jobs aidevboard.com
Job board focused on AI/ML engineering roles with 5,400+ listings and a free REST API.
-
Highcharts highcharts.com
A series of charting libraries for a variety of uses. Can be compatible back to IE6.
-
D3.js d3js.org
Allows the user to manipulate documents based on data to render charts in SVG.
-
C3.js c3js.org
D3 based reusable chart library.
-
Apache Thrift thrift.apache.org
Efficient cross-language IPC/RPC, works between C++, Java, Python, PHP, C#, and many more other languages. Originally developed by Facebook. [Apache2]
-
ArangoDB arangodb.com
A distributed free and open-source database with a flexible data model for documents, graphs, and key-values.
- next page of items loading…
