Data Engineering
313 items from igorbarinov/awesome-data-engineering ★9,166
-
FlexViz github.com
Python library for interactive, cross-filtered dashboards that stay responsive on 100M+ rows by aggregating with Polars on the server.
-
dna-claude-analysis github.com
Personal genome analysis toolkit with Python scripts analyzing raw DNA data across 17 categories (health risks, ancestry, pharmacogenomics, nutrition, psychology, etc.) and generating a terminal-style single-page HTML visualization.
-
Pinball github.com
DAG based workflow manager. Job flows are defined programmatically in Python. Support output passing between jobs.
-
mysql_utils github.com
Pinterest MySQL Management Tools.
-
Snakebite github.com
A pure python HDFS client.
-
Heroic github.com
A scalable time series database based on Cassandra and Elasticsearch, by Spotify.
-
FirstData github.com
The world's most comprehensive authoritative data source knowledge base. 210+ curated sources from governments, international organizations, and research institutions. MCP integration for AI agents. MIT licensed.
-
Seaweed File System github.com
Fast, Simple and Scalable Distributed File System with O(1) disk seek.
-
Gockerize github.com
Package golang service into minimal Docker containers.
-
dbmask github.com
Open-source tool that scans SQL databases for sensitive columns, masks them with deterministic fakes, and validates the masked copy row by row against the original.
-
IonDB github.com
A key-value store for microcontroller and IoT applications.
-
MemDB github.com
Distributed Transactional In-Memory Database (based on MongoDB).
-
Eventsim github.com
Event data simulator. Generates a stream of pseudo-random events from a set of users, designed to simulate web traffic.
-
dvt github.com
Data Validation Tool compares data from source and target tables to ensure that they match. It provides column validation, row validation, schema validation, custom query validation, and ad hoc SQL exploration.
-
kafkat github.com
Simplified command-line administration for Kafka brokers.
-
Dekart github.com
Open-source SQL to map platform for BigQuery, Snowflake, and PostGIS.
-
Rocker-compose github.com
Docker composition tool with idempotency features for deploying apps composed of multiple containers. Deprecated.
-
ReductStore github.com
High-performance blob and time-series storage, with edge deployment, selective replication, and efficient querying of multimodal data.
-
Kroxylicious github.com
A Kafka Proxy, solving problems like encrypting your Kafka data at rest.
-
Grai github.com
A data catalog tool that integrates into your CI system exposing downstream impact testing of data changes. These tests prevent data changes which might break data pipelines or BI dashboards from making it to production.
-
Hivemall github.com
Scalable machine learning library for Hive/Hadoop.
-
Kyoto Tycoon github.com
A lightweight network server on top of the Kyoto Cabinet key-value database, built for high-performance and concurrency.
-
docker-logstash github.com
A highly configurable Logstash (1.4.4) - Docker image running Elasticsearch (1.7.0) - and Kibana (3.1.2).
-
Zodiac github.com
A lightweight tool for easy deployment and rollback of dockerized applications.
-
Deep Spark github.com
Connecting Apache Spark with different data stores. Deprecated.
-
DQOps github.com
An open-source data quality platform for the whole data platform lifecycle from profiling new data sources to applying full automation of data quality monitoring.
-
Scriptella ETL github.com
Open-source, Java-based ETL and script execution tool for transferring and transforming data between databases, files, and other sources.
-
pg-kafka github.com
A PostgreSQL extension to produce messages to Apache Kafka.
-
OrionBelt Semantic Layer github.com
Open-source semantic sidecar that compiles YAML-defined dimensions, measures, and metrics into optimized SQL across 8 engines (BigQuery, ClickHouse, Databricks, Dremio, DuckDB, MySQL, PostgreSQL, Snowflake). Unified REST, MCP, and Postgres wire protocol; one model powers AI…
-
pdfmux github.com
Python PDF-to-Markdown orchestrator. Classifies each page and routes to the optimal backend (PyMuPDF, Docling, RapidOCR, Gemini Flash), emitting Markdown plus a per-page confidence score so ingestion pipelines can quarantine low-trust pages before feeding LLMs or retrieval.
-
daffy github.com
Decorator-first DataFrame contracts/validation (columns/dtypes/constraints) at function boundaries. Supports Pandas/Polars/PyArrow/Modin.
-
FutureSearch SDK github.com
Python SDK that dispatches parallel web-research agents across table rows, synthesizing multi-agent findings into structured columns.
-
Snowflake Emulator github.com
A Snowflake-compatible emulator for local development and testing.
-
datatrax github.com
Data engineering and classic ML toolkit with batch processing, type coercion, and 7 algorithms in pure Go with zero dependencies.
-
Kafka-logger github.com
Kafka-winston logger for Node.js from Uber.
-
PACE github.com
An open source framework that allows you to enforce agreements on how data should be accessed, used, and transformed, regardless of the data platform (Snowflake, BigQuery, DataBricks, etc.)
-
QueryGPT github.com
Natural language database query interface with automatic chart generation, supporting Chinese and English queries.
-
drt github.com
OSS Reverse ETL CLI. Sync data from warehouses to business tools via YAML.
-
Google Sheets ETL github.com
Live import all your Google Sheets to your data warehouse.
-
AKF github.com
The AI native file format. Trust scores, source provenance, and compliance metadata that embed into 20+ formats (DOCX, PDF, images, code). EXIF for AI.
-
OneQuery github.com
Self-hosted gateway for safe, auditable queries for agents across approved data sources.
-
Provero github.com
A vendor-neutral, declarative data quality engine. Define checks in YAML, run anywhere. Includes 16 built-in check types, SQL batch optimizer, anomaly detection, and data contracts.
-
data-genie github.com
High-performance, streaming-first ETL engine for Node.js and TypeScript with constant memory footprint.
-
faucet-stream github.com
Config-driven data-movement platform for Rust with pluggable source and sink connectors, running ETL, CDC, and streaming pipelines declaratively from YAML or embedded as a library.
-
Micro S3 persistence github.com
Docker microservice for saving/restoring volume data to S3.
-
SnackFS github.com
A bite-sized, lightweight HDFS compatible file system built over Cassandra.
-
Scherlok github.com
Zero-config data quality CLI. Profiles every table on first run, then auto-detects anomalies (volume drops, schema drift, freshness misses, distribution shifts) on subsequent runs. No YAML, no rules to write. Works with Postgres, BigQuery, Snowflake, and dbt.
-
Rawbbit github.com
Open-source self-hosted game analytics pipeline. HTTP event collector with NATS JetStream buffering, raw Parquet in object storage you own, and ClickHouse for queries via Metabase, SQL, or a read-only MCP server for AI agents. Designed for teams that want to own their raw game…
-
Dotflow github.com
A lightweight Python library for building execution pipelines with retry, parallel execution, cron scheduling, and async support.
-
koala-diff github.com
A high-performance Python library for comparing large datasets (CSV, Parquet) locally using Rust and Polars. It features zero-copy streaming to prevent OOM errors and generates interactive HTML data quality reports.
- next page of items loading…
