Data Engineering
313 items from igorbarinov/awesome-data-engineering ★9,166
-
OrionBelt Semantic Layer github.com
Open-source semantic sidecar that compiles YAML-defined dimensions, measures, and metrics into optimized SQL across 8 engines (BigQuery, ClickHouse, Databricks, Dremio, DuckDB, MySQL, PostgreSQL, Snowflake). Unified REST, MCP, and Postgres wire protocol; one model powers AI…
-
pdfmux github.com
Python PDF-to-Markdown orchestrator. Classifies each page and routes to the optimal backend (PyMuPDF, Docling, RapidOCR, Gemini Flash), emitting Markdown plus a per-page confidence score so ingestion pipelines can quarantine low-trust pages before feeding LLMs or retrieval.
-
daffy github.com
Decorator-first DataFrame contracts/validation (columns/dtypes/constraints) at function boundaries. Supports Pandas/Polars/PyArrow/Modin.
-
FutureSearch SDK github.com
Python SDK that dispatches parallel web-research agents across table rows, synthesizing multi-agent findings into structured columns.
-
Snowflake Emulator github.com
A Snowflake-compatible emulator for local development and testing.
-
PostgreSQL postgresql.org
The world's most advanced open source database.
-
MySQL mysql.com
The world's most popular open source database.
-
Census getcensus.com
A reverse-ETL tool that let you sync data from your cloud data warehouse to SaaS applications like Salesforce, Marketo, HubSpot, Zendesk, etc. No engineering favors required—just SQL.
-
FluentD fluentd.org
An open source data collector for unified logging layer.
-
Apache Beam beam.apache.org
A unified programming model that implements both batch and streaming data processing jobs that run on many execution engines.
-
Apache Tez tez.apache.org
An application framework which allows for a complex directed-acyclic-graph of tasks for processing data.
-
Apache Sqoop sqoop.apache.org
A tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases.
-
Apache Samza samza.apache.org
A distributed stream processing framework.
-
Apache Oozie oozie.apache.org
A workflow scheduler system to manage Apache Hadoop jobs.
-
Apache NiFi nifi.apache.org
An easy to use, powerful, and reliable system to process and distribute data.
-
Apache Mahout mahout.apache.org
An environment for quickly creating scalable performant machine learning applications.
-
Apache Hive hive.apache.org
Data warehouse software facilitates querying and managing large datasets residing in distributed storage.
-
Apache HBase hbase.apache.org
The Hadoop database, a distributed, scalable, big data store.
-
Apache Drill drill.apache.org
Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud Storage.
-
Cascading cascading.org
Java based application development platform.
-
Azkaban azkaban.github.io
A batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy-to-use web user interface to maintain and track your workflows.
-
d3plus d3plus.org
D3's simpler, easier to use cousin. Mostly predefined templates that you can just plug data in.
-
QuestDB questdb.io
A relational column-oriented database designed for real-time analytics on time series and event data.
-
DuckDB duckdb.org
in-process SQL OLAP Database Management System. [MIT] website
-
Simple Storage Service (S3) aws.amazon.com
Object storage built to retrieve any amount of data from anywhere.
-
Redshift aws.amazon.com
A fast, fully managed, petabyte-scale data warehouse that makes it simple and cost-effective to analyze all your data using your existing business intelligence tools.
-
Relational Database Service (RDS) aws.amazon.com
Makes it easy to set up, operate, and scale a relational database in the cloud.
-
Kinesis aws.amazon.com
A fully managed, cloud-based service for real-time data processing over large, distributed data streams.
-
neo4j neo4j.com
The leading graph database!
-
datatrax github.com
Data engineering and classic ML toolkit with batch processing, type coercion, and 7 algorithms in pure Go with zero dependencies.
-
Kafka-logger github.com
Kafka-winston logger for Node.js from Uber.
-
PACE github.com
An open source framework that allows you to enforce agreements on how data should be accessed, used, and transformed, regardless of the data platform (Snowflake, BigQuery, DataBricks, etc.)
-
QueryGPT github.com
Natural language database query interface with automatic chart generation, supporting Chinese and English queries.
-
drt github.com
OSS Reverse ETL CLI. Sync data from warehouses to business tools via YAML.
-
Google Sheets ETL github.com
Live import all your Google Sheets to your data warehouse.
-
AKF github.com
The AI native file format. Trust scores, source provenance, and compliance metadata that embed into 20+ formats (DOCX, PDF, images, code). EXIF for AI.
-
OneQuery github.com
Self-hosted gateway for safe, auditable queries for agents across approved data sources.
-
Provero github.com
A vendor-neutral, declarative data quality engine. Define checks in YAML, run anywhere. Includes 16 built-in check types, SQL batch optimizer, anomaly detection, and data contracts.
-
data-genie github.com
High-performance, streaming-first ETL engine for Node.js and TypeScript with constant memory footprint.
-
faucet-stream github.com
Config-driven data-movement platform for Rust with pluggable source and sink connectors, running ETL, CDC, and streaming pipelines declaratively from YAML or embedded as a library.
-
Micro S3 persistence github.com
Docker microservice for saving/restoring volume data to S3.
-
SnackFS github.com
A bite-sized, lightweight HDFS compatible file system built over Cassandra.
-
Scherlok github.com
Zero-config data quality CLI. Profiles every table on first run, then auto-detects anomalies (volume drops, schema drift, freshness misses, distribution shifts) on subsequent runs. No YAML, no rules to write. Works with Postgres, BigQuery, Snowflake, and dbt.
-
Rawbbit github.com
Open-source self-hosted game analytics pipeline. HTTP event collector with NATS JetStream buffering, raw Parquet in object storage you own, and ClickHouse for queries via Metabase, SQL, or a read-only MCP server for AI agents. Designed for teams that want to own their raw game…
-
Dotflow github.com
A lightweight Python library for building execution pipelines with retry, parallel execution, cron scheduling, and async support.
-
koala-diff github.com
A high-performance Python library for comparing large datasets (CSV, Parquet) locally using Rust and Polars. It features zero-copy streaming to prevent OOM errors and generates interactive HTML data quality reports.
-
Bistro github.com
A light-weight engine for general-purpose data processing including both batch and stream analytics. It is based on a novel unique data model, which represents data via functions and processes data via columns operations as opposed to having only set operations in conventional…
-
VoltDB voltdb.com
An ACID-compliant RDBMS which uses a shared nothing architecture.
-
Vertica vertica.com
Distributed, MPP columnar database with extensive analytics SQL.
-
TimescaleDB timescale.com
Built as an extension on top of PostgreSQL, TimescaleDB is a time-series SQL database providing fast analytics, scalability, with automated data management on a proven storage engine.
- next page of items loading…
