dogear

enter for all results · esc to close

Data Engineering

313 items from igorbarinov/awesome-data-engineering ★9,159

all kinds
  1. 0
    Protocol Buffers (⭐72k) github.com
    tool category #integration link ↗

    A language-neutral and platform-neutral serialization mechanism that is designed to be highly efficient and extensible. It supports rich data types and is widely used in distributed systems, such as gRPC and Apache Kafka.

  2. 0
    tidb github.com
    tool category #go link ↗

    TiDB is a distributed SQL database. Inspired by the design of Google F1.

  3. 0
    dash github.com
    tool category #python link ↗

    Flask, JS, and CSS boilerplate for interactive, web-based visualization apps in Python.

  4. 0
    prometheus github.com
    tool category #go link ↗

    Monitoring system and time series database.

  5. 0
    Metabase github.com
    tool category #bigdata link ↗

    The easy, open source way for everyone in your company to ask questions and learn from data.

  6. 0
    Luigi github.com
    tool category #bigdata link ↗

    A Python module that helps you build complex pipelines of batch jobs.

  7. 0
    Dagster (⭐16k) github.com
    tool category #integration link ↗

    Data orchestrator with a declarative, asset-based programming model for building and observing data pipelines.

  8. 0
    zvec github.com
    tool category #python link ↗

    A lightweight, lightning-fast, in-process vector database. [Apache2] website

  9. 0
    Superset github.com
    tool category #machine-learning link ↗

    A modern, enterprise-ready business intelligence web application.

  10. 0
    Apache Airflow (⭐47k) github.com
    tool category #integration link ↗

    Platform for programmatically creating, scheduling, and monitoring workflows, ideal for managing complex data pipelines.

  11. 0
    Kestra (⭐28k) github.com
    tool category #integration link ↗

    Event-driven declarative orchestrator with workflows defined in YAML and hundreds of plugins for data and infrastructure automation.

  12. 0
    cAdvisor github.com
    tool category #docker link ↗

    Analyzes resource usage and performance characteristics of running containers.

  13. 0
    JuiceFS github.com
    tool category #go link ↗

    Distributed POSIX file system built on top of Redis and AWS S3.

  14. 0
    pathwaycom/pathway github.com
    tool category #rust link ↗

    Performant open-source Python ETL framework with Rust runtime, supporting 300+ data sources.

  15. 0
    InfluxDB github.com
    tool category #data-engineering link ↗

    Scalable datastore for metrics, events, and real-time analytics.

  16. 0
    Multiwoven github.com
    tool category #ruby link ↗

    The open-source reverse ETL, data activation platform for modern data teams. (using Rails 7.1.1) - 🌍

  17. 0
    rqlite github.com
    tool category #go link ↗

    The lightweight, distributed, relational database built on SQLite.

  18. 0
    Nomad github.com
    tool category #docker link ↗

    Easily deploy applications at any scale. A Distributed, Highly Available, Datacenter-Aware Scheduler.

  19. 0
    cayley github.com
    tool category #bigdata link ↗

    An open-source graph database. Google.

  20. 0
    Heka github.com
    tool category #bigdata link ↗

    Data Acquisition and Processing Made Easy. Deprecated.

  21. 0
    Manticore Search github.com
    tool category #data-engineering link ↗

    An open-source search database for full-text, vector, and hybrid search with real-time indexing and SQL.

  22. 0
    DataComPy github.com
    tool category #python-data-science link ↗

    A library to compare Pandas, Polars, and Spark data frames. It provides stats and lets users adjust for match accuracy.

  23. 0
    cocoindex github.com
    tool category #rust link ↗

    ETL framework to build fresh context for AI agents, with incremental processing

  24. 0
    Hamilton github.com
    tool category #python-data-science link ↗

    A microframework for dataframe generation that applies Directed Acyclic Graphs specified by a flow of lazily evaluated Python functions.

  25. 0
    Kryo github.com
    tool category #data-engineering link ↗

    A fast and efficient object graph serialization framework for Java.

  26. 0
    RudderStack github.com
    tool category #bigdata link ↗

    A warehouse-first Customer Data Platform that enables you to collect data from every application, website and SaaS platform, and then activate it in your warehouse and business tools.

  27. 0
    aws-data-wrangler :fire::fire::fire::fire::fire: github.com
    tool category #aws link ↗

    Utility belt to handle data on AWS.

  28. 0
    ingestr github.com
    tool category #bigdata link ↗

    CLI tool to copy data between databases with a single command. Supports 50+ sources including PostgreSQL, MySQL, MongoDB, Salesforce, Shopify to any data warehouse.

  29. 0
    ScyllaDB github.com
    tool category #data-engineering link ↗

    NoSQL data store using the seastar framework, compatible with Apache Cassandra.

  30. 0
    Tarantool github.com
    tool category #bigdata link ↗

    An in-memory database and application server.

  31. next page of items loading…