Big Data
1,992 items in 9 lists
-
Latent Space latent.space
Technical deep dives on AI engineering, from model training to deployment.
-
Kontena kontena.io
Application Containers for Masses.
-
Kedro kedro.readthedocs.io
A framework that makes it easy to build robust and scalable data pipelines by providing uniform project templates, data abstraction, configuration and pipeline assembly.
-
ImageLayers imagelayers.io
Visualize Docker images and the layers that compose them.
-
Ilum ilum.cloud
A modular Data Lakehouse platform that simplifies the management and monitoring of Apache Spark clusters across Kubernetes and Hadoop environments.
-
Spark RDD API Examples homepage.cs.latrobe.edu.au
Examples by Zhen He.
-
HDFS hadoop.apache.org
A distributed file system designed to run on commodity hardware.
-
Hadoop MapReduce hadoop.apache.org
A software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) - in-parallel on large clusters (thousands of nodes) - of commodity hardware in a reliable, fault-tolerant manner.
-
SwimOS github.com
A framework for building real-time streaming data processing applications that supports a wide range of ingestion sources.
-
CronQ github.com
An application cron-like system. Used w/Luigi. Deprecated.
-
GreenPlum github.com
The Greenplum Database (GPDB) - An advanced, fully featured, open source data warehouse. It provides powerful and rapid analytics on petabyte scale data volumes.
-
Helium MCP github.com
Remote MCP server for real-time financial data, 3.2M+ news articles, ML options pricing, and news bias analysis. Free, no API key. MCP
-
BottledWater github.com
Change data capture from PostgreSQL into Kafka. Deprecated.
-
Giraph giraph.apache.org
An iterative graph processing system built for high scalability.
-
dbt getdbt.com
A command line tool that enables data analysts and engineers to transform data in their warehouses more effectively.
-
Apache Geode geode.apache.org
An open source, distributed, in-memory database for scale-out applications.
-
FlightPath Data flightpathdata.com
FlightPath is a gateway to a data lake's bronze layer, protecting it from invalid external data file feeds as a trusted publisher.
-
Fixzi fixzi.ai
JSON/XML validation and API contract monitoring tool for debugging and testing structured data.
-
Eventum eventum.run
Data generation platform for producing synthetic event streams with complex correlations.
-
Enrich.sh enrich.sh
Managed event ingestion service that converts JSON sent to a REST API into Hive-partitioned Parquet on Cloudflare R2, queryable from DuckDB, ClickHouse, BigQuery, Snowflake, and Python.
-
Cassandra Calculator ecyrd.com
This simple form allows you to try out different values for your Apache Cassandra cluster and see what the impact is for your application.
-
Wikipedia dumps.wikimedia.org
Wikipedia's complete copy of all wikis, in the form of Wikitext source and metadata embedded in XML. A number of raw database tables in SQL form are also available.
-
Riak docs.basho.com
A distributed database designed to deliver maximum data availability by distributing data across multiple servers.
-
dlt dlthub.com
A fast&simple pipeline building library for Python data devs, runs in notebooks, cloud functions, airflow, etc.
-
Twitter Realtime developer.twitter.com
The Streaming APIs give developers low latency access to Twitter's global stream of Tweet data.
-
The Data Stack Show datastackshow.com
A show where they talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.
-
DataScreenIQ datascreeniq.com
Real-time data quality firewall for pipelines and APIs. Screens rows in milliseconds for schema drift, null spikes, type mismatches, and data anomalies with PASS / WARN / BLOCK decisions.
-
Data Mechanics datamechanics.co
A cloud-based platform deployed on Kubernetes making Apache Spark more developer-friendly and cost-effective.
-
DataKitchen datakitchen.io
Open Source Data Observability for end-to-end Data Journey Observability, data profiling, anomaly detection, and auto-created data quality validation tests.
-
Dataform dataform.co
An open-source framework and web based IDE to manage datasets and their dependencies. SQLX extends your existing SQL warehouse dialect to add features that support dependency management, testing, documentation and more.
-
Data Council datacouncil.ai
The first technical conference that bridges the gap between data scientists, data engineers and data analysts.
-
CsvPath Framework csvpath.org
A delimited data preboarding framework that fills the gap between MFT and the data lake.
-
Crustdata crustdata.com
A real-time B2B data API for company and people intelligence, providing firmographics, headcount signals, job listings, web traffic, and funding events via REST API and webhooks for data enrichment pipelines.
-
Couchbase couchbase.com
The highest performing NoSQL distributed database.
-
Common Crawl commoncrawl.org
Open source repository of web crawl data.
-
Spring Cloud Dataflow cloud.spring.io
Streaming and tasks execution between Spring Boot apps.
-
ClickHouse clickhouse.tech
Distributed columnar DBMS for OLAP. SQL.
-
chDB chdb.io
Embedded ClickHouse — full ClickHouse SQL dialect, ~80 data formats, and 12+ source connectors (S3, Postgres, MongoDB, Kafka, Iceberg) in core. Python, Go, Rust, Node, Bun, Zig, and Ruby bindings.
-
CEPH ceph.com
A unified, distributed storage system designed for excellent performance, reliability, and scalability.
-
Airbyte airbyte.com
open-source data movement platform for ELT pipelines and connector-based replication.
-
Bonobo bonobo-project.org
A data-processing toolkit for python 3.5+.
-
Bonnard bonnard.dev
Governed, multi-tenant MCP access to your customers' data. Turn your warehouse, dbt, or semantic layer into a secure, per-customer MCP for AI agents.
-
Actian Ingres actian.com
commercially supported, open-source SQL relational database management system.
-
Actian Versant actian.com
commercial object-oriented database management systems .
-
Actian SQL for Hadoop actian.com
high performance interactive SQL access to all Hadoop data.
-
Actian Vector actian.com
column-oriented analytic database.
-
Artie artie.com
Real-time data ingestion tool leveraging change data capture.
-
Arpe.io arpe.io
High-speed CLI tools for database export, import, replication and migration with parallel streaming to CSV, Parquet, JSON and cloud storage, supporting PostgreSQL, MySQL, Oracle, SQL Server and 80+ sources.
-
Best Data Science Books appliedaicourse.com
This blog offers a curated list of top data science books, categorized by topics and learning stages, to aid readers in building foundational knowledge and staying updated with industry trends.
-
LatAm Synth apify.com
Synthetic financial savings behavior generator for Latin America: users, savings goals, and transactions calibrated on 506K real records (2015–2024). Reproducible by seed, 100% synthetic.
- next page of items loading…
