Apache Spark
78 items from awesome-spark/awesome-spark ★1,902
-
ORYX github.com
Lambda architecture platform built on Apache Spark and Apache Kafka with specialization for real-time large scale machine learning.
-
Microsoft ML for Apache Spark github.com
A distributed ml library with support for LightGBM, Vowpal Wabbit, OpenCV, Deep Learning, Cognitive Services, and Model Deployment.
-
Spark NLP github.com
Natural language processing library built on top of Apache Spark ML.
-
Apache Iceberg github.com
Upserts, Deletes And Incremental Processing on Big Data..
-
Delta Lake github.com
Storage layer with ACID transactions.
-
BigDL github.com
Distributed Deep Learning library.
-
Apache Hudi github.com
Upserts, Deletes And Incremental Processing on Big Data..
-
Delight github.com
A free & cross platform monitoring tool (Spark UI / Spark History Server alternative).
-
Cromwell github.com
Workflow management system with Spark backend.
-
Hail github.com
Genetic analysis framework.
-
adam github.com
Set of tools designed to analyse genomics data.
-
H2O Sparkling Water github.com
H2O interoperability layer.
-
deequ github.com
Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
-
Koalas github.com
Pandas DataFrame API on top of Apache Spark.
-
spark-jobserver github.com
Simple Spark as a Service which supports objects sharing using so called named objects. JVM only.
-
Apache Sedona github.com
Cluster computing system for processing large-scale spatial data.
-
Apache Kyuubi github.com
A distributed multi-tenant JDBC server for large-scale data processing and analytics, built on top of Apache Spark.
-
.NET for Apache Spark github.com
.NET bindings.
-
Spark Cassandra Connector github.com
Cassandra support including data source and API and support for arbitrary queries.
-
spark-testing-base github.com
Collection of base test classes.
-
MLeap github.com
Execution engine and serialization format which supports deployment of o.a.s.ml models without dependency on SparkSession.
-
Optimus github.com
Data Cleansing and Exploration utilities with the goal of simplifying data cleaning.
-
neo4j-spark-connector github.com
Bolt protocol based, Neo4j Connector with RDD, DataFrame and GraphX / GraphFrames support.
-
sparkmagic github.com
Jupyter magics and kernels for working with remote Spark clusters, for interactively working with remote Spark clusters through Livy, in Jupyter notebooks.
-
GraphFrames github.com
Data frame based graph API.
-
sparklyr github.com
An alternative R backend, using dplyr.
-
Livy github.com
REST server with extensive language support (Python, R, Scala), ability to maintain interactive sessions and object sharing.
-
python-deequ github.com
Python API for Deequ.
-
Photon ML github.com
A machine learning library supporting classical Generalized Mixed Model and Generalized Additive Mixed Effect Model.
-
chispa github.com
PySpark test helpers with beautiful error messages.
- next page of items loading…
