Apache Spark
78 items from awesome-spark/awesome-spark ★1,902
-
spark-daria github.com
A Scala library with essential Spark functions and extensions to make you more productive.
-
sequenceiq/docker-spark github.com
Yarn images from SequenceIQ.
-
Apache Toree github.com
IPython protocol based middleware for interactive applications.
-
Mongo-Spark github.com
Official MongoDB connector.
-
quinn github.com
A native PySpark implementation of spark-daria.
-
Archives Unleashed Toolkit github.com
Open-source toolkit for analyzing web archives.
-
Flintrock github.com
A command-line tool for launching Spark clusters on EC2.
-
Spark XML github.com
XML parser and writer.
-
Kotlin for Apache Spark github.com
Kotlin API bindings and extensions.
-
spark-fast-tests github.com
A lightweight and fast testing framework.
-
sparkle github.com
Haskell on Apache Spark.
-
spark-connect-go github.com
Golang bindings.
-
Joblib Apache Spark Backend github.com
joblib backend for running tasks on Spark clusters.
-
Crossdata github.com
Data integration platform with extended DataSource API and multi-user environment.
-
spark-connect-rs github.com
Rust bindings.
-
JPMML-Spark github.com
PMML transformer library for Spark ML.
-
itachi github.com
A library that brings useful functions from modern database management systems to Apache Spark.
-
sparkly github.com
Helpers & syntactic sugar for PySpark.
-
Apache Beam beam.apache.org
A unified programming model that implements both batch and streaming data processing jobs that run on many execution engines.
-
Spark in Action manning.com
New book in the Manning's "in action" family with +400 pages. Starts gently, step-by-step and covers large number of topics. Free excerpt on how to setup Eclipse for Spark application development and how to bootstrap a new application using the provided Maven Archetype. You can…
-
Apache Zeppelin zeppelin.incubator.apache.org
Web-based notebook that enables interactive data analytics with plugable backends, integrated plotting, and extensive Spark support out-of-the-box.
-
spark-connect-csharp github.com
C# bindings.
-
Apache SystemML systemml.apache.org
Declarative machine learning framework on top of Spark.
-
Advanced Analytics with Spark shop.oreilly.com
Useful collection of Spark processing patterns. Accompanying GitHub repository: sryza/aas.
-
PredictionIO prediction.io
Machine Learning server for developers and data scientists to build and deploy predictive applications in a fraction of the time.
-
Polynote polynote.org
Polynote: an IDE-inspired polyglot notebook. It supports mixing multiple languages in one notebook, and sharing data between them seamlessly. It encourages reproducible notebooks with its immutable data model. Originating from Netflix.
-
Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing people.csail.mit.edu
Paper introducing a core distributed memory abstraction.
-
Learning Spark, 2nd Edition oreilly.com
Introduction to Spark API with Spark 3.0 covered. Good source of knowledge about basic concepts.
-
MLflow mlflow.org
Machine learning orchestration platform.
-
ModelDB mitdbg.github.io
A system to manage machine learning models for spark.ml and scikit-learn .
-
Mahout Spark Bindings mahout.apache.org
[status unknown] - linear algebra DSL and optimizer with R-like syntax.
-
KeystoneML keystone-ml.org
Type safe machine learning pipelines with RDDs.
-
Mastering Apache Spark jaceklaskowski.gitbooks.io
Interesting compilation of notes by Jacek Laskowski. Focused on different aspects of Spark internals.
-
datamechanics/spark hub.docker.com
An easy to setup Docker image for Apache Spark from Data Mechanics.
-
apache/spark hub.docker.com
Apache Spark Official Docker images.
-
Spark with Scala Gitter channel gitter.im
"A place to discuss and ask questions about using Scala for Spark programming" started by @deanwampler.
-
jupyter/docker-stacks/pyspark-notebook github.com
PySpark with Jupyter Notebook and Mesos client.
-
Apache DataFu github.com
A library of general purpose functions and UDF's.
-
Data Science and Engineering with Apache Spark (edX XSeries) edx.org
Series of five courses (Introduction to Apache Spark, Distributed Machine Learning with Apache Spark, Big Data Analysis with Apache Spark, Advanced Apache Spark for Data Science and Data Engineering, Advanced Distributed Machine Learning with Apache Spark) covering different…
-
lakeFS docs.lakefs.io
Integration with the lakeFS atomic versioned storage layer.
-
Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark cs.stanford.edu
Structured Streaming is a new high-level streaming API, it is a declarative API based on automatically incrementalizing a static relational query.
-
Big Data Analysis with Scala and Spark (Coursera) coursera.org
Scala oriented introductory course. Part of Functional Programming in Scala Specialization.
-
Apache Bahir bahir.apache.org
Collection of the streaming connectors excluded from Spark 2.0 (Akka, MQTT, Twitter. ZeroMQ).
-
Large-Scale Intelligent Microservices arxiv.org
Microsoft paper that presents an Apache Spark-based micro-service orchestration framework that extends database operations to include web service primitives.
-
Apache Spark User List apache-spark-user-list.1001560.n3.nabble.com
and Apache Spark Developers List - Mailing lists dedicated to usage questions and development topics respectively.
-
Spark SQL: Relational Data Processing in Spark amplab.cs.berkeley.edu
Paper introducing relational underpinnings, code generation and Catalyst optimizer.
-
AMP Camp ampcamp.berkeley.edu
Periodical training event organized by the UC Berkeley AMPLab. A source of useful exercise and recorded workshops covering different tools from the Berkeley Data Analytics Stack.
-
almond almond.sh
A scala kernel for Jupyter.
