Programming
Software development: programming languages, coding, libraries and frameworks, software engineering and architecture, source control, developer tools, databases, and infrastructure.
-
Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark cs.stanford.edu
Structured Streaming is a new high-level streaming API, it is a declarative API based on automatically incrementalizing a static relational query.
-
SparkR amplab-extras.github.io
R frontend for Spark.
-
Big Data Analysis with Scala and Spark (Coursera) coursera.org
Scala oriented introductory course. Part of Functional Programming in Scala Specialization.
-
Free Dev License dev.splunk.com
Request a free Splunk Developer license.
-
Splunk Dev dev.splunk.com
Develop on Splunk.
-
Quick Intelligence quickintelligence.co.uk
A blog where a skilled team of Qlik developers lead by Steve Dark, providing tips on working in Qlik Sense and QlikView.
-
QlikView Cookbook qlikviewcookbook.com
A technical 'how-to' blog for QlikView developers, by Rob Wunderlich.
-
QlikView 11 for Developers amazon.com
By Miguel García and Barry Harmsen.
-
Large-Scale Intelligent Microservices arxiv.org
Microsoft paper that presents an Apache Spark-based micro-service orchestration framework that extends database operations to include web service primitives.
-
Apache Spark User List apache-spark-user-list.1001560.n3.nabble.com
and Apache Spark Developers List - Mailing lists dedicated to usage questions and development topics respectively.
-
Spark SQL: Relational Data Processing in Spark amplab.cs.berkeley.edu
Paper introducing relational underpinnings, code generation and Catalyst optimizer.
-
Zingg zingg.ai
Open source Master Data Management platform using machine learning for entity resolution at scale. Native to Databricks, Microsoft Fabric, Snowflake, AWS, and GCP. Golden records are maintained through a persistent Zingg ID across all systems and sources.
-
Xquik xquik.com
Real-time X (Twitter) data extraction platform with REST API (76 endpoints), 20 bulk extraction tools, account monitoring, HMAC-signed webhooks, and MCP server for AI agent integration.
-
Titan titan.thinkaurelius.com
A scalable graph database optimized for storing and querying graphs containing hundreds of billions of vertices and edges distributed across a multi-machine cluster.
-
SuprSend suprsend.com
Create automated workflows and logic using API's for your notification service. Add templates, batching, preferences, inapp inbox with workflows to trigger notifications directly from your data warehouse.
-
DBConvert Streams streams.dbconvert.com
Self-hosted database migration and change data capture (CDC) tool with built-in SQL IDE.
-
stratif.io stratif.io
Open-source, self-hosted, warehouse-native product analytics. Runs funnels, retention, and paths directly on DuckDB, Postgres, Snowflake, or ClickHouse.
-
SSDB ssdb.io
A high performance NoSQL database supporting many data structures, an alternative to Redis.
-
SQLMesh sqlmesh.readthedocs.io
An open-source data transformation framework for managing, testing, and deploying SQL and Python-based data pipelines with version control, environment isolation, and automatic dependency resolution.
-
Spark Playground sparkplayground.com
Write, run, and test PySpark code on Spark Playground's online compiler. Access real-world sample datasets & solve interview questions to enhance your PySpark skills for data engineering roles.
-
Spark MLlib spark.apache.org
Spark's scalable machine learning library consisting of common learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dimensionality reduction, as well as underlying optimization primitives.
-
Software Engineering Daily softwareengineeringdaily.com
Daily interviews about technical software topics, including data infrastructure.
-
SmoothieCharts smoothiecharts.org
A JavaScript Charting Library for Streaming Data.
-
Sling slingdata.io
CLI data integration tool specialized in moving data between databases, as well as storage systems.
-
SchemaCrawler schemacrawler.com
Open-source and free relational database schema discovery and comprehension tool. Documents and diagrams relational database schemas from your Java programs, build tools and the command line. Find database design issues with lint, and write scripts against the database…
-
RunSQL runsql.com
Free online SQL playground for MySQL, PostgreSQL, and SQL Server. Create database structures, run queries, and share results instantly.
-
Rivestack rivestack.io
Managed PostgreSQL with pgvector for AI workloads. HNSW indexing, sub-4ms latency, and built-in SQL editor with automatic embedding generation.
-
/r/dataengineering reddit.com
News, tips, and background on Data Engineering.
-
Presto prestodb.github.io
A distributed SQL query engine designed to query large data sets distributed over one or more heterogeneous data sources.
-
Prefect prefect.io
An orchestration and observability platform. With it, developers can rapidly build and scale resilient code, and triage disruptions effortlessly.
-
Percona XtraBackup percona.com
A free, open source, complete online backup solution for all versions of Percona Server, MySQL® and MariaDB®.
-
Percona Server for MongoDB percona.com
Percona Server for MongoDB® is a free, enhanced, fully compatible, open source, drop-in replacement for the MongoDB® Community Edition that includes enterprise-grade features and functionality.
-
ParquetKit parquetkit.com
Browser-based viewer, SQL workbench and converter for Parquet files powered by DuckDB-WASM. Fully client-side, no upload.
-
Nakadi nakadi.io
An open source event messaging platform that provides a REST API on top of Kafka-like queues.
-
LunaPad lunapad.dev
Open-source analytics notebook for reusable SQL workflows, interactive reports, and AI-assisted data exploration.
-
LizardFS lizardfs.com
Software Defined Storage is a distributed, parallel, scalable, fault-tolerant, Geo-Redundant and highly available file system.
-
ImageLayers imagelayers.io
Visualize Docker images and the layers that compose them.
-
Ilum ilum.cloud
A modular Data Lakehouse platform that simplifies the management and monitoring of Apache Spark clusters across Kubernetes and Hadoop environments.
-
Spark RDD API Examples homepage.cs.latrobe.edu.au
Examples by Zhen He.
-
Hadoop MapReduce hadoop.apache.org
A software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) - in-parallel on large clusters (thousands of nodes) - of commodity hardware in a reliable, fault-tolerant manner.
-
GreenPlum github.com
The Greenplum Database (GPDB) - An advanced, fully featured, open source data warehouse. It provides powerful and rapid analytics on petabyte scale data volumes.
-
Helium MCP github.com
Remote MCP server for real-time financial data, 3.2M+ news articles, ML options pricing, and news bias analysis. Free, no API key. MCP
-
BottledWater github.com
Change data capture from PostgreSQL into Kafka. Deprecated.
-
Apache Geode geode.apache.org
An open source, distributed, in-memory database for scale-out applications.
-
Fixzi fixzi.ai
JSON/XML validation and API contract monitoring tool for debugging and testing structured data.
-
Enrich.sh enrich.sh
Managed event ingestion service that converts JSON sent to a REST API into Hive-partitioned Parquet on Cloudflare R2, queryable from DuckDB, ClickHouse, BigQuery, Snowflake, and Python.
-
Wikipedia dumps.wikimedia.org
Wikipedia's complete copy of all wikis, in the form of Wikitext source and metadata embedded in XML. A number of raw database tables in SQL form are also available.
-
Riak docs.basho.com
A distributed database designed to deliver maximum data availability by distributing data across multiple servers.
-
dlt dlthub.com
A fast&simple pipeline building library for Python data devs, runs in notebooks, cloud functions, airflow, etc.
-
Twitter Realtime developer.twitter.com
The Streaming APIs give developers low latency access to Twitter's global stream of Tweet data.
- next page of items loading…
