PAPERS-05-007 · Paper — Apache Spark (RDDs)

The paper

The landmark paper on Apache Spark (RDDs) introduced the Resilient Distributed Dataset abstraction, establishing a new way to process big data in memory. It presented the core ideas that made Spark a breakthrough in distributed computing.

Before this paper

Before this work, MapReduce wrote intermediate results to disk between every step, which made iterative algorithms like machine learning painfully slow. The need to constantly read and write data limited performance and scalability. As a result, many advanced analytics could not run efficiently on Hadoop clusters. The community was looking for a faster, more interactive approach.

The key idea

The key insight was to keep data in memory across steps using Resilient Distributed Datasets (RDDs). Instead of replicating data for fault tolerance, Spark records the lineage of each RDD so lost partitions can be recomputed. This approach combines speed with reliability without the overhead of disk I/O.

How it works

An RDD is an immutable, partitioned collection built by applying transformations to existing RDDs. Each transformation records its operation in a lineage graph. When a job runs, Spark schedules tasks that operate on the in‑memory partitions. If a worker fails, Spark consults the lineage and recomputes the missing partition from its ancestors rather than fetching a replica. Because the data stays in RAM, subsequent steps avoid repeated disk writes. The system coordinates these steps across the cluster, ensuring both parallelism and fault tolerance.

See it — recompute from lineage

flowchart LR
  A["Data in memory<br/>(RDD)"] --> B["Transform"]
  B --> C["Transform"]
  C --> D["Lost partition?"]
  D --> E["Recompute from<br/>lineage"]

What they showed

The authors demonstrated that Spark ran iterative and interactive jobs up to orders of magnitude faster than Hadoop MapReduce while keeping fault tolerance. This performance gain was shown on real‑world machine‑learning workloads, highlighting the practical impact of in‑memory processing.

What it changed

The paper changed the field by making Spark the dominant big‑data processing engine and the platform for large‑scale analytics and machine learning. Its memory‑centric model replaced the slower disk‑based MapReduce paradigm. As a result, many organizations adopted Spark for data engineering and ML pipelines, reshaping the big‑data ecosystem. Next: Raft Consensus.

The one-line takeaway

Keep data in memory and recompute lost pieces from lineage to get speed and fault tolerance together.

Read it / meet it

Reading the original Spark RDD paper gives insight into the design choices that still power modern data platforms. Today you can see its legacy in every Spark‑based data engineering and machine‑learning pipeline across industry.