A Guide to RAID Multiple Local SSDs & Mount Them on Dataproc

Heavy Apache Spark shuffles, wide joins, and terabyte-scale external sorts live and die by local scratch disk IOPS and throughput. Attaching Local SSDs (375 GB per partition on Compute Engine) to your Google Cloud Dataproc workers eliminates network-attached Persistent Disk bottlenecks—but out of the box, Dataproc mounts each Local SSD on a separate directory (/mnt/1, /mnt/2, …).

When a single skewed Spark task spills an oversized partition exceeding 375 GB, individual 375 GB mounts can still trigger No space left on device errors even when sibling SSDs sit half-empty! Striping all attached Local SSDs into a single RAID 0 volume using mdadm pools capacity into one contiguous high-throughput filesystem and aggregates read/write bandwidth across every drive.

Why RAID 0 Makes Sense for Dataproc Ephemeral Scratch

• Pooled Spill Headroom: Four 375 GB Local SSDs become a single 1.5 TB volume—preventing single-partition Spark skew spills from exhausting an individual 375 GB mount.
• Linear IOPS & Bandwidth Scaling: Striping blocks across 4–8 NVMe/SCSI Local SSDs multiplies sequential shuffle write/read throughput.
• Ephemeral by Design: Durable source/sink data lives in Google Cloud Storage (GCS), while worker Local SSDs hold transient Spark shuffle files (SPARK_LOCAL_DIRS) and ephemeral HDFS scratch—making RAID 0’s zero-redundancy speed trade-off ideal.

Dataproc Initialization Action Script (raid-local-ssd.sh)

Using a Dataproc Initialization Action (github.com/datasherlock/initialization-actions), we check the node’s dataproc-role metadata so we run only on worker VMs, assemble the unmounted Local SSD block devices into /dev/md0 with mdadm --level=0, format with ext4, mount at /mnt/raiddisk1, and point both HDFS DataNode storage and Spark’s SPARK_LOCAL_DIRS to the striped array:

#!/bin/bash
set -euxo pipefail

ROLE=$(/usr/share/google/get_metadata_value attributes/dataproc-role)

if [[ "${ROLE}" != "Master" ]]; then
  # 1. Install mdadm RAID management utility
  sudo apt-get update && sudo apt-get install -y mdadm

  # 2. Stripe 4 attached Local SSDs into a single RAID 0 array (/dev/md0)
  # (Adjust device names—e.g. /dev/nvme0n* for NVMe or /dev/sd[b-e] for SCSI)
  sudo mdadm --create --verbose /dev/md0 \
    --level=0 \
    --raid-devices=4 \
    /dev/sdb /dev/sdc /dev/sdd /dev/sde

  # 3. Format with ext4 (disable lazy_itable_init delays or reserve 0% root blocks for scratch)
  sudo mkfs.ext4 -m 0 -F /dev/md0

  # 4. Mount the striped array and persist in /etc/fstab
  sudo mkdir -p /mnt/raiddisk1
  sudo mount -o defaults,noatime /dev/md0 /mnt/raiddisk1
  echo "/dev/md0  /mnt/raiddisk1  ext4  defaults,noatime,nofail  0 0" | sudo tee -a /etc/fstab

  # 5. Grant ownership & scratch permissions for YARN, HDFS, and Spark
  sudo mkdir -p /mnt/raiddisk1/hdfs /mnt/raiddisk1/spark
  sudo chown -R hdfs:hadoop /mnt/raiddisk1/hdfs
  sudo chmod 1777 /mnt/raiddisk1 /mnt/raiddisk1/spark

  # 6. Point HDFS DataNode and Spark local shuffle directories to RAID 0
  sudo bdconfig set_property \
    --configuration_file /etc/hadoop/conf/hdfs-site.xml \
    --name dfs.datanode.data.dir \
    --value "/mnt/raiddisk1/hdfs"

  echo 'export SPARK_LOCAL_DIRS=/mnt/raiddisk1/spark' | sudo tee -a /etc/spark/conf/spark-env.sh

  sudo systemctl restart hadoop-hdfs-datanode.service
fi

Provisioning a Dataproc Cluster with the Initialization Action

Upload raid-local-ssd.sh to your staging GCS bucket and pass --num-worker-local-ssds=4 (with --worker-local-ssd-interface=NVME on N2/N2D/C3 families; if Dataproc’s built-in auto-mount claims the drives first, pass custom unmounted GCE disks or unmount default /mnt/1..4 before running mdadm --create):

gcloud dataproc clusters create high-iops-spark-cluster \
  --region=us-central1 \
  --master-machine-type=n2-standard-8 \
  --worker-machine-type=n2-standard-16 \
  --num-workers=4 \
  --num-worker-local-ssds=4 \
  --initialization-actions=gs://my-dataproc-scripts/disk/raid-local-ssd.sh

Combine striped Local SSD shuffle storage with right-sized executors (Spark Config Calculator) and YARN CPU scheduling tuning to squeeze maximum throughput out of every Dataproc worker!

Originally published in the Google Cloud Community on Medium: A Guide to RAID Multiple Local SSDs & Mount It to Dataproc. Initialization script repo: github.com/datasherlock/initialization-actions.