Tag: dataproc
Chatting With Dataproc Clusters Using an AI Agent
Managing fleet-scale Apache Spark and Hadoop environments on Google Cloud Dataproc routinely requires juggling gcloud dataproc clusters CLI commands, Cloud…
Spark Config Calculator
The Spark Configuration Calculator is an interactive Streamlit web application designed to take the guesswork out of sizing and tuning…
Using Spark on Dataproc & Apache Iceberg to Build an Open Lakehouse
Raw Parquet lakes on object storage solved cheap, decoupled scalability—but left data engineers stitching together fragile directory-rename hacks whenever workloads…
A Guide to RAID Multiple Local SSDs & Mount Them on Dataproc
Heavy Apache Spark shuffles, wide joins, and terabyte-scale external sorts live and die by local scratch disk IOPS and throughput….
Understanding Driver Pools in Dataproc
Let’s learn about driver pools in Dataproc. But before we delve into the details of driver pools, it’s important to…
Hadoop — Understanding Splits, Blocks & Everything In Between
Understanding Hadoop is like trying to unravel a tangled ball of yarn while wearing oven mitts. I’ve had my fair…