Stuff I Write About
JR
My Blog
A Detailed Guide to Key Generation in BigQuery
Migrating an enterprise data warehouse from traditional SMP or MPP databases (Oracle, Teradata, SQL Server, Netezza) to Google BigQuery inevitably…
Chatting With Dataproc Clusters Using an AI Agent
Managing fleet-scale Apache Spark and Hadoop environments on Google Cloud Dataproc routinely requires juggling gcloud dataproc clusters CLI commands, Cloud…
Implementing Your Logical Data Model in BigQuery: A Practical Guide
A clean Logical Data Model (LDM) captures your enterprise’s business entities, attributes, grain, and relationships independent of any database engine….
Build a Fraud Detection Model in Minutes with BigQuery ML
Traditionally, shipping a fraud-detection classifier meant exporting operational tables to object storage, spinning up a Python training cluster, wrestling with…
Parallel & Serverless CSV Ingestion to Cloud SQL Using Cloud Dataflow
Anyone who has tried bulk-loading hundreds of gigabytes of multi-file CSV exports from Google Cloud Storage (GCS) into Cloud SQL…
Spark Config Calculator
The Spark Configuration Calculator is an interactive Streamlit web application designed to take the guesswork out of sizing and tuning…
Using Spark on Dataproc & Apache Iceberg to Build an Open Lakehouse
Raw Parquet lakes on object storage solved cheap, decoupled scalability—but left data engineers stitching together fragile directory-rename hacks whenever workloads…
A Guide to RAID Multiple Local SSDs & Mount Them on Dataproc
Heavy Apache Spark shuffles, wide joins, and terabyte-scale external sorts live and die by local scratch disk IOPS and throughput….
Understanding Driver Pools in Dataproc
Let’s learn about driver pools in Dataproc. But before we delve into the details of driver pools, it’s important to…
Build Your Own Ask-Me-Anything Search Engine Using Vertex AI, LangChain & Streamlit
Artificial Intelligence and more specifically, its “Generative” heir has experienced a golden year thus far. Generative AI has laid the…
Hadoop — Understanding Splits, Blocks & Everything In Between
Understanding Hadoop is like trying to unravel a tangled ball of yarn while wearing oven mitts. I’ve had my fair…
Understanding CPU Oversubscription in Dataproc & Hadoop YARN
Let’s start by understanding 2 important properties in your Yarn configuration. For simple Hadoop workloads, it is best to leave…
Dataproc — Why Is My Cluster Not Scaling?
“Autoscaling” is a Dataproc API that automates the process of monitoring YARN memory utilisation and adding/removing capacity to achieve optimal…
GCP Cloud Logging: How to Enable Data Access Audit Logs for Selected GCS Buckets
Introduction Data Access Audit Logs are used to trace and monitor API calls that create, modify, or read user data…
Hilton Shillim — Expensive, Expansive, Exclusive!
The Hilton Shillim Estate Retreat & Spa is a property nestled quietly into the western ghats. Taking that turn from…