Tag: ETL
August 31, 2024
/ Technology
Parallel & Serverless CSV Ingestion to Cloud SQL Using Cloud Dataflow
Anyone who has tried bulk-loading hundreds of gigabytes of multi-file CSV exports from Google Cloud Storage (GCS) into Cloud SQL…
August 22, 2023
/ Technology
Understanding Driver Pools in Dataproc
Let’s learn about driver pools in Dataproc. But before we delve into the details of driver pools, it’s important to…
April 21, 2023
/ Technology
Hadoop — Understanding Splits, Blocks & Everything In Between
Understanding Hadoop is like trying to unravel a tangled ball of yarn while wearing oven mitts. I’ve had my fair…
October 7, 2020
/ Technology
The All-New AWS Glue Studio: Visual Drag-and-Drop Spark ETL
Until recently, AWS Glue offered a static visual graph of generated scripts without true drag-and-drop authoring—making the launch of AWS…
October 6, 2020
/ Technology
Real-Time Data Streaming into Kinesis & Ingestion into Postgres Using AWS Glue — Part 2 (Glue Catalog Setup)
Before we start building Glue jobs, we need to understand that one of the unique features of Glue is its…
October 5, 2020
/ Technology
Real-Time Data Streaming into Kinesis & Ingestion into Postgres Using AWS Glue — Part 1 (Setup)
Earlier this year, Amazon announced serverless streaming ETL support in AWS Glue. Under the hood, AWS Glue runs managed Apache…