Stuff I Write About
JR
My Blog
Real-Time Data Streaming into Kinesis & Ingestion into Postgres Using AWS Glue — Part 2 (Glue Catalog Setup)
Before we start building Glue jobs, we need to understand that one of the unique features of Glue is its…
Real-Time Data Streaming into Kinesis & Ingestion into Postgres Using AWS Glue — Part 1 (Setup)
Earlier this year, Amazon announced serverless streaming ETL support in AWS Glue. Under the hood, AWS Glue runs managed Apache…
AWS Glue: Develop & Test PySpark ETL Jobs Locally Using Development Endpoints
AWS Glue jobs—especially Streaming ETL jobs—can incur cold-start spin-up times of several minutes per run. Setting up an AWS Glue…
The Juggernaut Resolution: Why You Should Start New Year Goals on December 1
One thing almost everyone has struggled with at some point is sticking to New Year’s resolutions. My resolutions have evolved…
DataStage Warning NGBR-00019: Input Data Set 0 Not Present in Design Time Information
This IBM InfoSphere DataStage warning cost me two full days of investigation. Here was the exact scenario: I needed to…
How to Replace Substrings in an IBM DataStage Parallel Transformer
Suppose you want to replace a multi-character substring inside an IBM DataStage Parallel Transformer—for instance, replacing “choc” with “mock” in…
Generate a Unique Sequence Counter in a DataStage Parallel Transformer
Need a deterministic, collision-free sequential row number across all parallel partitions in an IBM DataStage Parallel Transformer without adding a…
Understanding LUTCreateOp and LUTProcessOp in IBM DataStage
Every so often on DSXchange, DataStage veterans like Ray Wurlod, Craig Hulett, and ArndW share concise explanations of what the…
Aggregator Methods in DataStage: Hash Mode vs. Sort Mode (Courtesy: Ray Wurlod)
Choosing between Hash mode and Sort mode in the IBM DataStage Parallel Aggregator stage is one of the highest-leverage performance…
What Exactly Are Hash Files in DataStage? (Courtesy: Ken Bland)
Editor’s Note: Archived here for quick reference from Ken Bland’s classic DSXchange newsletter tech tip (DSXchange). Full credit to Ken…
DataStage Parallel Transformer Date Derivations Cheat Sheet
When your pipeline cannot join against a calendar/time dimension table, these battle-tested IBM DataStage Parallel Transformer expressions compute common month-end,…
IBM DataStage orchadmin Command Reference Guide
In IBM InfoSphere DataStage, Parallel Datasets (.ds) store a lightweight descriptor file at the target path while distributing raw partition…
Notes on DataStage Bulk Load Tuning with the Oracle Connector
Quick production checklist when configuring high-volume direct-path bulk loads into Oracle using the IBM InfoSphere DataStage Oracle Connector stage:
ETL — Code, Build, or Both?
Data has been growing at an exponential pace—not just in raw row volume, but across entirely new dimensions fueled by…
Business Alert & Exception Data Management Within an ETL Ecosystem
Introduction Data transparency and traceability are foundational to any enterprise ETL ecosystem. Both business data stewards and platform support engineers…