Tag: ETL
AWS Glue: Develop & Test PySpark ETL Jobs Locally Using Development Endpoints
AWS Glue jobs—especially Streaming ETL jobs—can incur cold-start spin-up times of several minutes per run. Setting up an AWS Glue…
DataStage Warning NGBR-00019: Input Data Set 0 Not Present in Design Time Information
This IBM InfoSphere DataStage warning cost me two full days of investigation. Here was the exact scenario: I needed to…
How to Replace Substrings in an IBM DataStage Parallel Transformer
Suppose you want to replace a multi-character substring inside an IBM DataStage Parallel Transformer—for instance, replacing “choc” with “mock” in…
Generate a Unique Sequence Counter in a DataStage Parallel Transformer
Need a deterministic, collision-free sequential row number across all parallel partitions in an IBM DataStage Parallel Transformer without adding a…
Understanding LUTCreateOp and LUTProcessOp in IBM DataStage
Every so often on DSXchange, DataStage veterans like Ray Wurlod, Craig Hulett, and ArndW share concise explanations of what the…
Aggregator Methods in DataStage: Hash Mode vs. Sort Mode (Courtesy: Ray Wurlod)
Choosing between Hash mode and Sort mode in the IBM DataStage Parallel Aggregator stage is one of the highest-leverage performance…