My Blog
A BACKUP PLAN
I have a web server where I manage 4 websites that we developed over the past 5 years. While performing…
Presentation Pro Tip – 1
“If you just saw the stats and numbers, you’d think that I love losing at chess!” Hence the reason why…
Spark Architecture – Notes & FAQ
I’ve spent the past couple of weeks trying to master the Spark Architecture and this post is a running summary of all my notes and questions gathered from across the internet & Stackoverflow. If you feel something is incorrect, I’ll be happy to discuss. Hope you find it useful!
Quiz Quotient
To View The App, Please switch to “Desktop Mode” on your mobile browser [advanced_iframe use_shortcode_attributes_only=”true” src=”https://jeromerajan.com/custom/qqjs.html” width=”1000″ height=”550″ ] Quiz…
What, Why, How – Continuous Loading To Snowflake Using Snowpipes (AWS)
WHAT Loading into Snowflake can be done in multiple ways – Bulk loading from Snowflake stages (internal and external) Using…
What, How, Why : Storage Integration Object in Snowflake
WHAT From a security POV, Storage Integration is a very useful feature provided by Snowflake. At it’s core, Storage Integration…
The All New AWS Glue Studio
Up until now, AWS provided a visual representation of your code but never really allowed you to build using a…
Real Time Data Streaming Into Kinesis & Ingestion Into Postgres Using AWS Glue – Part 2 (Configure Glue Catalog Tables)
Before we start building Glue jobs, we need to understand that one of the unique features of Glue is its…
Real Time Data Streaming Into Kinesis & Ingestion Into Postgres Using AWS Glue – Part 1 (Setup)
This April, Amazon announced support for serverless streaming ETL using AWS Glue. For the uninformed – AWS Glue is built…
AWS – Develop ETL jobs using AWS Glue Endpoints
AWS Glue scripts can have start up times that could be as long as 12 minutes especially if you are…
Juggernaut Resolution
One of the things I have always struggled with, and I’m sure you’ll relate to this, is sticking to New…
NGBR-00019 – Input Dataset 0 not present in design time information
NGBR-00019 : Input data set 0 not present in design time information. Extended field properties for this field will not…
Replace Strings in DataStage
Trying to find a way to replace strings. e.g. In the string “chocolate”, I want to replace “choc” with “mock”….
Generate Unique Counter in a Parallel Transformer Without Surrogate Key generator
Old but handy trick. @INROWNUM * @NUMPARTITIONS + @PARTITIONNUM Update – To start the series from 1, use (@INROWNUM –…