Hi Sir, Thank you for giving the answer for removing duplicate records but at the end this answer is wrong. Removing duplicates means at the end It should keep one records of Amir shahzad in the unique list and The duplicate output file should Only contain one record of Amir shahzad. I we know IF Amir Shahzad has record comes twice that means one record is duplicate and one record is unique. In the practical scenario we will keep one record.
In this video, I learnt that after performing the Aggregation step in which we create a column of COUNT (*), and if we want to split the columns based on this COUNT column, we conditionally split it where we write the Count > 1 contents saved in SINK 1 and unique content(not repeating) in SINK 2.
Great video my friend! Perhaps a next idea - do you know of a way to loop through a REST API call to return all the pages of the call if it has a next value in the API?
Your videos are amazing. I learnt a lot from you. By mistake, you removed unique records too. It's wrong bro. If there are 6 records, 5 records are duplicate then one must be unique. But, in this video you are considering 6 records as duplicate. Can you please make another video and delete this video.
