Spark Interview Question | Scenario Based | Merge DataFrame in Spark | LearntoSpark 

being a newbie to spark I find it very helpful boss.keep it up brother.looking forward to see more such from you.

Now you can use unionByName() function as well. df3 = df.unionByName(df2, allowMissinColumns=True) df3.show()

Real and true looking forward to see more videos

Last approach was incredible. Did not know it was possible to subtract the columns to get the delta!!

Excelent video Azarudeen, you helped me alot! Thankssss

Thanks Azar for making such a nice scenario based question series with demo.

Really its nice help friend

Good videos. Thank you. One small info, in "Automated Approach" if number of columns difference between two data frame is more than one and not in alphabetical order then it won't work. We need to sort the columns while performing union operation like below. df_final=df_file1.select(sorted(df_file1.columns)).union(df_file2.select(sorted(df_file2.columns)))

u r doing great job and its helping a lot to the beginners. Thanks

When merging 5 different data format files how it will work ?? Your answer will be helpful

Very good explanation of each scenario .... Thanks a lot @Azarudeen Shahul... Keep it up

Hi Shaul, Superb content. Never seen such an clear and all possible approaches in RU-vid. Thanks a lot. Not only for the interview , to get out daily jobs done ,you're videos so helpful.

great work Azar. I used the automatic technique for a datawareshousing project.

very nice approach and clear explanation! Thank you very much.

The tutorial is very lucid and clear

Very clear and useful. Thank you very much

Thank you so much for these real time scenario videos brother Eagarly waiting for more such All the best

Thank you so much for the videos. They definitely increased my hope towards practical learning!!!

Good video ..please keep posted on new scenario based questions

Awesome Azharuddin, your videos are very helpful...Do you take any online coaching?

Superb bro 👌 👏

Great example and nice explaination

Awesome work man. Appreciated

Nice video.. informative.. ❤❤

Excellent. Thanks for sharing. Can u make a video on reading data from multiple parquet files of different schema using schema evolution.

Great pyspark tutorial thanks

Can you also make some videos on spark using scala? All your videos are brilliant

HI Azarudeen its Awanish your video really helpful,,, actually i have installed Spark but while i am checking on command prompt by entering pyspark its saying path is not specified , even though i have made many correctness and checked even environment variables as well many times

Hi Azarudeen. Thank you so much for this video. I have implemented the same question in spark scala but I am facing problem in implementing the automated approach in spark scala. Could you please help me on this and provide me solution for the same.

Boss , you are beauty!!’

Very nice explanation of the concepts. How we can achieve this in scala. Also it will be great if you also explain some scenarios using Scala . Thank you

How outer join worked? We have same columns in both the DF, which columns it will take?

For the same scenario, I have used motonically I'd column for two then I have done left join. Is that approach was correct?

How can I achive same in scala? I tried following code but not working.consider a and b as two dataframe Val diffcol=a.columns.diff(b.columns) for(i

Hi..your videos are really helpful... could you please post a video on spark incremental data load and merge that data with scd2 type (using SCALA)...

thanks a lot bro,

so can you help me to fix it ? can you check i am ready to share my screen ? dear please helpp i have learnt theory part of Hadoop and spark but not feeling confident because of no good hands on because of no environment

Awesome Bro !.. If you can, please do the video on the same scenario by using Scala.

Can you help in merge two dataframes with date column and big int column i am getting error like failed to merge

How can we get the code for all the scenarios in this playlist?

nice !

I'm trying string (json style) -> parquet for merging different columns dataframe

We can use unionByName in scala

How to compare two data frames, with matched records and unnmatched record values?

Hi bro how to achieve the same using scala

Can we do this using unionByName

Hi Sir, in for loop we see df2=df2.withColumn(i,lit("null")) here we are able to update the dataframes, but how is it possible if dataframes are immutable.

Thanks Azar for making real-time scenario based videos.. how automated process works when both data frames have different column names ?

@Azarudeen Shah - In the example the missing column is at the last for one of the dataframe. So with_column automatically adds at the end. What if the column is missing in middle of the table structure ? Thank you!!

bro pls help me to install spark share me doc of steps i have windows 10

Can you please share the scala code for automated approach

Just add a scenario if we do not have columns in same order in both dataframes after loop? New columns arrive or some columns may disappear over time but the merge/union should keep happening daily. - we need to select columns in right order before doing union we use foldLeft instead of loop (more functional programming way)

Thank you , but in automated approach , updating df2 in for loop it won't work in java

how to get your mail id ?

Your methods will not work if both tables have one an extra column. For example TableA: name, age, salary TableB: name,age,gender

From where input1.csv is fetched, do u have uploaded any CSV file there.?

Awesome content!! please help me if we save the output => df1.union(df2).show() and save it to new dataframe as df, and apply df.show(), it didn't work, why?

we can achieve this by using UnionByName: union_df = df1.unionByName(df2, allowMissingColumns = True)