This is the best interview till now. He worked on security , CDC concepts etc. Thanks sumit sir for conducting these interviews. Really helpful who is preparing for the interviews.
I am getting prepared for the interview next week. As you said failed interview better then not attending interview and these mock interviews giving me the confidence to face the interview. Great initiative, learning and help. Thanks!
These videos have been very helpful. I have gone over many and this would be beneficial for a long time. Great work, initiative and thanks for the efforts.
This is one of the best interviews and got good insights Sumit sir.Learnt the new concept of thin executor and fat executor and his optimisation techniques are also unique like join re order Adq etc…
I would like to know the answer for the question, "why in spark why we will get duplicate of the column on which join is performed, whereas in sql we do not get that duplicate ?" From my limited knowledge, I know that in both spark and sql joins we will get the duplicate columns when join is performed on two tables. I would really appreciate it if you could please let me know the correct answer.
The reason behind this lies in the fundamental differences between Spark and traditional SQL databases. Spark performs parallel processing to handle partitioned data distributed across different nodes in the cluster. It is inevitable that there would be duplicates while performing joins on such datasets in this context. In contrast, traditional SQL databases usually operates on a single machine or a tightly coupled cluster and ensures uniqueness by preventing the duplicated from being generated in the final results
Very nice interview sir can we have an interview for someone who is having more than 5 years of experience and is looking to join as an Architect level or Team Lead, Manager in Data Engineering world.
