Hi Manish I saw your channel recently and I found it very insightful. You are explaining the spark core concepts nicely. Keep continue ❤ You have that caliber to grow on RU-vid.
Hi Sir, I have just started learning spark, I found your videos really helpful to clear fundamentals, However u r mentioning u have shown few things in ur practical session, could you please send the link to practical sessions.
Could you please make a video on Spark Web UI. I see you've already explained the UI partly in the stages, jobs and tasks video but a dedicated and detailed video would be very useful. Thank you!
Hi Manish, Can you pls make a video on local setup to practice PySpark and Python? If already made, can you pls share the link ? Much Appreciated. Thanks :)
traditional drivers and executors aren't available in local environment because a single JVM is present, and processes are executed in parallel across these threads.
Hi, How to handle situation in broadcast hash join where we have OOM error in executor level or let's say executor is out of memory because of broadcast table ?
Hello Manish, you are just awesome and I hardly found one other than you who teaches the in-depth. I am from Bangladesh, and my Hindi is not that good so can you please add English subtitles to your video?
I have a doubt. In shuffle sort merge join and shuffle hash join, is it correct that sorting and hashing are performed first before the join? Furthermore, does the join process occur the same way as you taught in the previous video?
Hi Manish, shuffling takes place when we are joining two tables then how in broadcast we are saying that we are not doing shuffling and due which performance is good as we are using broadcast. As broadcast mein bhi toh join toh lag raha hai na toh shuffling toh hogi phir kaise it is different from shuffle sort or hash
Hi Manish Thank you very much for sharing great knowledge . Currently I have 10.5 Year Experience in IT including SQL,PLSQL(7 Year), SQL Server T-SQL (1.5 Year) and Snowflake Query Optimization 6 Month . When I was joined before 2 Year as Data Engineer (Spark with Scala) in one MNC company but He was given project on T-SQL . I was only taken trainings and search interview question and clear interview . At time I on bench what should be we take decision Please suggest me?.
Chat me kaise batau. Aap ek session book Kar sakte hai topmate par if you are confused. Waise to main yaha par padha hi rha hu DE. To aap isko follow karte jaiye aapko sab idea lagne lagega
If I have a spark job running perfectly fine in prod someday got crashed, how to check in prod env? As not everyone directly gets spark prod access. Please answer.
You will have to ask infra team to extract the logs from spark history server or you can store your error logs somewhere in DB. And ask the table read permission from the infra team
