26 Spark SQL, Hints, Spark Catalog and Metastore | Hints in Spark SQL Query | SQL functions & Joins 

It will become more fantastic when you share it with your network on LinkedIn and tag us... 🤩 We definitely need some exposure ☺️

Thanks. Please make sure to share with your network 🛜

Yes, this all should be sufficient for you to get started

In that case it doesn't matter, however AQE always prefer to broadcast the smaller table.

@@easewithdata Thanks & I'm following you for more than a month its been a great learning experience , we want you to make End to End Project in Pyspark

Hello, I understand the request, but it will not be possible to capture all issues/scenarios on RU-vid sessions. I will try to create a mini series later which will cover this topic. Easiest way to create an OOM exception and the most common one - is to create a driver with smaller memory size and then read dataset with bigger size and collect() it for display. Collect will try to fit all data in driver memory which will result in OOM. And to fix this OOM to use take() in place of collect. Hope this helps.

@@easewithdata I can understand. Thanks you are my big data guru

In case you are working with in memory catalog, the metadata will be lost once the compute or cluster is restarted. This is why it is recommended to have a permanent catalog.

@@easewithdata Thank you. This is the best content I have seen about spark

Hello, Local execution only supports with single node which is driver. It uses threads in your machine to execute tasks parallely. Now if you need more executors then you have to configure a cluster and use it in your master. Please checkout the beginning of the series to understand more.

Three more to go before a wrap up.