one of the best explanation in depth, Thanks :) Could you please make a video on "end to end Data engineering" project, from requirement gathering to the deployment.
I have one query, Cache() is equal to persist(pyspark.StorageLevel.MEMORY_AND_DISK). Only difference in this scenario is that cache() uses deserialized and persist used serialized data. So, if persist is better in terms of data serialization and functionality, what is the use case of using cache over persist ?
You already have the answer in your question, for cache the data is already de serialized thus no hassle but in persist the data is serialized and need to be deserialized before processing.
Consider you have a orders dataframe with 25 million records now you applied a projection and a filter and cached this dataframe as shown below orders_df.select("order_id","order_status").filter("order_status == 'CLOSED'").cache() Now you execute the below statements... 1) orders_df.select("order_id","order_status").filter("order_status == 'CLOSED'").count() 2) orders_df.filter("order_status == 'CLOSED'").select("order_id","order_status").count() 3) orders_df.select("order_id").filter("order_status == 'CLOSED'").count() 4) orders_df.select("order_id","order_status").filter("order_status == 'OPEN'").count() please answer the below queries... question 1) what point of time the data is cached (partially/completely) ? question 2) Which all queries serves your request from the cache, and which all will have to go to the disk. Please explain.
