Spark context vs spark session difference between RDD, Dataframe and DataSet in spark what is On Heap memory what is Off Heap memory what is Garbage Collector Explain Spark internal architecture Difference between Spark cluster mode vs Client mode how spark do memory management what is driver out of memory exception and how to fix it what is executor out of memory exception and how to fix it what are transformation and action in spark difference between narrow and wide transformation what is fault tolerence in spark what is lazy evaluation in spark can one spark application have multiple spark sessions what is spark directed acyclic graph (DAG) what is spark application , job, stages and tasks how to calculate number of cpu cores required to process data in spark how to calculate number of executors required to process data in spark how much each executor memory is required to process data in spark how to calculate the total memory required to process data in spark how to setup spark configuration for cluster managed tables vs external tables temporary view vs global temporary view what is materialized view types of slowy changing dimensions how to create a dataframe by reading different file format(csv,json,parquet etc) how to create a dataframe out of a hive table how to write dataframe explain the concept of lazy evaluation in spark and its significance what is predective pushdown in spark what is sortmergejoin how can you perform a broadcast join what is partitioning and bucketing cache vs presist storage level of presists repartition vs coalesce how to create a new column in table using pyspark how to remove duplicates in spark how to fill null values in spark how can you select specific columns from spark dataframe how can you rename a column in a spark dataframe how do you perform a groupby operation in spark how can you join two spark dataframe explain the use of StructType and StructField classes in spark with example what is incremental load? how to implement? can you discuss the role of structed streaming in spark what is databricks unit catalog? what is the difference between with and without unity catalog? what is the difference with and without catalog what is RLS and CLS in databricks what is role based access control why unity catalog is better than hive metastore what is different roles in unity catalog what is medallion architecture what is delta lake what is delta table what are features of delta tables what is lakehouse architecture data warehouse vs data lake vs data lakehouse what is optimize in databricks and what does it do? explain about z-order function what is vaccum in databricks what is autoloader in databricks what is delta live tables in databricks types of databricks cluster and their uses?
