DAG and Lazy Evaluation in spark 

yes, sum is an action in spark

NameError: name 'flight_data_repartition' is not defined what to do in this case even i import functions and types from pyspark please I stuck here

Jobs count is right it is 4 only because sum() function is an action, which I guess Manish missed by mistake. Btw @Manish thank you so much for the amazing course.

@@jatinyadav6158 If 'sum' is an action then why it didn't create a job before adding 'show' codeline ?

@deepanshuaggarwal7042 yes sum is an action, I am not sure why it didn't show a job earlier

Nope

Yes until you write .head(2) for 2 highest record your process will not start although in backend it will create DAG.

Aapke 3 actions me 4 jobs create hue hai? Aapne show use kara hai? And aap apna code v paste kar dijiye comment section me

@@manish_kumar_1 Aapne jo code likha hai usme bhi 4 jobs create hue hain. Can you explain that? Below is my code and same thing is happening. 4 Jobs are getting created. Stage is getting skipped but why do we have an extra job with 4 diff Job Ids? from pyspark.sql import SparkSession from pyspark.sql.functions import * spark = SparkSession.builder.master('local[5]') \ .appName("Lazy Evaluation internal working") \ .getOrCreate() flight_data = spark.read.format("csv")\ .option("header","true")\ .option("inferSchema","true")\ .load("D:\\Spark\\flight_data.csv") flight_data_repartition = flight_data.repartition(3) us_flight_data = flight_data.filter(col("DEST_COUNTRY_NAME")=='United States') us_india_data = us_flight_data.filter((col("ORIGIN_COUNTRY_NAME")=='India') | (col("ORIGIN_COUNTRY_NAME")=='Singapore')) total_flight_ind_sing = us_india_data.groupby("DEST_COUNTRY_NAME").sum("count") total_flight_ind_sing.show() input("Enter to terminate")

Aapko google karke pata karna chahiye ki kon kon se actions hai. Rest are transformation

Spark optimization is very limited. So as a developer we should write optimized code to run our process faster

Completely different purpose and different technology. Google kijiye aap

@@manish_kumar_1 thanku bhaiya , ap Apache airflow pr bhi series bna do please, i really connect with you , jo paid courses se samj nhi aata wo ap ache se samjha rhe ho , thanku bhai , love you bro 🙌❤️

@@rampal4570 is it true bro

Aaj aa jayega

@@AmitSharma-ow8wm NameError: name 'flight_data_repartition' is not defined what to do in this case even i import functions and types from pyspark please I stuck here

Seems like your df is not defined

@@manish_kumar_1 ok working now thanks

1 job skip hua hoga. Agar data Kam hai to explain karke dekhiye 3 aana chahiye

Because it doesn't make cycle. If it's get into cycle or you can consider it as a circle then it will run endlessly

I don't provide notebook or pdf. Take notes and type every line of code by yourself. This will give you confidence

@@manish_kumar_1 this is indeed really great point! However, if possible, do share your own reference material for our benefit! Thanks! This series is really helpful, I've 4+ YoE in DE but never tried to go into spark internals, now while interviewing for switch, I'm definitely going to utilize all this! Keep 'em coming!! 🙌🏻👏🏻

Around 20-30. It's just beginning of spark

Basic questions would be there. Like what is DAG and what is edges and vertices in it.

Aapko practical and fundamentals sath me dekhne hai. First video me hi bataya tha

Aur inferschema v use kiye honge. Isliye aa rha hoga

@@manish_kumar_1 thank you sir, can you please share the action list how many action hai spark me, if possible please share sir

Col ko import karna padenga voh ek function hai toh import functions karo I think so

Correct "from pyspark.sql.functtions import *"

@@AliKhanLuckky NameError: name 'flight_data_repartition' is not defined what to do in this case even i import functions and types from pyspark please I stuck here

@@manish_kumar_1 NameError: name 'flight_data_repartition' is not defined what to do in this case even i import functions and types from pyspark please I stuck here

Sure

Sir also considers spark 5s problems.spill, shuffle, storage, etc

Can you please paste your code in comment section

@@manish_kumar_1 from pyspark.sql.functions import col flight_data=spark.read.format("csv")\ .option("header","true")\ .option("inferSchema","true")\ .load("dbfs:/FileStore/tables/flight_data.csv") flight_data_reparition=flight_data.repartition(3) us_flight_data=flight_data_reparition.filter("DEST_COUNTRY_NAME='United States'") us_india_data=us_flight_data.filter((col("ORIGIN_COUNTRY_NAME")=='India')|(col("ORIGIN_COUNTRY_NAME")=='Singapore')) total_flight_ind_sing=us_india_data.groupby("DEST_COUNTRY_NAME").sum("count") total_flight_ind_sing.show() (5) Spark Jobs Job 22 View(Stages: 1/1) Job 23 View(Stages: 1/1) Job 24 View(Stages: 1/1) Job 25 View(Stages: 1/1, 1 skipped) Job 26 View(Stages: 1/1, 2 skipped) flight_data:pyspark.sql.dataframe.DataFrame = [DEST_COUNTRY_NAME: string, ORIGIN_COUNTRY_NAME: string ... 1 more field] flight_data_reparition:pyspark.sql.dataframe.DataFrame = [DEST_COUNTRY_NAME: string, ORIGIN_COUNTRY_NAME: string ... 1 more field] us_flight_data:pyspark.sql.dataframe.DataFrame = [DEST_COUNTRY_NAME: string, ORIGIN_COUNTRY_NAME: string ... 1 more field] us_india_data:pyspark.sql.dataframe.DataFrame = [DEST_COUNTRY_NAME: string, ORIGIN_COUNTRY_NAME: string ... 1 more field] total_flight_ind_sing:pyspark.sql.dataframe.DataFrame = [DEST_COUNTRY_NAME: string, sum(count): long] +-----------------+----------+ |DEST_COUNTRY_NAME|sum(count)| +-----------------+----------+ | United States| 100| +-----------------+----------+

Aap sql tab me jaake dekho. Kitne jobs skip hue hai. And share me your code and screenshot of the sql tab on LinkedIn or Instagram.

@@manish_kumar_1 abe chutiya tere video ka bol Raha he wo 😂

Kisi ek video me bataya tha. Please follow all videos in sequence

Got the file, thanks

2 playlist hai. Parallely dekhiye. Practical wale me data milega description me usko copy karke save Kar lijiye as csv

Bhaari mistake ho gaya 😂