Can anyone provide me the pdf of this all classes?

sets are mutable.

person_df.join(address_df,address_df["personid"]==person_df["personid"],"left").select(col("firstName"), col("lastName"), col("city"), col("state")).show()

Bhaiya Videos kyun nhi aa rahi h, Ek community post daal dijiye. Thank you bhaiya for your Python and Spark Series.❤❤

@manish_kumar_1 Sir kya mast content banaya, maja aa gaya, thank you!

Wow, Manish, you are such an amazing human. Your ability to teach complex things such easily is not common. Please continue the Playlist. I have learned so much from your videos. Your lectures helped secure a job. I am post this only to let you know that your work isn't going to waste. A lot of people like me are learning and also implementing in real life.

Dsa ka question nahi puchha tha?

Very Good.....

for item in data.get('MAINDATA'): # print (item) for next_item in item.get('HeaderFields'): print (next_item) for key in next_item.keys(): if key =='FieldTypeName': count +=1 print (count)

hi manish !!! are you going to continue this series or this was the last video of this course please let me know

count = 0 for word in data["MAINDATA"]: for headerFile in word["HeaderFields"]: if "FieldTypeName" in headerFile: count += 1 print(count) Ans : count = 38

labour_with_cost = {"Mahesh" : 500,"Mithilesh" : 400, "Ramesh" : 400, "sumesh" : 300, "jagmohan" : 1000, "Rampyare" : 800} total_working_days = 50 absent_day = {"Mahesh" : 3 ,"jagmohan" : 7} for labour, cost_per_day in labour_with_cost.items(): if labour in absent_day: absent_count = absent_day[labour] present_days = total_working_days - absent_count total_cost = present_days * cost_per_day else: total_cost = total_working_days * cost_per_day logger.info(f"Total cost for {labour} is ₹{total_cost}")

The staging layer is not part of data warehouse. It is an intermediate storage used for data processing before it is loaded into the data warehouse. Otherwise, there will be no difference in data warehouse and data lake.

Ye pure project ke liye prerequisites kya hai plz rply

solution: df.filter(col("refer_id").isNull()).select("name").union( df.filter(col("refer_id") != 2).select("name") ).show()

please give data set so we can practice on thAT

I have recommended this channel to 100 of people , This is by far best channel for those who want to become data engineer.

more video in SCD manish bhai

good job man

where could we see high cpu usage in spark UI when data coming from disk to memory and gets deserialized ?

Hi Manish, I am using the same code and getting 4 jobs: flight_data=spark.read.format("csv")\ .option("header","true")\ .option("inferSchema", "true")\ .load("/FileStore/tables/2010_summary.csv") flight_data_repartition = flight_data.repartition(3) us_flight_data=flight_data.filter("DEST_COUNTRY_NAME=='United States'") us_india_data = us_flight_data.filter((col("ORIGIN_COUNTRY_NAME")=='India') | (col("ORIGIN_COUNTRY_NAME")=='Singapore')) total_flight_ind_sing = us_india_data.groupby("DEST_COUNTRY_NAME").sum("count") total_flight_ind_sing.show() Can you explain the reason for the same.

Thank you

Bhai mast batate ho

count = 0 for i in range(len(data["MAINDATA"])): for j in range(len(data["MAINDATA"][i]["HeaderFields"])): for k in (data["MAINDATA"][i]["HeaderFields"][j]): if k =="FieldTypeName": count = count+1 else: count logger.info(count)

Hi, Thank you for such informative videos I am not able to find Lecture-6 in Spark Fundamental Series, please guide me from where I can watch lecture -6

Did you missed to upload the video because after lect 5 you taught spark architecture.. didn't told of CSV files or corrupted data storing part.

Thank you so much for this. I now understand the concept of Window functions crystal clear!

Answer for Q1: sales_df = sales_df.withColumn('sales_date',from_unixtime(unix_timestamp('sales_date', 'dd-MM-yyyy'))) grouped_df = sales_df.groupBy('product_name',year('sales_date').alias('year'),month('sales_date').alias('month')).agg(sum('sales').alias('total_sales_monthly')) sum_window = Window.partitionBy('product_name','year') grouped_df.withColumn('total_sales', sum('total_sales_monthly').over(sum_window))\ .withColumn('percent_month_sales_wrt_total',\ round(100*col('total_sales_monthly')/col('total_sales'),2)).show()

Very informatic video.. like it love it

Python 3.10.7 and Spark 3.5.1 worked in my case

Sorry SCD0 explanation is wrong. In SCD0, Once the data is written to the table, we can't alter it no matter if the source data is changed

nice explanation

Hi Manish Sir, can you start a playlist on how to work on streaming data with Apache Kafka? Basically expecting a playlist on streaming data analyze and processing, Thank You.

I have done ineer join to filter matching records of right table with left table, althugh left_semi is now one shot solution 😅

Hii bro, how's work pressure and work life balance at jio

Count bhi ek action hoga na means 3 job create hua

Amazing explanation sir jii

l_w_cost={"mahesh":500,"Ramesh":400,"Mithilesh":400,"Jagmohan":1000,"Rampyare":800} total_cost=0 for i in range(0,50): for j in l_w_cost: total_cost=total_cost+l_w_cost[j] print(total_cost) #sub Jagmohan for 7 days for i in range(0,7): total_cost=total_cost-l_w_cost["Jagmohan"] print(total_cost) #sub Mahesh for 3 days for i in range(0,3): total_cost=total_cost-l_w_cost["Ramesh"] print(total_cost) 155000 148000 146800

your videos are very useful but your speaking content is more which makes the video large and boring

Thank you Manish bhai for the awesome videos, thanks for giving the script.

Love from Pakistan

superb Teaching Skills a lot of love from Pakistan 😘

superb Teaching Skills

Love from Pakistan ❤

proud u sir Love from Pakistan

real teacher🥰

Aashirwad dijiye guru ji,Aapke pure course ko complete kr paun 🙏

w = Window.partitionBy("Product_ID") Total_df = df.withColumn("Total", sum(col("Sales")).over(w)) Total_df.withColumn("Percen_Sales", round((col("Sales")*100)/col("Total"),2) ).show()

Hello Manish sir, first of all thank you so much for the knowledgeable playlist. we learned lot from it. can you make a video on. "How to read data from Database table using pyspark in Databricks?" Thanks.

i have attribute error in this video when I write df.write.format