Hello, what if I wanted to do this in VSCode instead of Jupyter Notebook with docker as shown in the video?

What is invited to apply in naukri?and how to reply?

so in deployement mode the driver program is submitted inside a executer which is present inside a cluster. am I rignt?

Thanks a lot for sharing. It will be very helpful if you add data directory in git hub repo

Great 👍

thanks for this valuable insight, expecting the same video for apache iceberg and hudi in future

Very nice video

need to understand one thing why yyyy and dd not in capital letter is there any reason for that

Thanks for your lessons! You covered all the gaps between the main concepts.

Thanks for such a detailed work. You're a Hero.

Sir can u make a video to setup spark and hadoop with docker

Can a fresher become data engineer?

please make a video on creating resume for senior data engineers and please share the template thanks

Thanks, pls share the senior data engineer resume template.. will help

spark's standalone cluster is where on docker or any where please tell me my cluster execution codes are not running why?

Thanks bro :)

how to contact you sir

great work

thank you for your efforts

thanks for your efforts it helps lot

can accumulator variables be used to calculate avg as well? as when we are calculating the sum it can do for each executors but average wont work in the same way.

thank you in real time do we use cluster node or cline mode which you are using now?

localhost:4040 is not working when I done ".master("spark://e75727ddf432:7077")" how can be solved?

Why you made 32 shuffle partition if you have 8core, If one partition is going to process on single core, from where it will get other remaining 24 cores?

when I am refreshing my spark ui is giving error how can be solved giving this "spark://6b16b66805db:7077" and on "localhost:4040" also not working I give this".master("spark://6b16b66805db:7077")" how can be solved please.

Thanks for the Explanation, Very useful

From the vidoe : Select, where, group by etc ate transformations. We have narrow transformation and wide transformation. Wide transformation are those when data has to move or interact with data of other partitions in next stages

Again from video itself: executors are jvm processes, 1 core can do 1 task at a time, above pic we have 6 cores, so 6 tasks were possible

Shuffle is the boundary which divides job into stages

Great explanation.. liked the illustration that 2 counts happened and the fact that after local count and before global count, some shuffling happened

Awesome. Super explanation . I love it

why are we not using withCloumn instead of expr?

You are awesome bro.. Thanks a lot

What is the use case of slide duration in streaming app... any real world example ?

You are awesome man. I was trying to setup spark and kafka in docker for long time. done today. thank you very much

Hi, I have a doubt: How can we check if a stream has multiple sink from spark UI?

what if both the tables are very small like one is 5 MB and other is 9 MB then which df is broadcasted across executor?

one of the best explanation in depth, Thanks :) Could you please make a video on "end to end Data engineering" project, from requirement gathering to the deployment.

Hello, and thanks for the sharing of these useful videos. How to handle the writing in delta tables: Because the best practice is that the size of each parquet file should be between 128 MB to 1 GB. How to handle this situation while each batch has very less than the size that is mentioned? or how to handle to collect the number of batches and to reach the mentioned size and finally to write in deltalake.

Hi and thanks I have a question: What is the best practice in distributed mode for Spark? Is it good that the number of kafka partitions and spark executors are equal?

For me clearSource and sourceArchive is not working ,files are not getting archived and archive folder is not getting created.whta colud be the issues?

How do pyspark handles consumer group and consumer of kafka , generally by converting consumer group we can start consuming data from starting of topic , how to start consuming from beginning, startingoffset as earliest , will always read from starting .

I have implemented watermarking using tumbling window, of 5 minutes to aggregate values, triggering 5 minutes and used update mode, Problem is that I have got late data again, how it should be fixed. ///////////////////////////////////////////////////////////////////////////////////////////// df = df.withWatermark("open_time", "5 minutes") # Group data into 5-minute candlesticks by symbol candlestick_df = df \ .groupBy("symbol", window("open_time", "5 minutes", startTime=0)) \ .agg( expr("first(open_price) as open_price"), expr("max(high_price) as high_price"), expr("min(low_price) as low_price"), expr("last(close_price) as close_price"), expr("sum(volume) as volume") ) query_kafka = candlestick_df \ .selectExpr("CAST(symbol AS STRING) AS key", "to_json(struct(*)) AS value") \ .writeStream \ .format("kafka") \ .option("kafka.bootstrap.servers", kafka_bootstrap_servers) \ .option("topic", output_topic) \ .option("checkpointLocation", checkpoint_dir) \ .outputMode("update") \ .option("truncate", "false") \ .trigger(processingTime='5 minutes') \ .start()

Thanks for the video, but can you shred some lights which cases we prefer Window functions to "traditional" expr?

if you can cast it to a string, why not able to cast it to json, or map it out and use a json parser?

you can also just alias the struct and then select * from it. for example exploded_df.select(“data_devices.*”) and it gives all the columns

Why is there a shuffle write for the read stage from files to a dataframe? Does that involve a shuffle? Also a shuffle write for only kbs worth data?

Thanks, buddy. explode_outer is also an important concept.

Great explanations & effort taken. I have encountered an Py4JJavaError while executing the df_agg.writeStream.format("console").outputMode("complete").start().awaitTermination command.Could you explain why?

It's really great, I do have a question. What if the JSON is 5 to 6 level nested struct. How can we make it to tabular format?