No video :(

Apache Spark - 03 - Architecture - Part 1

Подписаться 75 тыс.

Просмотров 122 тыс.

50% 1

Spark Programming and Azure Databricks ILT Master Class by Prashant Kumar Pandey - Fill out the google form for Course inquiry.
forms.gle/Nxk8...
-------------------------------------------------------------------
Data Engineering using is one of the highest-paid jobs of today.
It is going to remain in the top IT skills forever.
Are you in database development, data warehousing, ETL tools, data analysis, SQL, PL/QL development?
I have a well-crafted success path for you.
I will help you get prepared for the data engineer and solution architect role depending on your profile and experience.
We created a course that takes you deep into core data engineering technology and masters it.
If you are a working professional:
1. Aspiring to become a data engineer.
2. Change your career to data engineering.
3. Grow your data engineering career.
4. Get Databricks Spark Certification.
5. Crack the Spark Data Engineering interviews.
ScholarNest is offering a one-stop integrated Learning Path.
The course is open for registration.
The course delivers an example-driven approach and project-based learning.
You will be practicing the skills using MCQ, Coding Exercises, and Capstone Projects.
The course comes with the following integrated services.
1. Technical support and Doubt Clarification
2. Live Project Discussion
3. Resume Building
4. Interview Preparation
5. Mock Interviews
Course Duration: 6 Months
Course Prerequisite: Programming and SQL Knowledge
Target Audience: Working Professionals
Batch start: Registration Started
Fill out the below form for more details and course inquiries.
forms.gle/Nxk8...
--------------------------------------------------------------------------
Learn more at www.scholarnes...
Best place to learn Data engineering, Bigdata, Apache Spark, Databricks, Apache Kafka, Confluent Cloud, AWS Cloud Computing, Azure Cloud, Google Cloud - Self-paced, Instructor-led, Certification courses, and practice tests.
========================================================
SPARK COURSES
-----------------------------
www.scholarnes...
www.scholarnes...
www.scholarnes...
www.scholarnes...
www.scholarnes...
KAFKA COURSES
--------------------------------
www.scholarnes...
www.scholarnes...
www.scholarnes...
AWS CLOUD
------------------------
www.scholarnes...
www.scholarnes...
PYTHON
------------------
www.scholarnes...
========================================
We are also available on the Udemy Platform
Check out the below link for our Courses on Udemy
www.learningjo...
=======================================
You can also find us on Oreilly Learning
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
=========================================
Follow us on Social Media
/ scholarnest
/ scholarnesttechnologies
/ scholarnest
/ scholarnest
github.com/Sch...
github.com/lea...
========================================

Опубликовано:

21 авг 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 141

@ScholarNest 3 года назад

Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code. www.learningjournal.guru/courses/

@jayeshpatiljp 2 года назад

atleast show courtesy to share link of 2nd part in description.

@wiwauhichale 4 года назад

I love the concise language and diagrams you use to explain each concept. Makes it easy to understand for new learners like me. Many manuals and books are overcomplicating things by jargons and not goes straight to the point like you do.

@juniorfarmer1351 4 года назад

sir multiple people saying u became your self like a god...........then i did not belive but...after when i watched the video then i fixed s u bacame ur self like a god.....yes its true.....and one more thing i have seen multiple videos regarding spark architecture but i didnt get this much of clarity.....so tnx a lot.....its amazing video

@payalbhatia6927 5 месяцев назад

Great Video .. I got this much information by reading thousand articles and getting confused. Worse , still in doubt. your one video in 2024 made it all clear.. Thanks

@thedude6341 6 лет назад

You are pure gold. Content is amazing, structured and well deductively explained, thank you

@mohindersingh0711 2 года назад

You are one of the best teachers on youtube. Your style of teaching with simple examples is really helpful. thank you so much for making these videos. you are awesome sir.

@althafmohammed5285 6 лет назад

Yu ll become trend setter the way ur showing slides the explanation evry thng peeks

@ganeshsundarachary8882 6 лет назад

Sir, i seriously suggest you to post your videos in UDEMY, to reach max and all. This is excellent and better than many paid videos. Excellent ,pls do share more such videos. we are thankful to you.

@ScholarNest 6 лет назад

+ganesh sundarachary thanks for the suggestion. I will think about it. However, I know my viewers are good at sharing, and sooner or later, all those udemy viewers that are looking for Spark will reach to this place as well.

@rameshthamizhselvan2458 6 лет назад

for all other videos i have to watch 5 to 6 times then only i can able to grasp few things but your videos are quite impressive in the single shot itself i can able to grasp more things ... thanks a lot.... i'm willing to donate you tooo

@hamidkhan0590 5 лет назад

I could never imagine I can learn Apache Spark without attending the physical sessions. You not only saved my time but give me confidence in all senses. Thanks for such a honest efforts !

@Bikerbug 2 года назад

Best video to understand Spark Architecture

@hamidkhan0590 4 года назад

Everything is very nicely explained. Thank you.

@ranjandas1797 5 лет назад

I am unable to stop myself to subscribe and like this video. its clean and fantastic explanation... making things clear.

@hariparuchuri8406 5 лет назад

the quality of your content is amazing.

@vickgoldgmailcom 4 года назад

Best tutorial so far on youtube

@sudhanshuguru209 5 лет назад

Pandey Sir... Teacher ho to aap sa.. warna na ho :). What a clear way of teaching.

@ab-zo4kw 4 года назад

Wow, best lecture by far. Well done

@chetansp912 6 лет назад

One of the best videos of Spark i have come across untill now.. Thank you for explaining it so beautifully.. :)

@anuradha8700 5 лет назад

I have been following your videos for quiet a long time.I found each and every videos of yours to be so informative and easy.Thanks Prashant for all of these videos

@VikashSingh-cj1qw 4 года назад

This video is pure gold.

@sushmashamsundar8219 4 года назад

sir.. great work.. very nice explanation. i have seen lot of links, gone thru many videos, but didn't get clarity. your way of explaining is unique.. thank you soo much sir..

@jayanthsn4581 5 лет назад

one of the best spark arch explanation ever seen on youtube,,, wondeful!!! Thank u so much sir

@shivam.shakya 2 года назад

Wow! This was fun to watch. Great demo. Thanks.

@MR-vm7jp 4 года назад

Dude I am falling in love with you..amazing teaching content and style.

@sairajaramandoju3025 5 лет назад

The pace at which you are explaining and the way is excellent. Please make videos on Hadoop on AWS

@4ukcs2004 6 лет назад

No words to express..simply awesome video... :) thank a lot Prashant sir

@tejasnandola9375 3 года назад

This is mind blowing .. Fan of you

@gcpdev123 5 лет назад

Prashant Sir, I don't know how to thank you for such a wonderful content. I had to revise the spark and hadoop concepts, and what a wonderful way to do this. Thank you Prashant.

@papunisahu2411 5 лет назад

Superb explanation.Expecting more spark video from you.Thanks for sharing this video

@ryas8360 5 лет назад

This is very good explanation.. this is first video on RU-vid where I am commenting..

@yitbarekmirete6098 4 года назад

you are a awesome instructor. good job

@AudioVoiceStories 2 года назад

Sir I'm very thankful for your video. If I stuck in any point I come here to clear my doubts. Just a small request what ever you explain on CLI please increase the font size to make it visible. Thanks.

@harishurgent 6 лет назад

Wonderful. We are lucky to have your materials. Simple and powerful.

@anshulgautam2344 3 года назад

One word - Best

@samhusksr 6 лет назад

Much appreciate the standard of the content.It's simple,precise and context.Thank you

@lovetoka 2 года назад

Best in the internet...

@temp9483 6 лет назад

One more gem of a video from Mr. Pandey. Awesome work !

@edsonsabino 2 года назад

3:15 How spark execute a program on a cluster 5:04 Who executes where 8:09 Who controls the clusters 14:00 Demo

@cherrejim 4 года назад

Thanks 🙌🙌🙌

@vladvlad2798 2 года назад

AMAZING VIDEO AND EXPLANATIONS!

@palashwarghane1420 5 лет назад

Sir its avery informative video series..pls keep uploading..big fan after watching first few videos..thanks

@sundarrajkumaresan8045 4 года назад

Wow, vera level. Thanks for the course. im one of the follower in Udemy & Linkedin. Once again thanks for this low hanging fruitful tutorial :) :) :) I got it what i was looking for :)

@pariksheetde4573 4 года назад

Hi Sir...I really like and admire your efforts that you put for us. Thanks once again Pariksheet

@JawDroppingFacts 6 лет назад

WOW. This video was so helpful!! Thank you

@duongpham9861 3 года назад

This video is so amazing! It's very very clear. Thank you so much

@gabancariaok 4 месяца назад

Thank you sooo much for the great explanation!!

@mrcool4uall 5 лет назад

Thank you very much.. This is one of the best tutorial I have come across.Appreciate your efforts.

@raminology 6 лет назад

Thank you, for the very practical and compact introduction to Spark.

@zp4288 3 года назад

wow Excellent !

@aneksingh4496 6 лет назад

Fantastic teaching style...please continue to do so

@mthatipamula 4 года назад

Excellent explanation.

@amardhillon314 6 лет назад

You are the most awesome teacher. hats off

@abhi-wanderlust 5 лет назад

These videos are simply amazing. Thanks a lot. Hope to see more tutorials from you.

@mohammadhaque1873 2 года назад

Great explanation.

@dibyajyotisarkar5288 5 лет назад

OMG! This video is so amazing! Thank you so much for the amazing session.

@pallavigupta321 5 лет назад

Sir can you please upload videoes regarding the project in spark or Hadoop. If not please suggest me some project related to spark and Hadoop. Bcoz your way of explanation and clearness in voice is great

@josedejesuslopezdiaz 5 лет назад

Thank you sir, the quality of your content is amazing.

@SurendraKapkoti Год назад

Great learning!!!!

@sarfarazhussain6883 5 лет назад

Great video.. Came to know many unknown things from this video... :)

@chsrinivas2426 4 года назад

Million Thanks Sir, Course of the content amazing sir.

@rohangupta4290 2 года назад

Loved the content.

@bhavaniv1721 5 лет назад

Thanks sir for sharing your knowledge, expecting more videos like this

@mohitbansal6360 6 лет назад

Very useful Sir.. In depth knowledge you are providing with proper examples. Thanks a lot..!!! Looking forward to watch your more videos. And Sir, My jupyter notebook is not working..I have specified my problem in that video. Look into that too.

@sudheerthulluri 6 лет назад

Sir ur explanation was very good. waiting for the next video

@piyushdoifode8066 6 лет назад

Great Video sir!!!.. Really clears my concepts Thank you

@ashishlimaye2408 5 лет назад

Very well explained the concepts in the video. Great work Sir!!!

@robind999 6 лет назад

Very good demo, learned more today, thank you.

@ReinePoukou 4 года назад

Thank you billion times…..

@murifedontrun3363 5 лет назад

Hi sir, your videos are great. I have a request. Will it be okay if you make a video on real time Spark applications getting slow or getting failed. The causes of such failures and slowness and what could be done to prevent it. Thanks in advance :)

@AwaraGhumakkad 4 года назад

Very informative sir thanks 🙏. I have few questions- 1. Let’s suppose we have 3 nodes + 1 master - while staring the executors does that mean - each node will have 1 executor . 2. Do we need to keep the spark binaries in executor as well . Or that will be taken by spark master node

@rmaleshri 5 лет назад

good explanation

@pc0riginal870 5 лет назад

thank you so so so much Prashant sir

@Knigh7z 3 года назад

19:39 - is that correct? From what I can see, the driver is not on your local computer. I thought the driver would've started in the spark-shell process which you ran from the master node that you were SSH'd into. So is the driver actually running on the master in that case?

@mamtajain1882 5 лет назад

When we start spark-shell through edge node of cluster,it is which mode(client/cluster),because driver in running on one node of cluster right.Could you please explain.

@rahuljaiswal9379 5 лет назад

Supperb explaination sir.... Thanks 😊

@MrVivekc 4 года назад

Sir what is the difference between spark session and spark context. What purpose does spark session solve? Does it helps in overcoming any shortcomings of spark context??

@niharranjanbiswas3812 4 года назад

suitable

@tamilarasan948 5 лет назад

Really nice... 👏

@sudhakark3244 6 лет назад

Excellent demo video... Thank you..I feel that i couldn't see spark shell content and spark summit command...It would be better if video quality is more on clarity

@ScholarNest 6 лет назад

+sudhakar k, video is full HD 1920x1080. Check your internet speed. However, all the commands and the code is also available at www.learningjournal.guru for copy/paste.

@AwaraGhumakkad 4 года назад

+1 3. All the json or other files those you are processing , are residing in the same machine (node), in that case if we have 3(executor ) + 1 (master) architecture- does spark copy the file to each executor for processing , or master node shares the file partially with each node ? If we need to connect to S3 to process a huge file ? How will be the executor processing the file?

@svdfxd 6 лет назад

Awesome explanation sir

@haneefsk2845 6 лет назад

awesome video very useful thanks

@amirboutaghou274 5 лет назад

first of all i want to thank for this amazing video. please i have one questions : what is the difference between container and executor? what do you mean by this sentence " application master will start an executor in each container or an executor will run in container" Thank in advance

@ScholarNest 5 лет назад

Executor is a Spark worker process. Container is a resource bundle allocated to the process to run. For more details please check my Hadoop tutorial playlist. I have a video on Yarn in that playlist that explains container allocation in more detail.

@pratheek5096 2 года назад

You rock !!

@bharathjii 6 лет назад

Excellent !

@HimanshuSingh-id4xl 6 лет назад

Very well explained. While trying the hands on , Spark UI is not getting accessible on my laptop even though I received message that spark UI is available on :4040. While hitting that , it is not reachable. Tried with the :4040, but that also not worked. Please share if any additional step required to access spark UI from laptop when VM is running and spark-shell already started.

@ScholarNest 6 лет назад

+Himanshu Singh if you are trying it in Google cloud, you should use ssh tunnel to access Spark UI. I have explained that in one of the videos.

@Jainadarsh3092 6 лет назад

sir, the way you explain very use and simple.could you please any for spark reference.Thanks in advance.

@mdtausif3480 6 лет назад

Great explanation sir. Thank you so much!! I have one doubt. In the example you are using 1000 slice. What is the background concept of it and how is it helping to make the process run longer?

@ScholarNest 6 лет назад

Keep watching, I don't remember exactly, but it is covered in some video.

@kirangrandhe1559 3 года назад

So do you mean driver process will be created in Node manager or Resource Manager. Also Application master is also Node manager ?

@balabhaskararaoguntupalli1275 5 лет назад

thank you, wonderful connect.

@sbylk99 5 лет назад

So much appreciated bow

@deepakgupta-hk9ig 6 лет назад

Sir, It means that as a developer we worked on local and standalone mode and never worked on cluster using yarn or other cluster manager?

@csharma8732 5 лет назад

Nice. Thank you

@chetanbonde2256 6 лет назад

Good Explanation sir

@jamestanaka686 5 лет назад

I have a question. Why your available executors are 5 as a default? As you have 1 master node and 3 workers nodes with 2 cores for each node, I thought the available executors are 6 (1 executor per core). Or is it just a default behavior of YARN? Also I am thinking that in YARN, we can increase the number of executors as much as we want, e.g. 100 executors on 3 worker nodes (though it is probably inefficient). Also based on my research online, I see that 5 executors per each core is ideal in YARN.

@ScholarNest 5 лет назад

Number of available executers = (total cores/num-cores-per-executor) - (1 for application manager) So in my case it is (6/1)-1=5 If you ask for more than what is available, it goes into a queue.

@poonamraykar4426 6 лет назад

Excellent explanation. Do you have videos on spark streaming ?

@parthdayala7519 5 лет назад

god bless you all your life sir

@bentyryan 6 лет назад

Thank you for the video, it was very informative. When can we expect the sequel?

@ScholarNest 6 лет назад

Next week for sure. It's another 20-minute long video and hence taking some time.

@codewithrajranjan 6 лет назад

very nice explanation

@AwaraGhumakkad 4 года назад

At -19:43 when you did a ssh to the master node and started spark-shell with 3 executors then why the driver will start from your machine ? It shd start from one of the nodes