No video :(

Apache Spark - 04 - Architecture - Part 2

Подписаться 75 тыс.

Просмотров 64 тыс.

50% 1

Spark Programming and Azure Databricks ILT Master Class by Prashant Kumar Pandey - Fill out the google form for Course inquiry.
forms.gle/Nxk8...
-------------------------------------------------------------------
Data Engineering using is one of the highest-paid jobs of today.
It is going to remain in the top IT skills forever.
Are you in database development, data warehousing, ETL tools, data analysis, SQL, PL/QL development?
I have a well-crafted success path for you.
I will help you get prepared for the data engineer and solution architect role depending on your profile and experience.
We created a course that takes you deep into core data engineering technology and masters it.
If you are a working professional:
1. Aspiring to become a data engineer.
2. Change your career to data engineering.
3. Grow your data engineering career.
4. Get Databricks Spark Certification.
5. Crack the Spark Data Engineering interviews.
ScholarNest is offering a one-stop integrated Learning Path.
The course is open for registration.
The course delivers an example-driven approach and project-based learning.
You will be practicing the skills using MCQ, Coding Exercises, and Capstone Projects.
The course comes with the following integrated services.
1. Technical support and Doubt Clarification
2. Live Project Discussion
3. Resume Building
4. Interview Preparation
5. Mock Interviews
Course Duration: 6 Months
Course Prerequisite: Programming and SQL Knowledge
Target Audience: Working Professionals
Batch start: Registration Started
Fill out the below form for more details and course inquiries.
forms.gle/Nxk8...
--------------------------------------------------------------------------
Learn more at www.scholarnes...
Best place to learn Data engineering, Bigdata, Apache Spark, Databricks, Apache Kafka, Confluent Cloud, AWS Cloud Computing, Azure Cloud, Google Cloud - Self-paced, Instructor-led, Certification courses, and practice tests.
========================================================
SPARK COURSES
-----------------------------
www.scholarnes...
www.scholarnes...
www.scholarnes...
www.scholarnes...
www.scholarnes...
KAFKA COURSES
--------------------------------
www.scholarnes...
www.scholarnes...
www.scholarnes...
AWS CLOUD
------------------------
www.scholarnes...
www.scholarnes...
PYTHON
------------------
www.scholarnes...
========================================
We are also available on the Udemy Platform
Check out the below link for our Courses on Udemy
www.learningjo...
=======================================
You can also find us on Oreilly Learning
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
=========================================
Follow us on Social Media
/ scholarnest
/ scholarnesttechnologies
/ scholarnest
/ scholarnest
github.com/Sch...
github.com/lea...
========================================

Опубликовано:

21 авг 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 112

@ScholarNest 3 года назад

Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code. www.learningjournal.guru/courses/

@mjeewani123 2 года назад

I really like your content, very easy to understand. THANK YOU. Have you covered any where how RDD helps recover from fault tolerance?

@iitgupta2010 5 лет назад

This is the best video....wt a explanation sir..mind blowing. I feel bad where bad teacher get too much attention where people like you don't get much... U r brilliant

@philippederome2434 5 лет назад

I love the curtains opening up special effect!

@sethiabhithemaverick 6 лет назад

This is the easiest and most explanatory explanation of complete spark architecture one can ever get

@davidpeng8431 Год назад

Your video is one of the best for Spark, not spend too much on theory and high level, but down to the earth, very practical.

@manasamathi7157 2 года назад

Even beginners can understand the flow who has zero knowledge about spark.. Great explanation 😊

@SachinChavan13 4 года назад

Wow ! Very crisp and to-the-point explanation. Really helpful. Thank you Prashant!

@philippederome2434 5 лет назад

I like the animal logos for the 3 APIs, turtle for RDD (slowest), cat for SQL-Dataset (medium), rabbit for DataFrame (fastest), but see Brian M. Clapper recent video on Frameless API (fast, compile-safe, and more functional, i.e. can compose actions).

@muhammadarslanbhatti2139 3 года назад

Hands down the best explanation you'll find on youtube

@sthirumalai 4 года назад

I started learning Spark by enrolling to a self learning course in UDEMY but by far this is the best video i have ever watched which explained the core concepts of SPARK clear and precise. I appreciate your efforts

@rajatsharma6137 3 года назад

extremely lucid and to the point...congrats !

@franciscovinueza5320 4 года назад

Images, Colors, examples and clear explanations! This video has everything! Keep up the good work! Thank you Sir.

@maheshpoddar4065 4 года назад

Exceptional way of explaining and making concepts crystal clear. I am enjoying it the way I used to enjoy your earlier videos on Hadoop.

@sanjaysoni5017 4 года назад

Awesome video with correct explanation.

@anilpatil3056 6 лет назад

Highly recommended for every Spark newbee. BTW thanks a lot..

@Nasirmah 5 лет назад

It should be a=> (a(1),1)) to get the second field if you want the result to be as shown in 9:30. First field of the array is empty string, but you can still reduce by key since it will be all empty string at the end but not if all files didnt contain /etc root path. I find it useful to run collect() at each step like kvRDD.collect() to see. Thank you very much for the best spark tutorial, I let the adds run to help out.

@gefeizhu3953 5 лет назад

Fantastic video,I have subscribed your video!

@erpoojadak 6 лет назад

the best tutorial i have ever seen..simply awesome

@abhishekt450 4 года назад

Just brilliant 👌.. point to point..

@nishantgupta8562 6 лет назад

Best video by far.. What a teacher you are.

@shubhampatil2391 Год назад

Thank you for the great content!! just one request though please add a highlighter to your pointer it is kind of hard to track its movement and often have to rewind to check what actually you clicked on

@sandeepkumarvadde 5 лет назад

This is an extraordinary explanation of spark architecture. Sir, please pick a few examples to implement on cluster mode too.

@pramodswain6043 6 лет назад

It is really appreciated,i never ever seen an explanation like this,so thanks a lots sir for revealing such extraordinary skills.....

@bhavaniv1721 4 года назад

Please post more videos,I following all Ur video,Ur videos are something different to others.....it easily understandable way

@pratiksingh9480 3 года назад

you mean Data Savvy :P , Yeah Prashant is really good in explanation

@shivam.shakya 2 года назад

Great video

@sachinpatil4218 4 года назад

Outstanding..

@kannanarumugam9257 6 лет назад

Thank you ver much!, nicely explained spark architecture. there is no other better way than this.. keep the good work.!

@vijaykumar-wq9db 4 года назад

Thank you sir...super video

@malapatiprasanna 5 лет назад

Thanks a lot, sir for your outstanding efforts in making us brilliant. Could you please add some more spark videos shared variables, detailed transformations, and actions. Really I am double satisfied with your explanations, going forward we want to see more from you on spark.

@AzharHussain2u 3 года назад

just awesome

@khelifakemouche4070 5 лет назад

Great tutorial and Excellent teaching

@deepanshunagpal6440 3 года назад

nicely explained.

@Vihaan_Nigam16 5 лет назад

Excellent way of teaching ...Thank you

@KoushikPaulliveandletlive 5 лет назад

just too good, you need too much of knowledge, when you can explain the complex things such easily

@elvinanoronha6032 4 года назад

Awesome explanation !!!!!

@sachinhaldankars 5 лет назад

Simply Awesome explanation...

@vishalteli7343 5 лет назад

Simply Best!

@damodharable 6 лет назад

excellent teaching skills,thanks a lot :)

@avijitmukherjee678 4 года назад

Thanks so much, Sir

@akashhudge5735 4 года назад

nice explanation

@premrajkumar6910 6 лет назад

Nice video with very clear explanation. But We will have to wait very long for a new session . Please try to upload fast, otherwise it will take a year to learn Spark.

@tejaswianagani8756 5 лет назад

Very very good explaination sir, am very much thankfull to you.

@pratiksingh9480 3 года назад

Hi Prashant Sir , First things first : I am planning to take-up the course . Your explanation viz etc. are awesome kudos for that. The only thing that concerns me is that I have lot of questions when I study anything , some silly as well. Is there any channel (Slack/Discord etc for enrolled students) , where the doubts are cleared. Some AMA kind of sessions etc , becuse going through stuffs and having uncleared doubts will leave a learner is almost the same state. Will share the same message with you over lnkedin as well , not sure how frequently you look into RU-vid comments.

@TheVikash620 6 лет назад

Great explanation sir. Waiting for new concepts to be covered in future videos.

@helloworld4u 3 года назад

Thankyou

@althafmohammed5285 6 лет назад

It's really amazing it's really real time level

@RahulEdvin 5 лет назад

excellent explanation !

@asksmruti 6 лет назад

Your tutorials are simply awesome.. :-) Super Like

@sanjaykumarmahapatra 6 лет назад

Nice way of explanation. Thank you so much for your effort on making so nice tutorials. I am becoming a fan of you man! keep it up (Y)

@pc0riginal870 5 лет назад

thank you so much from the bottom of my heart. god make you happy.

@repsycled1605 6 лет назад

One of the best video series for learning .. Do you also provide classroom trainings as well

@sagarsinghrajpoot3832 5 лет назад

Great video 🤓🤓sir

@sunilgaikwad3254 5 лет назад

Hello Sir, loved this tutorial..thanks a lot. I have one doubt, consider following scenario: Input data size(raeding from hdfs): 20 GB No of executors: 2 executor memory : 8 GB RDD partition factor: 2 and we run a spark job in client mode. So in this case: 1. how total 20GB data will get processed through sparkjob? 2. How many stages and task will get created? 3. how total 20gb data will be partitioned?

@ScholarNest 5 лет назад

1. Do you need 20GB memory to process 20 GB data? No. More memory can improve performance but you can still process it with less memory. 2. Stages depend on your logic and the number of task on executors. 3. You asked for two partition so it will shuffle and make it two in that stage. Next stage depends on other factors.

@csharma8732 5 лет назад

Very nice video sir. Thank you.

@damodargoud6263 5 лет назад

thanks for sharing your knowledge.

@paritoshahuja5058 5 лет назад

Really amazing explanation thank you

@nationviews6760 6 лет назад

Thank you so much, Sir, for providing such a nice practical explanation.

@user-ol6xv2gq3k 5 лет назад

very very amazing . thank you

@raunakgpt 4 года назад

Very Good video. Thanks sir. But I didn't anything with Apache Spark -05 in Playlist. Do we have some more videos on architecture?

@biswajitsarkar5538 6 лет назад

Great explanation !!

@a143r 5 лет назад

xcellent sir....!

@sbylk99 5 лет назад

Best tutorial thank you!

@ramkumarananthapalli7151 3 года назад

Hi Thanks a lot for these videos. They are quite helpful. In this video you mentioned that RDD is immutable, but you have overridden same RDD right, by changing number of partitions. Also we can load different text file into the same named variable(RDD). Could you explain how it is immutable in this case. Thanks in advance for your help.

@AIMLBites 5 лет назад

Thanks for the wonderful explanation in this video. Can you please tell if this is a general scenario for each job in spark, that map and reducebykey operations usually run in 2 different stages always or if there are cases that they can run in a single stage as well. Any examples or leads would be appreciated!

@ScholarNest 5 лет назад

Think about it. Map and Reduce? You are already talking about two stages.

@NareshJadapalli236 5 лет назад

I am confused in one step. When we say, RDD distributes data into nodes. We create 5 partitions from RDD. It means RDD has loaded all the data and do partitioning, is it? Will it load data from different nodes to the driver node and keep it in memory and distribute across? If yes, it is not following data locality paradigm and data movement is very costly. (I am sure spark follows data locality) What am I missing?

@Modern_revolution 5 лет назад

Super happy

@4ukcs2004 6 лет назад

Sincerely looking for spark streaming with Kafka tutorial sir...when r u pubishing sir..you are the best..

@kidslearningscience 5 лет назад

A supplementary video with Amazon EMR please.

@NikhilKekan 5 лет назад

Hello,Great tutorial. Can you please elaborate more on reduceByKey((x,y) => x+y) that you have used to count number of pairs with same key. I am a bit confused how x+y will give us the total count

@xmankamal 5 лет назад

Here reduceByKey is aggregating the result of array (for similar key) to one value. Suppose you have (key, value) list :- List((hello, 1), (world, 1), (hello, 1), (hello, 2)). reduceByKey will perform operation on similar key and x, y denotes the value only from key, value pair (you cannot perform operation on key here) For key: hello rdd.reduceByKey(x,y => x+y) -equivalent to (1,1 => 1+1) => List((hello, 2), (world, 1), (hello, 2)) There are still pair exists belongs to hello key here. so again operation will be perform rdd.reduceByKey(x,y => x+y) -equivalent to (2,2 => 2+2) => List((hello, 4), (world, 1)) Now list has only one hello key pair, so no further reduction can be possible here.

@annaynomouse2821 3 года назад

How do you create animation shown from 4:50 to 4:55. Which software. I like how you bring clarity visually.

@ScholarNest 3 года назад

Power point :-) Office 365

@amirboutaghou274 5 лет назад

hello , so first of all i want to thank you for this superb tutorial. please i have one question following your example of imagin we have 10 partitions and 2 executor and we lets suppose in this example we dont have transformation that gona cause shuffle how many task parrallel there is it 5 ? thank in advance for your answear

@ScholarNest 5 лет назад

Why do you think it's going to be 5? Because 10 partitions /2 executors? Number of executor have nothing to do with how many tasks are created. Once tasks are created, they will run on only two executors because you have only two executors.

@amirboutaghou274 5 лет назад

@@ScholarNest first of all thank for your quick reply. so i undersntand in my example number of task created per stage depend only by number of partition . number of execeutor have nothing to with it im a correct plaese ? so i my example i will still 10 task because i have 10 partition ?

@143badri 5 лет назад

What is the default number of partitions if we are not defining it...

@premrajkumar6910 6 лет назад

Also if possible please explain the code using Java APIs too. I am doing development using Java API, but some methods are not supporting even after it's mentioned in API document and throwing run-time error. Is that when we are doing development using Java API or Python API, will it get converted to Scala language internally?

@nidhidewan5173 6 лет назад

waiting for more videos :)

@ScholarNest 6 лет назад

+Nidhi Dewan, coming soon.... I am slightly busy to code a website for learning journal. Just another week away from the release.

@tushibhaque863 6 лет назад

I thank you from the deep of my heart for your hard work....

@tajirapb 5 лет назад

With spark 2.3.2, number of elements within each partition is not being displayed by the code that your have shown.

@rakeshsahoo16 4 года назад

Why proc and opt came in 1 partition ??

@surajpillai2117 5 лет назад

hello... I had a question. The intermediate RDDs which are generated, the partitioned data under them would also be distributed to the executors? or would the redistribution only happen on an action? Please help! :)

@ScholarNest 5 лет назад

Everything is lazy so nothing happens until an action is executed.

@robind999 5 лет назад

Very good one .any airflow demo?

@ScholarNest 5 лет назад

That's still incubating...I do not use open source until they graduate to become production ready.

@robind999 5 лет назад

@@ScholarNest Thank you so much for this info.

@abhishekbarnwal5867 5 лет назад

I am using spark 2.2.0 but the code shown by you in the video doesn't print any output in shell. val myrdd = sc.textFile("UserData.txt",4) myrdd.foreachPartition(x => println("No. of elements in partition: " + x.count(y=>true))) Please share the workable code.

@gopinathGopiRebel 5 лет назад

Sir i have a doubt like how no of cores of executors and processing of partitions depend on ?

@AwaraGhumakkad 4 года назад

Sir i have executed the textFile() command with 5 partitions in cluster mode (5 W) but every time I could see that job is being executed only 1 of the workers. I mean in every run only 1 worker was executing all the partitions. is there any extra configuration required here. i am using spark-shell mode

@AwaraGhumakkad 4 года назад

Please ignore this i got my answer . thanks anyways

@nikhil199029 5 лет назад

is reducybykey a spark/scala specefic function?

@DavidZYW 6 лет назад

thanks, I have a question, does the shuffle and sort executed in multiple nodes ?

@ScholarNest 6 лет назад

Yes, every node that owns a partition must participate in shuffle & sort.

@ravikirantuduru1061 5 лет назад

I have one doubt is no of partitons is equal to no of executive?

@csharma8732 5 лет назад

NO, Executor runs tasks in it.

@ravikirantuduru1061 5 лет назад

Sir I have one doubt is no of partitions is equal to no of executors?

@ScholarNest 5 лет назад

Not necessarily.

@KnowWorldsFact 5 лет назад

Thanks Sir, Can you please give me link for part-3. I couldn't find

@ScholarNest 5 лет назад

Check the playlist

@KnowWorldsFact 5 лет назад

Thanks,will Check. you have explained all videos in very simple language. :)

@jaineshmodi 6 лет назад

Sir I am doing development using spring Kafka, could you please help me with consumer question? how do i poll in regular intervals e.g every 5 mins and how do I specify number of records to be read in every poll? I saw batch listener can be used to specify number of records to read but did not find polling interval option. Thanks.

@ScholarNest 6 лет назад

+Jainesh Modi what is spring Kafka?

@jaineshmodi 6 лет назад

Learning Journal sir I meant to say Kafka with spring boot

@ScholarNest 6 лет назад

+Jainesh Modi have you seen my Kafka videos? I have discussed consumer APIs in detail. I am not sure what do you mean my batch listener?

@jaineshmodi 6 лет назад

Yes sir i have gone through ur videos. my requirement is as a consumer i want to put delay with every poll and also want to control number of records being read in every poll.

@ScholarNest 6 лет назад

+Jainesh Modi to be honest, I haven't used spring Kafka, just saw the documentation. Looks interesting. I will plan for some time to evaluate it and send you details if I find an answer to your problem.