Тёмный
No video :(

driver out of memory spark | spark memory management | Lec-18 

MANISH KUMAR
Подписаться 20 тыс.
Просмотров 17 тыс.
50% 1

In this video I have talked about spark memory management. I have shown how driver OOM occurs in spark. Next video will be on executor OOM. So stay tuned for the next video. If you want to optimize your process in Spark then you should have a solid understanding of this concept.
Directly connect with me on:- topmate.io/man...
Flight Data link:- github.com/dat...
For more queries reach out to me on my below social media handle.
Follow me on LinkedIn:- / manish-kumar-373b86176
Follow Me On Instagram:- / competitive_gyan1
Follow me on Facebook:- / manish12340
My Second Channel -- / @competitivegyan1
Interview series Playlist:- • Interview Questions an...
My Gear:-
Rode Mic:-- amzn.to/3RekC7a
Boya M1 Mic-- amzn.to/3uW0nnn
Wireless Mic:-- amzn.to/3TqLRhE
Tripod1 -- amzn.to/4avjyF4
Tripod2:-- amzn.to/46Y3QPu
camera1:-- amzn.to/3GIQlsE
camera2:-- amzn.to/46X190P
Pentab (Medium size):-- amzn.to/3RgMszQ (Recommended)
Pentab (Small size):-- amzn.to/3RpmIS0
Mobile:-- amzn.to/47Y8oa4 ( Aapko ye bilkul nahi lena hai)
Laptop -- amzn.to/3Ns5Okj
Mouse+keyboard combo -- amzn.to/3Ro6GYl
21 inch Monitor-- amzn.to/3TvCE7E
27 inch Monitor-- amzn.to/47QzXlA
iPad Pencil:-- amzn.to/4aiJxiG
iPad 9th Generation:-- amzn.to/470I11X
Boom Arm/Swing Arm:-- amzn.to/48eH2we
My PC Components:-
intel i7 Processor:-- amzn.to/47Svdfe
G.Skill RAM:-- amzn.to/47VFffI
Samsung SSD:-- amzn.to/3uVSE8W
WD blue HDD:-- amzn.to/47Y91QY
RTX 3060Ti Graphic card:- amzn.to/3tdLDjn
Gigabyte Motherboard:-- amzn.to/3RFUTGl
O11 Dynamic Cabinet:-- amzn.to/4avkgSK
Liquid cooler:-- amzn.to/472S8mS
Antec Prizm FAN:-- amzn.to/48ey4Pj

Опубликовано:

 

21 авг 2024

Поделиться:

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист
Посмотреть позже
Комментарии : 40   
@akhiladevangamath1277
@akhiladevangamath1277 3 месяца назад
That smile on your face when driver OOM error came😂, shows how much you enjoy teaching us🤗
@priyankasethi2922
@priyankasethi2922 3 месяца назад
I have recently joined MNC, and I must say this is so accurate content. thank you so much for sharing your knowledge and experience
@coding_BeastMode_ON
@coding_BeastMode_ON Год назад
Bhai itne easy or detailed way me koi ni samjhata hai. Great work !!
@Tech_world-bq3mw
@Tech_world-bq3mw Месяц назад
5:16 literally first guy in my life who is happy to get error.😆
@manish_kumar_1
@manish_kumar_1 Месяц назад
It's not first time when I was happy with error. Jab bhi hamara code Bina error ke galat output deta hai then I want it to give me error so that I know exactly where to fix this 😀
@kunalk3830
@kunalk3830 Год назад
ekdum mast hai bhai ye wala video...keep going
@engineerbaaniya4846
@engineerbaaniya4846 Год назад
Awesome most asked interview question this channel should get more subscribers and views
@kunalk3830
@kunalk3830 Год назад
sab reels dekhne me vyast hai bhai
@anviscreations1265
@anviscreations1265 Год назад
Awesome content please continue the series
@tanushreenagar3116
@tanushreenagar3116 2 месяца назад
perfect video sir
@mallangivinaykumar9500
@mallangivinaykumar9500 8 месяцев назад
Can you please make videos in English language. It will be easy to understand .
@praveenkumarrai101
@praveenkumarrai101 Год назад
hats off bro loving your channel
@ajaypatil1881
@ajaypatil1881 9 месяцев назад
great video bhaiya
@08rajdeepsonawane94
@08rajdeepsonawane94 Год назад
Thanks sir for your great video ❤
@rekhasingh4945
@rekhasingh4945 Год назад
Please make the series continue
@adarsharora6097
@adarsharora6097 7 месяцев назад
What do we mean by container here? Does it has any other name which we studied in earlier videos?
@akumar2575.
@akumar2575. 3 месяца назад
day 6 done 👍
@tnmyk_
@tnmyk_ 6 месяцев назад
Mastt video! Description me S22 ultra bilkul nahi lena hai aise kyu likha hai 😂
@yashwantdhole7645
@yashwantdhole7645 9 месяцев назад
What are objects here due to which driver overhead oom error occurs? Can you please explain?
@DpIndia
@DpIndia 11 месяцев назад
nice video
@raviyadav-dt1tb
@raviyadav-dt1tb 7 месяцев назад
How would we know that which file is small when we do broadcast Join, please tell me
@sammail96
@sammail96 7 месяцев назад
Hi Sir, I have came across some conflicting information regarding show() method. Kindly confirm the below information: The show method does not bring all the data from a single partition to the driver, but rather collects a sample of rows from each partition and display them.
@akhiladevangamath1277
@akhiladevangamath1277 3 месяца назад
As of my understanding, show method brings all the data from a single partition of single executor to the driver, but displays a sample of rows from that partition.
@arnabghosh106
@arnabghosh106 3 месяца назад
My driver is stopping while writing a dataframe. I didn't use any action rather than write. We can mot increase cluster configuration. How to resolve the issue? Any suggestion?
@manish_kumar_1
@manish_kumar_1 2 месяца назад
This information is not sufficient to provide some suggestion
@rekhasingh4945
@rekhasingh4945 Год назад
👍
@PavanKumar-vi7hd
@PavanKumar-vi7hd 5 месяцев назад
Hi Manish could you please upload same videos in english
@ajaywade9418
@ajaywade9418 7 месяцев назад
i saw few videos they mention max(0.07*memory, 384mb)
@prajwaljamunkar
@prajwaljamunkar Год назад
Hi Manish , when you will arranged live chat ....
@manish_kumar_1
@manish_kumar_1 Год назад
As soon as possible
@ashutoshkumarsingh3337
@ashutoshkumarsingh3337 Год назад
ok so garbage collection comes under overhead or jvm heap memory?
@manish_kumar_1
@manish_kumar_1 Год назад
Jvm heap memory
@vishaljoshi1752
@vishaljoshi1752 11 месяцев назад
Hi Manish, driver programme will run on master node or worker node as you have written Application Master (worker node)?
@amazhobner
@amazhobner 8 месяцев назад
Application Master/Driver is Driver node not worker node. Master Node is where your resource manager is, though DB does give option to have RM and AM/D to be on the same machine now.
@ajourney179
@ajourney179 Год назад
Hi Manish, Any good resource to lean Scala ?
@venkataramana-yh3th
@venkataramana-yh3th Год назад
Bro, please make vedio in English
@udittiwari8420
@udittiwari8420 6 месяцев назад
i think this is Lec-15 but by mistake it is written lec-18 sir
@rh334
@rh334 Год назад
Let's say we read a csv file of 10.1 GB stored in datalake and have to do some filtering of data, how many tasks will run? is there a possibility of, out of memory error in the above scenario?
@manish_kumar_1
@manish_kumar_1 Год назад
10100 mb/128mb = 79 partition. So total 79 task will be created and since you are doing filtering which doesn't require any dependency from other partition which means it is a narrow dependency transformation. Every partition can do it's own filtering. As long as you have more than 500mb of executor size you will not face any oom. But make sure that you are not calling a collect function otherwise you may face driver oom.
@rajnarayanshriwas4653
@rajnarayanshriwas4653 4 месяца назад
why 500 mb? @@manish_kumar_1
Далее
МЕГА МЕЛКОВЫЙ СЕКРЕТ
00:46
Просмотров 179 тыс.
Əliyev və Putin kilsədə şam yandırıblar
00:29
Просмотров 211 тыс.
Advancing Spark - Understanding the Spark UI
30:19
Просмотров 51 тыс.
The Tragedy of systemd
47:18
Просмотров 1,1 млн
МЕГА МЕЛКОВЫЙ СЕКРЕТ
00:46
Просмотров 179 тыс.