Understanding HDFS using Legos

Подписаться 230 тыс.

Просмотров 149 тыс.

50% 1

You've been hearing about Hadoop and HDFS. How does it work?
In this video, we use an innovative method to show how HDFS works with Legos. Jesse Anderson shows how HDFS handles files and replicates the data, then covers the read and write paths for the data. Finally, he talks about how HDFS handles failure scenarios and the importance of data locality.
** Hadoop & Apache Spark training from NewCircle: newcircle.com/category/big-data

Наука

Опубликовано:

17 фев 2015

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 111

@davidvella7141 3 года назад

This video was uploaded 5 years ago and it still one of the best explanations I've ever seen.

@ThorstenStaerk 2 года назад

the first explanation at all that could tell me what Hadoop has to do with MapReduce

@stackinglittlesats 5 лет назад

Nice explanation man, if I could, I would buy you an air conditioner. You deserve it.

@SIRabhinav 5 лет назад

may the force be with you

@miguelchris6374 2 года назад

sorry to be offtopic but does anyone know a trick to log back into an Instagram account? I was stupid forgot my account password. I would love any tricks you can give me.

@miguelchris6374 2 года назад

@Billy Dariel it worked and I actually got access to my account again. Im so happy! Thank you so much you really help me out!

@billydariel9140 2 года назад

@Miguel Chris Happy to help :D

@ZeTamboh 2 года назад

@@billydariel9140 nice ad

@taxatlanticinc6611 6 лет назад

This is a best explanation I have seen yet! It's a lot more engaging and informative than the traditional PowerPoint! Thank You!!

@shaikabdussalaam5431 16 дней назад

You really have a " hands-on" approach of teaching this. : )

@AdrianRodriguezWebDevelopment 8 лет назад

This video just made my day! Thank you New Circle Training! And thank you Doug Cutting for sharing this video on Twitter.

@Coffingdw 8 лет назад

Nice job Jesse. Very informative and creative. Thank you. TeraTom

@wisdomandpeace4897 9 лет назад

Excellent video. I actually understand Hadoop somewhat after watching this video.

@amitprakashpandeysonu 2 года назад

Really nice and innovative way to teach hdfs concept. loved it and understood it very clearly. Thank you.

@AhlamLamo 3 года назад

Amazing explanation !! one of the best videos I've seen about HDFS

@TheMrsStinsfire 7 лет назад

11:06 R.I.P. Data Node 3

@supermonkey965 5 лет назад

He was a good node, admired by his node friends.

@ElCuchu 4 года назад

I'm still crying, can't get over it, such a good node dude rippp

@parthnagdev 2 месяца назад

He is happy in the Node Heaven and is saving all the replicated data it ever wanted to save.

@MuzamilKhan-rl2sh 4 года назад

Wow man, you explain it in a creative way.

@klausdupont6335 6 лет назад

Incredible illustration! Would love to see more on this topic in this form :-)

@gustavogbfBR 9 лет назад

Nice work. Really help me to understand how HDFS works.

@maryoleary8660 3 года назад

I love learning with legos, even watching it at 1.5x, I was able to follow along easily. Well Done.

@Happymoon789 5 лет назад

Thanks for your efforts! Smart display!

@dingman081130 6 лет назад

gorgeous presentation, thanks

@ravianantharamaiah7567 3 года назад

Excellent teaching. Conceptually things are very clear now. Thank you.

@prohouse6088 8 лет назад

very nice teaching methodology jesse, thanks for sharing

@abhijeet_r 6 лет назад

Very innovative presentation thanks a lot!

@1234abcd2139 7 лет назад

nice illustrative way of teaching HDFS. Would have been wonderful if some more information was given about fallback mechanism of Name node or coordinator

@CarlosMercadoINIGTDY 6 лет назад

Excellent video, thanks!

@ThomasEhardt 7 лет назад

Great introduction to HDFS!

@joecordingley7071 8 лет назад

This was great, thanks.

@i_e_she 5 лет назад

This was great, thank you! Should have more views.

@thandekilenzungu7240 3 года назад

The explanation is so clear I understood everything

@kausaralam2605 8 лет назад

Great explanation!

@yicai7 4 года назад

Great explanation! Voted!

@haydo8373 5 лет назад

Superb, I had it running at 1.5x and it was still easy to follow! Thanks :) Can you explain every CS concept with Lego? - that would be amazing

@juiliussmith5702 2 года назад

thanks dude lol

@markhellel3371 6 лет назад

Great Job Jesse! Nicely done! :-)

@nirupamaj6140 7 лет назад

very informative, thank you

@olesyagorbacheva6991 7 месяцев назад

Thank you for such a good explanation!

@thndesmondsaid Год назад

Jesse! Great explanation as always.

@MSDlublin 7 лет назад

Very good work for begginers - THANKS A LOT!

@CB-fz3li 4 года назад

Nice clear explanation

@jesusoliveros9950 5 лет назад

Amazing !!! great Job

@manojprabhakar5522 4 года назад

Awesome, Thank you for the explanation, Could you please make videos of Spark with Yarn and how the communication is handled?

@mrdhksan 5 лет назад

Excellent, thank you. A serious question: What would happen if two of the four nodes crash?

@BabtaOfEinGedi 6 лет назад

Perfect! Thanks so much

@arisweedler4703 6 лет назад

Great explanation! I assume that another benefit of the HDFS is that reading large files will be quicker, because you would be able to effectively "BitTorrent" from your cluster. Does HDFS do that?

@underlecht 3 года назад

looks like we have a perfect explaination!

@Guitarman01 7 лет назад

Good Presentation, however I do have question. Since the file is split to other nodes, doesn't replication also take places so that if a node does go down, then you can retrieve. Node 3 went down, but I would have figured I could get it from another node. Does master save a copy of all files as well? I didn't see how that works on the video.

@alaayari6391 3 года назад

thanks for the explanation

@bugs181 9 лет назад

I'm just now learning about the methods used in distributed file systems. I'm an application developer and it's a bit difficult to wrap my head around the lower level storage systems like HDFS. This video explained replication in an easy to understand way. Now only if I could have one other BIG question answered. What kind of file system would we use if we want an application to use a virtualized file system stored over many nodes? For example, we want each node to add additional storage capacity. To the application layer, this would look like a single big storage drive but to the lower level facilities this would use network coordination to serve the files to the application.

@kishorpatil6260 8 лет назад

ĺm ĺ

@rimchatti3807 5 лет назад

Nice job, it is helping getting familiar with HDFS. I'm new to Data Engineering and so on.. Could you please explain to me what is a cluster. Thanks;

@buzz-uk 5 лет назад

Hi, While setting up pseudo or full cluster, do we need to format data node with HDFS file system or we only have to format namenode. I am asking this because, I have read this on many blogs that, HDFS stores the data in sequential order on the hard disk and it is an abstract layer which stores data on big blocks rather than default block size storage provided host file system. If we are not formatting datanode than the powerful feature of HDFS will come to toss.

@TzGiwrgos15 7 лет назад

Brilliant!

@shyland20 6 лет назад

why s3 service streaming with embedded link is slow (get stock every 2-3 second) when embedded on wp site? after understanding what you saying how can i improve the speed? i read something about the hdfs but i don't understand how it's related to s3 if at all. thanks in advance

@CosmeJunior 6 лет назад

Nice Job. Brazil thanks you!

@mahdiamrollahi8456 3 года назад

Hello, nice job, I have a question, as a file system how hadoop can manage a database file(like mssql or mysql) file? how can it devide an .MDF file to other separeted files to store them on different machines? Because such files, have meta data and overhead and they are not like a basic txt file. Does hadoop have special system to treat each file type differently? Regards.

@ulrikkallblad6698 5 лет назад

Very nice video! Only one question: If node 3 is down, how can the data from node 3 be moved to the other nodes?

@forbin80 4 года назад

@@brianbitchballs3902 thanks for the great explanation BrianBitchBalls

@abhimanyukarkara4218 3 месяца назад

Question: when we have to read from let's say the red file. Would all three nodes be processing simultaneously different data (chunks) and give us an combined output or would only one node process the complete the data processing alone?

@marflem12 6 лет назад

Thank You

@taharhalloub8721 3 года назад

Thanks a lot

@JackyA123 5 лет назад

yoou haveabsolutely no need to be nervous! Doing a great job here

@ahmedaj2000 3 года назад

Thanks 😊

@joseenrique6723 2 года назад

For the red file, are EACH of the replicas still 64 mb in size?

@Gorlung 3 года назад

what does happen when you add a new and empty node?

@myeverymusic 5 лет назад

What will happen once Data Node 3 is alive again? will the Name Node asks other nodes to copy some data to Data Node 3?

6 лет назад

The fact that hbase write on a node as you say is the reason why it corrupts the HDFS filesystem so easily?

@stivstivsti 6 лет назад

thanx!

@nocontentnoname5922 4 года назад

Did we find who broke node 3 yet?

@AbhinavSingh-oq7dk 3 года назад

if a data node malfunctions, then name node instructs remaining data nodes to create replicas of files that malfunctioned data node held. why create another replica when there are two others already? I mean they are there for the backup, right? Do correct if I am missing something. Thanks.

@Irresponsibleful 4 года назад

did you get a AC by now ?

@elwyndude Год назад

If a node goes down, why does it need 3 replications to pull the data, could it not just read from the existing two?

@wow376 5 лет назад

feel like buying Legos already!

@satwindersetia4367 7 лет назад

Very creative, indeed.

@malesamuel7736 5 лет назад

Cool

@barefeg 3 года назад

What if hulk smashes the naming node?

@amni5tianone263 2 года назад

tnx

2 года назад

Interesting, it's very similar to how Elasticsearch works

@KenedyYinkfuChuye 7 лет назад

Now I like Legos :)

@GiacomoMilazzo 6 лет назад

I don't understand. If each set of blocks is "one" file (red, yellow, blue) why he says that blocks are replicated? He should say "distributed", not replicated! Replicating involve data resiliency, erasure code and so on. Is it? Then he put the case that one of the cluster's node crash. So in this case replications come on play. And, of course, he should not call the set of blocks "one" file. But it should say there's one file composed of "n" chunks that are replicated among nodes of cluster.

@draganglumac 6 лет назад

The way I understood it, = . I suppose (if my understanding is correct of course) the confusion then comes from the fact that at the beginning of the video he said that a = . He really should have started with just one row of lego bricks for each file, and just explained that a data node sends a copy of the block it just wrote to one its data peers as directed by the control node.

@user-oi3ce5nj3m 4 года назад

狡兔三窟说的就是这个意思

@user-ml2ci7wl1f 5 лет назад

英语不太好，但是觉得很棒

@danielleu.877 4 года назад

SUPER informative, but also i hear "Hadoop" and just think "Hadooken" just me? yeah okay hahaha

@ravatmehul3906 7 лет назад

Nican

@samiulsaeef2076 3 года назад

play in 1.25x

@guille.p 6 лет назад

It started off pretty well but then it got very confusing. He didn't seem so sure of what he was saying. It didn't work for me. Thank you, anyway.

@marcelscherzer8385 4 года назад

its lego, not legos... but nice vid.

@viewerone 5 лет назад

It's been quite a challenge to hear this video. Headphones are in but it doesn't seem to help.

@FredroStarr12 5 лет назад

audio volume is fine to me, must be an issue on your machine

@viewerone 5 лет назад

Freddy yes, that’s what it was. Guess my Mac needed a reboot. Worked fine afterwards.

@JM-fp3gf 9 лет назад

Why is he so sweaty?

@jessetanderson 9 лет назад

Yeah, it was the lighting. We tried moving the lights around, but I didn't have any makeup on which mitigates the lights.

@musasall5740 7 лет назад

You should not answer this moron. u doing a good job for free

@vishusingh008 7 лет назад

Looks like you are a moron!

@vishusingh008 7 лет назад

He replied so kindly and genuinely, you are calling him moron.

@stonemysterioserusss 7 лет назад

Pretty sure he was referring to the initial commenter, not Jesse. Rude remark nevertheless.

@cafecapes 9 лет назад

Why do Americans call Lego bricks Legos? Lego is a company name and small building bricks is what they make, they don't make Legos! You can't implicitly type Lego bricks as Legos it sounds silly.

@lucaborzani56 9 лет назад

we do the same in Europe. Where are you from?

@cafecapes 9 лет назад

I've been thinking deeply about this and decided I'm the worst person to be dictating English. Briton mate.

@bugs181 9 лет назад

cafecapes Every nation has it's own way of speaking. There's a very elaborate section on the Stack Exchange website that goes into great depth on the difference in languages, pronunciation, and word usage. If this is a serious inquiry, I'd suggest you go there. It's a very informative place to learn anything you want - and if the question hasn't already been asked, you can ask it yourself. One example is that there is a topic on how Americans pronounce the word solder as "sodder" while other countries pronounce it as "sold-er" and where this distinction came from. You might be surprised to know that the language variations have a lot to do with heritage dating way way back. Every language and dialect, regardless of what it is has become bastardized - and it's just a part of how languages evolve. For what it's worth, I used to pronounce it as "sold-er" until I got tired of being corrected - and I have no indication of where I learned this from. I now colloquially pronounce it as "sodder" just because of tradition and geographical linguistics. Also, pronunciations and accent vary widely in the United States from coast to coast also. Apologize for the long comment.