Brendan Gregg from Sun's Fishworks team makes an interesting discovery about inducing disk latency. For a ca. 2020 retrospective on this 2008 video: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-_IYzD_NR0W4.html#t=28m47s
My father always told me never run around running computers when I was a child. "Its bad for the spinning harddrives" . Watching this 20 some odd years later made me feel somewhat satisfied.
Between the 40s and 80s, my grandfather worked for a bunch of companies in the tree and steel industry. One of them produced wood boards (some kind of plywood or particle wood) that you would place underneath the computer in the computer room to absorb vibrations in the building. I was always fascinated by stories like that.
Our old computer was stationed on the 2nd floor of our house, and it was made of wood We always wondered why running or jumping would cause the computer to crash, now we know
I miss the old internet. This was the realest content Ive seen in years and its 13 years old. No ads, no sponsors, no 5 minute intro. Just a video of a guy with something cool he found and wanted to share with us. Their smiles and the way they carry themselves is wholesome and fun. Not taking things too seriously while still getting things done properly.
@@kaidwyer That one, now bankrupt company (I don't remember the name) used to make GFPs for like a year or two, but they got replaced by LED (Langer Emanagement Dyes) screens.
@@kaidwyer gamer proof monitors used to exist. they were called crt monitors. you could take a baseball bat and hit the screen on the glass as hard as you could. and it wouldn't shatter.
This reminded me of my father's lab; my father is a seismologist. I would jump where the seismograph sensors were located and it created 10-11 level earthquake readings on seismograph. Such fun times.
He is so happy because he finally has a cause for his latency issues. I have felt that relief before and the fact he had to make a video to prove the issue is just a testament to how unreal the cause ended up being.
While developing our technology, we discovered that the siren tone we used in our initial version would cause a number of laptops to Bluescreen due to persistent Read / Write failures. As it turns out that most drives are sensitive to sounds below 100Hz or neat 1kHz. We adapted our siren pattern and eliminated the problem. We found this video after the fact.. In hindsight, I wish we found this before all the headaches.
Soo serious question, I like to bass really hard to the point the walls shake a tad bit, and i'm having a problem with HHD's getting way more wear to them than they should be in my room... (SSD however is fine ) could this be the reason why? I'm basically ruining my drives with bass? lol
@@dementedpeep I am also interested in this question because I have a subwoofer on my desk beside my PC -- after watching this video I am mildly concerned about the concept of prematurely wearing my HDD due to excessive bass when I record/produce music and use the sub at higher volumes.
@@dementedpeep ABSOLUTELY... Go SSD all the way if you like bass... but it is not specifically bass which is the problem. There are specific frequencies which cause resonance in the head assembly. When we were developing our technology we hit one of them, and it caused irrecoverable R/W errors on the drives.
Just for reference, that 1 server he yelled at could hold a maximum of 24TB of data, and would be about $27k if Sun's $1/Gb marketing is to be believed. Nowadays, all that hard drive storage could genuinely fit in the space of your toaster and is reasonably affordable for a single person.
There was a very popular online photo service that was having cooling issues in their datacenter. They brought in some large floor fans to help cool it down while they waited for repairs. Within 8 hours they were in a full panic as they have lost 40% of the disks in that whitespace. Well one guy started mapping out the lost drive physical location and realized that servers that were within a few feet of the fans had almost complete disk loss. Vibration is Hard Disk killer.
Jarrod, spot on. Had one question come up about a loss of performance after adding a bunch of new disks to the arrays, I asked: what's the harmonic frequency of the spin rates between old drives versus the new? I love when interviewers ask real questions, and begin texting other engineers. And the look, "that… is unexpected" when the responses come back.
@@float32 vibration. If the read/write head were a Boeing 747, and the hard-disk platter the surface of the Earth: The head would fly at Mach 800; At less than one centimeter from the ground
@@jarrodvsinclair why did you stop there? Follow through with the analogy. That cm of air acts as a nearly impenetrable boundary layer, applying millions of lbs of force, as the Boeing approaches the ground. Heads only crash when drives stop spinning, or you apply massive g forces by dropping it to the ground. Tracking errors are *easy*, which is what this yelling, and almost certainly the fans, would be causing. There’s no protection for alternating magnetic fields though. That’s how you make a degausser.
@@float32 inverse square law and comparing induced emi magnitude to the remanence of the platter coating. I'm more likely to believe damage was caused by physical vibration sooner than oscillating magnetic field generated by a synchronous motor in an ac fan.
I kept screaming at my laptop trying to kick in the HDD protection... My parents ran inside my room thinking i was being murdered. Now they think i am insane.
today i got 4x VTrak m500f, i have absolulty no idea how to use thoes things besides taking out hdd's for spare parts :D And that i need SunSolaris or Windows 2003 server for thoes old beasts
It was Sun Storage Appliance 7000 Analytics. Now called Oracle ZFS Storage Appliance Analytics. In particular, I was showing heat maps of disk I/O latency - which are crucial for examining the full distribution of latency over time. Similar heat maps are now in Joyent Cloud Analytics. In both cases they used DTrace to fetch the data.
@@Teddemeister 9 years ago was probably before youtube comments work like they do now. They didn't use to be nested threads, so early discussions were an absolute nightmare
This video is now six years old. If you aren't aware, Brendan and Bryan are amongst the top tier engineers in the world, but more importantly, they made this video of them death metal screaming into disk arrays. I came to watch this for nostalgic reasons this evening, and it was certainly great. I think it's just as important as anything to say that they're still two guys that yell into servers for fun even to this day. They're also probably on my top ten list of serious engineers who are also seriously approachable. The world needs more people who can think like this.
Had this happen in a recording studio: A guitar amp was being tracked in the same room as the drive and the DAW kept complaining that the disks couldn't keep up. The amp had to stay so the drive was placed on a piece of foam and the problem went away. Turned out the floor vibration from the amp was interfering with the drive.
@@Sigma_Eight um yes that also helps them an ssd can take hits that a mech drive cannot and keep on going so it can handle vibrations like my shits in the toilet can
In my early admin days, there were many 2am, sitting on the floor of a data center, staring at a console screen and praying the server comes back up. It's a special PTSD attached to good nostalgia
I tried this, quite dangerous as the disk came loose and exited the room via a burning hole it made in the roof. On the bright side though, for a brief moment it was transferring several gigabytes a second.
I'm basically impressed by two things, the disc drives being depressed when you shout at them and that the video popped up that was recorded 14 years ago, just wow :o
This brings back memories- I remember the server room at a company I worked with was so cold you had to wear a parka and gloves, and was so windy that a loose piece of paper would quickly go flying if not held to a clipboard.
I thought I'd never see this video again! I remember being recommended it shortly after it was uploaded... and now 14 years later, it's happened again.
I had a data center on the top floor of a 150 year old brick building, next to a railroad track, two rail lines actually. We got a lot of hdd failures that I was convinced was the vibration. Random raid fails that a rebuild would fix. You could see monitors wiggle on the far end of the building from the tracks. Glad to be gone from there.
@@drac124 Laptop 2.5" drives have active protection against head crashes that enterprise 2.5" SAS drives and typical 3.5" drives of all flavors lack. This is also one of many reasons why flash storage is preferred for laptops these days.
@@drac124 The vibrations from the train passing by might not be as large in magnitude, but it's all about acceleration (you'd probably have to hit the laptop pretty hard on something to replicate the kind of acceleration the shaking from train can cause)
This research was already done at Sun Microsystems a couple years ago (2). They got the same results. This is why they use vibration dampeners on the fans. It would also be a good idea to use water cooling, or just go with a solid state drives if you have the money
One of our customers once stopped their entire operations because they thought it was a good idea to use a drill on their storage rack. They were installing a new appliance and it didn't quite fit. This was a major exam labs franchise in Brazil. Fun times.
*refreshing old information This is some of the most archaic shit that it needs to be taught in computing history class lmao this is not how it is nowadays.
I have a large subwoofer, originally from my car, connected to my computer. On one occasion, I played music at nearly 70% of the subwoofer's maximum volume capacity, which resulted in a BSOD. The low frequency sound waves from the subwoofer vibrated the HDD (Hard Disk) so much that it got damaged. Thankfully I've had a backup of the drive. The solution to be able to use the subwoofer at high volumes with the pc, was an SSD.
Just wait till you accidentally find a resonant frequency for those nand chips, I bet that same frequency will also cause problems with the rest of your computer.
@@andreasjoannai6441 The resonant frequency of NAND chips typically falls into the higher end and outside the audible range of human hearing, way beyond subwoofer-related frequencies. Sub sound frequencies are incapable of resonating them. A HDD is a totally different story though.
@@andreasjoannai6441 Of course you can. But, again, the resonant frequency (specifically of NAND chips) falls into the higher end of the sound frequencies and outside the audible range of human hearing, way beyond subwoofer-related frequencies. The reading needles in a HDD are susceptible by lower up to middle frequencies. On the other hand, NAND chips would need a high frequency so strong that probably no current twitter-speakers can produce. Subwoofers are incapable of producing so high frequencies since their range is typically from around 20 Hz to 200 Hz. The upper limit of human hearing is often considered to be around 20,000 Hz (20 kHz), while beyond the range of human hearing, there are ultrasonic frequencies used in various applications such as medical imaging, industrial processes, and animal communication. These frequencies typically start from around 20,000 Hz and extend into the megahertz (MHz) range and these are the ones that can actually influence NAND chips, if strong enough.
@@Lemon_Inspector I don't know. Sure one leg, one arm, and half my face aren't working anymore, but I'm doing fine! (Tries to leave room, but falls after first step.)
DAMN MAN VERY COOL VIDEO, I DIDNT HEAR THE LAST PART OF THE VIDEO BECAUSE OF ALL THE FANS, BUT IT WAS VERY COOL TO SEE THE LATENCY INCREASE FOR THE DISKS YOU SCREAMED AT, VERY COOL.
True story i used to live near brendan and was mates with him and his brother around 18 years ago, we used to get on it and talk computer talk, somehow i stumbled across this video and its a spinout seeing him again, apart from the goatee and filling out he hasnt changed a bit! Small world
It's been quite a while since I've seen this video. Still funny and most people don't stop to think how amazing the logic involved on read/write heads are to correct for vibration!
Comcast is laying conduit in my neighborhood and left a run buried with either end sticking out several feet above ground. When we saw this driving in earlier I parked the car at one end, ran to the other, and shouted into the pipe at my friend on the other end. This reminded me of that.
Yes, and yes. DTrace (the feature that makes this analytics magic happen) is available in both these operating systems. The Sun 7000 series storage appliance in the video is based on OpenSolaris. A large amount of Sun server hardware is supported for Linux and Win2K3 as well.
