Inside a 32x100GbE Switch and its Big Flaw

Подписаться 649 тыс.

Просмотров 49 тыс.

50% 1

STH Main Site article: www.servethehome.com/inside-a...
STH Merch on Teespring: the-sth-merch-shop.myteesprin...
STH Top 5 Weekly Newsletter: eepurl.com/dryM09
We check out the Celestica Seastone DX010 32x 100GbE switch and look at the different components inside and out. This is a popular switch platform for learning SONiC open networking. It is based on the Broadcom BCM56960 / StrataXGS / Tomahawk platform which means it has a lot of great L3 features. We then discuss the potentially major flaw that can brick this switch and many like it.
--------------------------------------------------------------------
Other STH Articles Mentioned in this Video
--------------------------------------------------------------------
- Get started with 40GbE SDN with Microsoft Azure SONiC for under $1K (Arista): www.servethehome.com/get-star...
- Intel Atom C2000 AVR54 bug released but hidden by NDAs: www.servethehome.com/intel-at...
- Intel Atom C2000 AVR54 bug prompts C0 stepping: www.servethehome.com/intel-at...
- The first time the AVR54 bug struck STH: www.servethehome.com/intel-at...

Наука

Опубликовано:

26 дек 2020

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 143

@bork3592 3 года назад

I got one of these and minecraft is smoother than ever

@bushhawk5460 3 года назад

I though: "Hey, that's a neat price" until I realised that you are going to spend at least twice that ammount on transcievers to populate the ports.

@ServeTheHomeVideo 3 года назад

DACs are pretty cheap for QSFP28. The QSFP56 ones for 200GbE are still expensive. I think a lot of people are using these with DACs which keeps power low and costs way down.

@GuruEvi 3 года назад

Twice? A branded 100G SR transceiver is basically the same cost as the switch, hell Cisco will sell you a brand new switch for this cost, given you buy their transceivers. Even second hand the good transceivers will end up going for $300+, not sure if you want to trust transceivers that are lower cost in a critical setup.

@ServeTheHomeVideo 3 года назад

@GuruEvi I do not think a $1000-1500 4+ year old switch like this is going to be using top-shelf optics. 😀

@bushhawk5460 3 года назад

@@GuruEvi or run in a critical setup

@JosephHalder 3 года назад

@@GuruEvi $29 for a 1m DAC, $39 for a 2m DAC on FS.com. No reason to use optics for a top-of-rack deployment. That being said, their SR4 transceivers are only $99 if you need to reach to other IDFs.

@jozefaz 3 года назад

We got hit by the intel chip bug and had to RMA over 2000 Cisco 4K routers. What a nightmare that was.

@ServeTheHomeVideo 3 года назад

For those of us who have seen it in action, it was not pleasant. Some folks have not seen it and are unaware/ do not believe in it. We even get comments to that effect now (see comments on this post www.servethehome.com/inside-a-celestica-seastone-dx010-32x-100gbe-switch/) but I still think it is important to raise awareness. 2000 routers! That stinks.

@wildmanjeff42 3 года назад

Thanks for the video Patrick, Love STH channel and content. Learned a lot from your channel :)

@henrik2117 3 года назад

2020: 2.5 Gbit becoming more common in retail computer motherboards 2016:.... 🤔

@michaltatka 3 года назад

Thank you for the review - I bought two, shortly I will test it (and SONiC sound interesting too). Time to strengthen the team Cisco 3064PQ (48x10GbE+4*40GbE) and 3164Q (64x40GbE), used for LAN.

@critical2210 3 года назад

Man I'm just happy with my cheap gigabit netgear switch lmao

@squelchedotter 3 года назад

I'll be very happy when that tier of switch finally gets to 2.5 and 5 gig :)

@spx2327 3 года назад

How can you not love this guy!

@TheBackyardChemist 3 года назад

If you are handy with a soldering iron, and you can find the correct pin/trace adding the resistor mod should be possible.

@zipp4everyone263 3 года назад

Man, this is cool! The bug isn't but the hardware is really really cool! I love the look of hardware and knowing what this is capable of just makes it so much more interesting! Thank you for the rare insight into this piece of DC hardware!

@nateb4630 3 года назад

How interesting! This is way outside my wheelhouse these days, but I used to play with big SONET gear, and lately I'm doing a lot more hardware hacking. While this had some gorgeous views, it left me hungry for more detail! I'd love to see more perspectives from different disciplines in here, especially a more detailed look at the board-stack connectors, all those debug headers all over, and the various silicon aside from the Broadcom piece de resistance. Also, how does that heatsink work being so wide; is there an integrated heatpipe or something? This hunk of hardware has more stories to tell!

@KrisLipscombe 3 года назад

I really like that placing on the power supplies, that’s just a nice feature and shows a bit of thought. Completely agree that you’d expect some better user servicing access at this price, particularly when this kind of switch is going to be key infrastructure

@someguy4915 3 года назад

You'd expect exactly less user serviceability for these, typically you won't even replace the fans or PSUs when they break, the manufacturer gets the call and replaces them for you. You don't pay them for nice equipment, you pay them for things to work, on cheaper servers I've had to replace a few broken HDDs but the blade systems just got taken care of by HPE and Dell if anything broke on them, too expensive to do it yourself.

@JuMPeRaBG 3 года назад

You got that switch for a $1000? This is insane!!!

@ewenchan1239 3 года назад

I got my 36 port 100 Gbps Infiniband switch for just under $2300 USD. (Retail is at LEAST $11k.) This is part of the reason why despite the cost of adapters, cabling, and the switch itself, on a $/Gbps basis, it’s still VASTLY cheaper than most other speed tiers.

@ion337 3 года назад

Nice teardown guys. If feasible it would've been interesting to see the underside and board-board interconnect for the switching PCB. Guessing it's regular PCIe between the BCM ASIC and x86 CPU?

@TheMschipp 3 года назад

Yep, normally PCIe 2.0 x1 for that generation

@excitedbox5705 3 года назад

Is that the bug where the Internal clock dies? You only need to add a new resistor/solder to another resistor. Dave from EEVblog had a NAS with that CPU and he shows how to fix it.

@Jonas_Meyer 3 года назад

Yes it is. He mentioned the resistor fix/rework in the video at around 7:33. Not sure how exposed the pin would be on this switch through.

@steven44799 3 года назад

The bright yellow battery holder is likely because they were cheaper than a black one, in a product like this where it's unlikely any customer is going to open the product it makes sense to just go with the weird colour part to save a bit on BOM cost as nobody buying them will care what colour it is.

@johndododoe1411 3 года назад

Depends if battery lifetime is less than service contract duration. Then there would be service instructions for customer staff changing batteries during maintenance windows, and then having a bright colour on the only user serviceable part makes sense. Besides, 12 screws is not as user unfriendly as the 4 non-removable screws I found when decommissioning a Gb D-link switch from 2010 and wanting to find and destroy the config flash chip before recycling.

@paulstubbs2778 3 года назад

Interesting, the bit about switch efficiency & cost per gigabit. Around here I am still mainly on a 100mb switch, it uses way less power and my internet connection cannot saturate it. If I am mucking about, and want to move a ton of data, I can easily patch in something a lot faster, however as a daily driver, watching ServeTheHome video's etc., the old Alloy 100mb is it.

@PrestonBannister 3 года назад

Interesting, as I really do not have a handle on the white-box network gear market. My use case is a high-end special-purpose gadget that moves a lot data. Kind'a sort'a looks like a datacenter in a box. (What I would *really* like to find is an *unmanaged* 25GbE switch, oddly.)

@Fuzzyfull 3 года назад

Did you test the power with all QSFP modules installed? As I understand, the large PSUs are for powering the transceivers.

@thatLion01 3 года назад

I'm surprised this 100gb switch is already 4 years old. I never actually seen 100gb used at any of the dc I sit. I did a few 25/40gbe upgrades that cost a fortune

@johndododoe1411 3 года назад

4 years ago, 100GbE was mostly for backbone providers and Internet exchanges, not datacenter internal networking.)

@thatLion01 3 года назад

@@johndododoe1411 thanks for clearing my vision. I never realized that! Excellent point 👍

@NvidiaTVH7 3 года назад

I would love to see a video on a low cost sonic network os setup, maybe a future video idea?

@ServeTheHomeVideo 3 года назад

We had the article by Chris linked in the description for this on a 40GbE network. It is pretty similar to other switches. I actually think that kind of CLI heavier content is better on the website.

@johndododoe1411 3 года назад

Is there a good alternative not from a company known to be hoarding data from "customers"? So obviously not Facebook / Google / Microsoft / AT&T.

@ws2940 3 года назад

Great video on this switch. Thank you for making it. Definitely looks like a Cisco clone of some sort.

@MRooodddvvv 3 года назад

I totally need one of those for my crappy 4G 1 megabit at good weather internet !

@Jonas_Meyer 3 года назад

Maybe they used the yellow battery holder for servicing. I suspect this would be the only component that actually get serviced so the technician can identify it faster. Or maybe they used them on another project or got them cheap.

@Jellman86 3 года назад

Love your content, are people actually serving the home with this class of equipment? Would love some fresh homelab content. Looking to refresh my stuff, what to buy in 2021?

@ServeTheHomeVideo 3 года назад

Totally. These are quiet enough that if you have a basement rack they are very usable. We have had people using these at home in the forums for over a year. On the home lab side, really Xeon Scalable/ EPYC is about the best there is aside from the TinyMiniMicro series we are doing. Intel has some new Atom and Ice Lake-D coming, but those have been delayed a bit.

@steven44799 3 года назад

One of the network guys we use at work has a cisco 9500 series switch in his home rack. Even he admits it's complete overkill and unnecessary, but he wanted it anyway.

@Jellman86 3 года назад

@@steven44799 @servethehome I can beleve it, If I had the money is be all about this. Hard one to get past the wife!

@jfbeam 3 года назад

@@steven44799 I have a small collection of Nexus kit as well, but it's still sitting around. They're too loud, and eat too much power to run them 24/7.

@ewenchan1239 3 года назад

@@Jellman86 I’ve deployed 100 gigabit Infiniband in my basement now. I was able to get it passed the wife because I used to have a CAE on consulting business where I was running HPC applications where stuff like this REALLY came in handy, and the Infiniband switch, even moreso. Start your own business and tell the wife it’s a business expense.

@AaronR-C 3 года назад

@ServeTheHome I'd love to hear some suggestions for other cheaply available switches in the multi-100Gb port range (using the same Broadcom chip as this, or equivalent) that aren't running on the C2000 platform. You mentioned that they exist?

@ServeTheHomeVideo 3 года назад

Have one inbound via FedEx ground.

@AaronR-C 3 года назад

@@ServeTheHomeVideo A) You're awesome B) Looking at this, and the STH article about SONiC on the Arista box led me to the SONiC supported hardware Wiki... and though it doesn't list which CPU a switch uses, it certainly seems like a great jumping off point for more research. github.com/Azure/SONiC/wiki/Supported-Devices-and-Platforms Thanks again for all you do.

@thibaultmol 3 года назад

That thing looks so similar to the dell switch that Linus looked at recently. So does dell use this company oem as well?

@draconightwalker4964 3 года назад

I’m thinking it’s the next model up the stack

@ewenchan1239 3 года назад

I don’t think so. Linus’ switch is 48 25 GbE SFP28 ports with 6 100 GbE QSFP28 ports. This is 32 100 GbE QSFP28 ports. Total switching capacity is higher here.

@nilswegner2881 3 года назад

I'm kinda mad that I only just upgraded to gigabit a little over a year ago. Now there's switches that do 100 gigabit for under 1000 euros... Thats so insane, I was totally fine with my 100 mbit equipment before. I mean sure, gigabit brought a hughe improvement, file server wise, but my internet connection still is only 50 mbit so I still have some devices on the old 100 mbit link. :x

@MK-xc9to 3 года назад

Well , this switch is far to oversized for my needs , the Intel Atom C2000 would be another " NO GO " . For my private usage i use the Mikrotik CSS 326 with 2 x 10 Gbit SFP+ and 24 one Gbit ports , passiv = silent and max 20 watts ( i believe ) for 130 Euro . I only need High Speed to my NAS , for the rest like TV , BD player etc is one Gbit enough

@zactodd3144 3 года назад

Point of interest. Is the c2000 resistor fix something that could be done to a switch like this. Have carried it out on a synology unit myself and Patrick did mention the fix, but sounded like it was implemented at the manufacturer level.

@michaltatka 3 года назад

It could be fix by resistor ? (on the CPU PCB board?)

@zactodd3144 3 года назад

@@michaltatka yeah, C2000 series Intel well known for the issue. Got a $5 DSJ415+ (company threw away as it couldn't be trusted). Running 2 years before died, and now running a year since fix.

@michaltatka 3 года назад

@@zactodd3144 Wow :) do you have any links how to fix it? waiting for mine switch, and if it's a 15 minutes it is worth it (if my model have the cpu bug). and anyway I will open it to switch fans to "PC RGB ;)" to make it silent like we done it with Cisco 3064PQ.

@michaltatka 3 года назад

@@zactodd3144 ok received, mine is 2017/2018 :) no resistor needed.

@zactodd3144 3 года назад

@@michaltatka good job. Yeah, the resistor fix is all over the web/RU-vid for the Synology NAS... I was just asking if anyone knew of a fix for the switches. Good to hear you got a later model one where the issues had already been fixed.

@Sbellins1109 3 года назад

WOW 1000dollar for a switch even if it is 100GB for me it's too much ahahha. I've bought a couple of days a go a Juniper EX3300 (48 1gb RJ45 ports and 4 SFP+ 10G ports that can also compatible with 1G SFP) for 110€. It works like a charm (for now only at 1gb because i need the cards) and it have a very nice web interface. I hope that this Juniper switch is not picki with the transcivers and does not require it's own brand for 10G, for 1G it works with off brand once)

@JosephHalder 3 года назад

I'm using fs.com and 10GTek DACs with my Juniper EX2300, it also has 4x SFP+ ports. I don't believe it will be picky. It might throw some minor events in logs, but it works fine. Mine are "Generic", you can get fs.com and others that show as branded "Juniper" to the switch for no additional cost.

@Phil-D83 3 года назад

And I am happy with 4x 10gbe ports on my.home network. Lol

@maxvideodrome4215 3 года назад

I’m good with 1G ports :)

@hariranormal5584 3 года назад

maxvideodrome Same. Modem ==>> Wifi router. End of my network

@michaelloving8004 3 года назад

I'm lookin @ buying this switch seller says it was manufactured 2019 will be safe from that bug ?

@NUCLEARARMAMENT 3 года назад

I can only get the 1.5 Tbps IB routers.

@justjoeblow420 3 года назад

I wonder how hardcore the re-work is to address the bug and if it can be bodged onto an existing switch with a steady hand and some know how. If I had the money to build a proper compute cluster like I would love to do right now this would be a good candidate switch wise outside of the bug with the processor.

@eDoc2020 3 года назад

It shouldn't be too hard. I don't know the specifics of this switch but the LPC bus is often brought out to a 2.54mm pin header. You'd just need to add a single resistor to this header. On EEVblog 1288 Dave did the repair (and explained the problem) on a Synology DS415+.

@tylerlindberg7881 3 года назад

I have two AG9032 V1 Agema Delta 100gbe switches, but I cannot find any way to load an OS on them. :-( Have you played with these switches?

@ServeTheHomeVideo 3 года назад

Not yet :-/

@Thorsummoner0 3 года назад

i really hate how behind the times my 10gbe home lan is :)

@veegee24 2 года назад

When you say "not come back up", do you mean permanently fail? Or does it just need a cold boot?

@ServeTheHomeVideo 2 года назад

AVR54 is beyond just a fold boot. It can usually be repaired via soldering, but it is not a simple fix like swapping a fan/ PSU.

@Dragirek 3 года назад

**Linus want to know your server location**

@ayuchanayuko 3 года назад

Too late, LTT just posted a review of a 48x switch ;)

@Dragirek 3 года назад

@@ayuchanayuko Terabyte Switch master race

@tnaxpw 3 года назад

@@ayuchanayuko wasn't that 4-6 100GbE and rest at 25GbE?

@nilswegner2881 3 года назад

@@tnaxpw yup because 100 GbE Nics are still not affordable nor good value

@bluegizmo1983 2 года назад

Currently in June 2022 you can find this switch on eBay for less than $500

@robertkelly5391 Год назад

If my switch has a manufacture date in 2018 I should be alright, right? I got mine for $450

@ServeTheHomeVideo Год назад

Wow nice! I would suspect that has a fix already.

@KuntalGhosh 3 года назад

Lol and i was thinking 10gbe switch is too expensive and unnecessary 😂

@jfbeam 3 года назад

They certainly used to be. If you can live with optics and/or DACs, 10G gear isn't too insanely expensive these days. (and "n-base" (less than 10G) stuff is getting cheaper) "Unnecessary" is still a matter of degree -- most don't need it, but it can be handy if it's there.

@velcroman98 3 года назад

This would be great for setting up very large computers. The type that consist of racks and racks of blades that make up a single computer.

@KuntalGhosh 3 года назад

@@owowowdhxbxgakwlcybwxsimcwx sfp is a pain the ass for home network. The cables r expensive.

@jfbeam 3 года назад

@@owowowdhxbxgakwlcybwxsimcwx In every way :-)

@someguy4915 3 года назад

@@KuntalGhosh SFP is just the socket, second hand transceivers can be found for ~$10-15 and in a home network you won't need hundreds of them. Fiber cabling second hand or new on FS.com is also pretty much the same cost as decent quality CAT6/CAT6a cabling, just DACs are typically kinda expensive...

@craigcordeiro8530 3 года назад

I have no money and no use case but I want it.

@MrCadilLACI 3 года назад

can you stack these for mc-lagg?

@johndododoe1411 3 года назад

The backbone chip can only do 3200Gb, so at 32 100Gb ports, it's maxed out and can't handle any stacking traffic beyond what can be done via the ports.

@miaudottk9080 3 года назад

Get someone to replace that CPU. With a Zhuomao rework station it's possible. See where you can find a CPU and contact Rossmann Repair Group and ask for the rework price. They have the station, but I have no idea if they're willing to do the job, because they repair macbooks, not servers.

@CCL13CN 3 года назад

They probably just purchased a huge amount of CR2032 mount for some legit reason, and just keep using them for all the products they have decades down the road.

@beauregardslim1914 3 года назад

So much bandwidth on old gear like this and yet major brands are still shipping 1Gbit ports on new motherboards, WiFi 6 APs, etc.

@everythingfeline7367 3 года назад

When the vast majority of internet connections don't get near gigabit (and often not even 100M), there's no good reason for the extra cost for most people

@LtdJorge 3 года назад

@@everythingfeline7367 for LAN

@TacoGrande007 2 года назад

Why are all these high end switches so cheap on Ebay? What is the catch?

@robertkelly5391 Год назад

This generation is at the replacement age. newer tech is faster and more efficient.

@clausdk6299 3 года назад

Maybe... it's time to upgrade from 10gbe to 100gbe at home 🤔 🏠

@marksapollo 3 года назад

100GB for a thousand bucks!! Hell most home networking enthusiasts are busy installing CAT 7 and 10GB networks. And from the sounds of it these are peanuts compare to what the enterprise market uses now. Amazing.

@ServeTheHomeVideo 3 года назад

This type of 32x100GbE switch is still ultra-popular. You can use breakout cables and hit 20x 2U 4-node servers on 25GbE with these switches and still have plenty of uplink bandwidth with a bit of over subscription.

@wiziek 3 года назад

No one should be installing cat7, cat6a is fine or just use fiber. Cat8 cables will be super short anyway,.

@marksapollo 3 года назад

@@wiziek And who is going to spend the time installing fibre network cables around there homes with all the hassle they give, over Ethernet Cat 7 cables?

@marksapollo 3 года назад

@@ServeTheHomeVideo Amazing, thank you for the video.

@chrisb945 3 года назад

this looks much like a dell switch on the inside... in the end... they all seem to be manufactured by a single vendor ^^

@ewenchan1239 3 года назад

Mellanox, I think, had this type of switches in like 2015.

@MrAtomUniverse 3 года назад

Can i dry an egg on top of it?

@chuck2501 3 года назад

looks like the inside of the switch on LTT recently.

@ServeTheHomeVideo 3 года назад

This is a bit of a higher-end unit.

@chuck2501 3 года назад

@@ServeTheHomeVideo yes, the spec is way greater but layout is very similar, I was supposing Dell are one of those purchasers.

@opiniondiscarded6650 3 года назад

Bruh do a video on how to DIY rework it

@martinum4 3 года назад

Maybe ask Louis Rossmann^^

@charlesselrachski34 3 года назад

100 giga-bits? Makes me feel like marty in 1955. Doc all I needs is a little 2.5 giga-bit ports.... ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-f-77xulkB_U.html Remember this moment, We'll be looking back in 2255(when we can haz 100 giga-bit ports) and feel like that video - looking in to the past.

@UpcraftConsulting 3 года назад

Looks VERY similar to my Dell S5232F-ON. I bet Dell is using these guys as their OEM. In fact I see Linus at LTT just did a video with internal shots on his 25gig version and the internals look extremely close. ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-18xtogjz5Ow.html The Dell s5200 series appear to have different switch controller (the pcb below toward the back) Likely Dell using their own design here and it would make sense since they bought Force10 and use that OS10 operating system so they are likely doing their own thing on the control side. A linkedin profile of a Dell employee definitely shows Dell, Sonic OS, and Celestica joint venture as keywords in the profile preview but I'm not logging in or paying linkedin to view the complete thing to dig further. Kind of confirms that Dell s5200 series is basically either an identical or very close offshoot of these. Not surprising though, Dell often takes some generic ODM design often customizing software or at least packaging it as a solution with other dell equipment to spin up a lot of their enterprise product lines.

@georgewright1093 3 года назад

Why is the name of this "ServeTheHome"? Are people using any of this at home?

@ServeTheHomeVideo 3 года назад

George, absolutely. A switch like this that lets someone learn SONiC on relevant hardware is extremely useful to those who are in the networking space. To your broader point, STH is now almost 12 years old so we have expanded a lot. There is a brief history on the STH RU-vid 1 min channel overview. While STH RU-vid is small, it is only a low single digit percentage of traffic we get across STH with the vast majority being on the STH main site. It is kind of like how the Wall Street Journal expanded beyond covering just what happens on a street in New York City.

@Gastell0 3 года назад

I guess you better not do a port mirroring on this

@AlfaPro1337 3 года назад

Will Linus benefit from this?

@ServeTheHomeVideo 3 года назад

I know they are deploying 100GbE/ 25GbE right now, but I think they are using a Dell switch.

@DJ-Manuel 3 года назад

Yes, they use a dell switch, but maybe in a few monthes they releas a video „25Gbit wasnt enought“ 😅

@ServeTheHomeVideo 3 года назад

@@DJ-Manuel I mean... I know they've gotten 100GbE running just from chatting with them.

@DJ-Manuel 3 года назад

Yes, the newest video on floatplane shows off their switch, but from what i can understand they use 100GbE mainly for server connectivity and not for client, not sure if the OM3 cables they‘ve deployed (seen in some youtube story of them) would even be capable of 100GbE, my guess would be that this is the limiting factor for them to do 100GbE as well to the clients (and of course, at the moment they don‘t need it) 😅

@wiziek 3 года назад

Linus doesn't really knows anything about networking to be honest.

@rodfer5406 3 года назад

Holy sh.... !

@majstealth 3 года назад

*cough* 32x100GbE being outdated, we still deploy 1GbE because, "costs"

@user-km7vz2xh2m 3 года назад

0:20 left board from a cheap laptop

@joeldoxtator9804 3 года назад

Interesting, but F.Y.I. you will never need this in your house. These are made for network backplane racks that connect multiple server rack rows together. The only way you could ever come close to saturating the bandwidth of this switch is with multiple racks of server all communicating at once over 10GbE. But then again, there is no kill like overkill.