Twenty Four 100Gb Ports for Under $300!? 

I'm a software engineer by trade, and I have to say Wendell is like the Rosetta Stone for me learning about Enterprise tech. I have never been exposed to it professionally, but I love learning about these things and how they might be brought down to the "mere IT mortal" level. It's really a treat, thank you for sharing your curiosity and imagination with the rest of us!

I still have nightmares and PTSD from troubleshooting token-ring networks.

I support two clusters which utilize OPA, and it works great. Much easier to setup than Infiniband. On one cluster, we use it with a BeeGFS clustered file system to achieve really high write concurrency. The real reason it didn't take off like it should is Mellanox cut their prices to compete for the first time ever. At that point, most IT shops didn't want to take a risk on the new tech when they could get the proven solution for the same price.

Never fear Wendell we actually love you talking about Computer and Networking History there are always snippets of useful knowledge. I still remember when 2Mb Broadband was cutting edge and a home Modem was just a small box with blinking lights and a single Ethernet Port, Things have come a long way in less than 30 years.

Back in school one of my projects was to set up a token ring network, they weren't really relevant at the time, far from it, but its something we were tasked with

I've never used Token Ring but I recently learned that it is fully software-compatible with Ethernet. They even made transparent bridges between the two standards. The magic of them both being part of the IEEE 802 family.

Did you know token ring is back again? Look up 10BASE-T1S. It's an industrial standard, for sure, and a nifty one at that, but it's still token ring.

We know what happened to omnipath. Intel had a CFO running it for about a decade.

Token Ring still has significant application. Most of us will never see said application, but it's there.

Don't forget Arcnet! There were dozens of us!

Wendell, these are the videos I LOVE! I love all the wayward research that goes into making this gear work. This is the true homelab experience getting complicated cast off enterprise gear hacked together to run at home.

Cool kit! Couldn't really use it in my homelab, but nice

Just picked up 4 of these, Would love to write some code to pass enormous amounts of data at very low latency. Thanks for the video, definitely another one of those cheeky things to play around with.

Token Ring! Try spending 3 years cutting out single strand fiber where ever you see it, then talk to me about "forgetting Token Ring".

Would love to see a ceph cluster backed by these cards to speed up VMs during operation and not only while migrating ;)

I don't have anything of value to write here. Please engegment gods accept this offering.

ahh good old token ring, those were the days. ohh wait damn I just aged my self there, now I feel old.

God damn it Wendell, I didn't need another colo project 😅

woohoo. I literally picked one of those up out of the tech discard bin at work and used it as a prop to talk about networking, and transceivers once. I was tempted to hold on to it for future demos until I looked it up and walked it right back to the bin 😛 Don't worry, it wasn't thrown away, it was probably sold off. I never did figure out who we sold that stuff off to though, because I'd LOVE to have a look at their auction pages.

Is OnmiPath related to QuickPath Interconnect (QPI)? This kind of tech could be used to produce motherboards that would be just child numa nodes to remote master numa node.

There was another networking system that was popular before token-ring and ethernet, and that was ARCnet. I did a lot of Novell Netware installations with ARCnet.

Watching this as I am 'upgrading' my EOP Dlink adapters to AV2 1000 from the AV500's 🤣Peak speeds of 8-9Mb/s which at least now matches my mediocre Internerd speed......

I was always fascinated by token-ring, seeing this "relative" technology is super interesting.

Well those Omni Path PCIe host adapters are pretty cheap compared to LAN, can we use two of them to setup a iSCSI NAS (without a switch) ?

Now I have to do research on OmniPath.

I have a project where we took all the star viewed by the Hubble telescope and put them in a 3d unity game connected to a 1tb mysql database. not sure if you would like to help with that. but I have some of the hardware that we need.

OPA isn't *technically* dead, people are still building systems with it (I know, because my company buys them). In any case, from an application PoV it reaaaally doesn't matter if it's IB or OPA.

Looking great Wendell.

Wendel just casually expanding his home lab

Token ring is NOT forgotten.

4 ports at 100Gb with suitable cards in my various computers for under $300 is interesting to the point of drool. 24 ports that refuse to talk with anything I own is cool but not compelling.

I miss the good ole days of coax, bnc and terminators. Ok, not really, but we did have some good fun on Doom and Rise of the Triad!

Looking GOOD Wendell!!!

Please more videos on those projects. I want to learn more

1:19 - *ATM has entered the chat* 😅😅😅😅😅

It would make sense that the FPGA is still configuring all its internal logic & RAM from SPI flash memory when the system scans the PCIe bus and tries to negotiate the links.

Token Ring. Now there's a name I haven't heard in many years. \ObiWan

"I've been buying things on Ebay again... " You and me both... (just that I don't feel like mine are a black hole of time, I've just been buying enterprise SSDs)

Thank you Wendell, very interesting.

I learned more about how modern high performance network architectures work in this one video, completely high out of my mind, than I have in the past five years of watching hundreds of hours of actual on-the-job training for Azure, AWS, GCP, and modern cloud solutions. Jesus Christ.

In the beginning there was ethernet and token ring!

Token ring old? Hell my first home network was using arcnet!

So about that thunderbolt networking video coming "soon".....🤞

So, can it handle ipx/spx routing?

Alas - Token ring is not forgotten! I've actually replaced a token ring network back in the day :D.

Here's a question I think you have never heard. Which motherboard manufacturer has mastering grade audio converters on board? I don't think there are any, but I was hoping that you could find out what converters the various manufacturers use for each motherboard. Many builders have been marketing their PC builds as best for music production, so I thought it was only fair that we be told what converters are built into onboard audio. Perhaps manufacturers will begin creating offerings for music creators.

I recently bought some fiber. After making sure to match wavelength, and intensity, having to figure out if the hbas were cut at an angle or not...

Noticed the bar code scanner. Are you using a MAC address to assign ips to these servers automatically with the scanner? What are you using for it?

Ohh, my... it does sound a lot like Infiniband for sure.

It would be awesome if these worked in Windows! But like on the forum thread, I wasn't able to find drivers :( (at least by searching.)

Token ring.. Actually tried that.. Never going back

why does the middle on the omni path poke out like that instead of just a normal squar ?

Token ring is kinda similar to CANBUS in cars now a days... Its still there. Kinda.

9:06 Darn, I was thinking about doing that after the video since I missed out on the Optane deals.

Fk man its such a privilege to born in a western country. This tech will still take 10 years or more to come to my region at such reasonable prices.

seems almost liek they need a fpga or dpu for the pci adapters

Hi, could you please make a build with the fractal terra case. Please 🙏

I came for the shiny... stayed for the history lesson...

Please-please, make a video on infiniband!

Wendell might need a new set of clothes soon!

The video is showing up really dark on my iPhone 14 pro max in the RU-vid app

I just love this channel. 😎

Does it work under OpenIndiana? My home Nas runs OpenIndiana and will do so for the foreseeable future. It doesn't have the disk speed for 100gb but maybe one day it will be NVME or something. (24 hdd direct connected to 3 lsi 2008's. With an IvyBridge-E 8 core xeon).

heh, we had those in our hpc cluster.

Would that be CDN$300?

Will it work on BSD? Unix?

Did I pick up on the possibility of 100Gbit back to back connection for under $100 all in? (Can I use two of these direct connected in my homelab?)

Maybe it would make a good video to put out a parts list for the fiber home user? I will be getting 2.5g fiber soon and been looking at parts to upgrade my small network. i dont know anything about fiber... does 2.5g even require fiber inside the home?

I love your videos, but a lot of the time the content is way over my head. This is a great example. I have used Alien though to convert packages. It really comes in handy when you need a .deb, and all you can find is the .rpm.

very interesting stuff

Thanks for the vid.. now I want it.. not sure what I'd do with it but tis wants it... Even if 10gb is more then "enough" hehe

Hope you have a Remote Server Cluster And a Security Block Between This Cluster

They are all upgrading to fibre channel .

Wendel, could you check out the SABRENT 4-Drive NVMe M.2 SSD to PCIe 3.0 x4 Adapter Card [EC-P3X4]?!

Would hate to see the CPU usage😁

The reason it's cheap is because it's being abandoned, once fiber channel caught up there was no need for proprietary standards and manufacturers abandoned it. The risk of adopting it being that stuff will fail or be incompatable and you won't be able to get replacement parts or support.

I remember token Ring

Oh omnipath... I keep waiting for our omnipath to age out... Its another victim of intel trying something and then killing it

Slowest backplane ever

I don't have any experience with Omni Path (but I do have 100 Gbps Infiniband deployed in the basement of my home), and this just sounds like it would be quite a massive headache, trying to get it do what you want it to do. IB has been around a LOT longer than Omni Path (defacto standard system interconnect for a LONG time for supercomputers), and so, overall, it's a vastly more mature technology, and it's gotten to the point where a lot of it "just works". If you get the Mellanox ConnectX VPI cards, that's even better because you can run the ports either in IB mode or in ETH mode (which is awesome). (Or you can stay with IB mode, and just use IPoIB.) I can't imagine Omni Path being nearly as capable, and I think that the lack of support, especially from the software side of things, is at least a part of what's holding Omni Path back. (i.e. with OpenMPI - you can tell it to use the IB transport protocol) My LTO-8 tape backup system uses said 100 Gbps IB connection back to my main Proxmox server, even though, at best, the LTO-8 tape only writes at an absolute max of 360 MB/s. IB is more expensive than this cast-off Omni Path hardware, but it is, based on my experience with IB, a LOT easier to get IB up and running.

After watching video notices title does NOT SAY ETHERNET ports

Wendell forever

I made the mistake of buying 4 of those pcie cards last month when I already have a 100Gb Ethernet switch and QSFP modules Stupid Intel trying to recreate Infiniband but proprietary

yea so nohoep ..all si shit on a cabel ..nods ..no a ok amen fro a servar ..and tha raith pce expres ..yea ..tolty nasti fail !

Doesn't the moral of this all drill home that ethernet isn't the BEST technology, but it is better because everyone has it?

Tolkien Ring.... One ring to rule them all?

pizza box

Omni-Path? Downvoted!

AND THAT`S WHY I WILL NEVER USE AZURE !!!

Microsoft technology, doesn't do any security, just passes messages. Reminds me of Win98. LOL

discord?

Hi, thank you for this video. Can someone tell me ebay customer service phone number ?

aida64 or aida cruize ship Dx # angry

click bait shame shame

When you live in a country without Amazon and Ebay... Paying $800 for 10 Gbit is like... My path atm.. 🥹