TWO AMD Epyc Servers for $500! 

Thanks to Maximum Settings Cloud Gaming for sponsoring today's video. Get started with your very own Cloud Gaming VM or Bare Metal Machine at bit.ly/3wxfUuB!

So Quanta will be a fine addition to my list of "never buy or even look at"

"don't use any cpus we don't recommend. also, we're not gonna tell you which cpus we recommend."

Did you notice the CONSOLE SELECT SW (lower part of the screen at 13:13)? Options are BIOS or BMC. I would guess it's set to BMC right now. Maybe switching to BIOS will give you access to the bios that you want.

Put a GPU into each slot. My Asrock AM4 server board has this same issue. You can disable the onboard VGA which connects to the ASPEED BMC. Get a dummy plug into a port on each GPU and I bet it might post

That "dead" memory channel coming back was the ultimate "it works at my place" or "it works for me"...

This is why my homelab is all Dell PowerEdge. Plug the service tag into their support site and you get ALL the manuals, drivers, and information you need, even the build sheet for your unit's factory configuration. The only things they seem to lock down behind an account are their XC series, but those are just rebadges of other models so you just have to look up the equivalent regular model. HP almost got it right, but they constantly break their support pages and practically disowned everything made before the split.

Nice shirt Jeff! 😉 Looking sharp! That's too bad about the BMC! I hope the community comes through!

In the video you tried a Naples CPU. Have you tried a Rome or Milan CPU? The firmware ("BIOS") on those systems contain one or more AGESA packages. Each AGESA is for only one generation. I don't know how Quanta does it (I've only worked with Quanta systems with Intel CPUs) but other barebones manufacturers often will have two AGESA in their firmware. e.g. so you could only use a Naples or Rome, unless you reflashed a different supporting version, then you could use a Rome or Milan CPU. When Milan came out, we had the fun time of having to install a Rome just to flash, then remove it and install the final Milan. First gen Epyc boards didn't have enough flash memory to hold an image with both AGESAs, so they were Naples only. tl;dr: Try a Rome and a Milan CPU to check for compatibility.

This is the sort of content I've been missing. "Bought this... and now I'm drinking..." sort of things.

The Quanta T22HF-1U is a 1U rackmount server manufactured by Quanta Cloud Technology Inc. It supports up to 400GbE networking bandwidth in x16 PCIe 5.0 slots, and has enhanced serviceability with tool-less, hot-swap designs. The server is NEBS compliant for Telco/5G data center deployments, and liquid cooling is supported with up to 385W CPU TDP. Might need some Verizon/ATT server guys to chime in...

This makes me appreciate my old Supermicro servers even more. Standard cables, standard board holes, standard behavior, straight forward BMC (well, as far as bmcs go anyway) everything just works.

What you accomplished was keeping my attention and doing the tech stuff that I would like to do myself. Thanks for the video!

If you archive the stuff on quanta's site now, in a few years you'll be glad you did

Dude likes blades so much his rack features a convenient side mounted blade to defend the internal blades

Some things: 1. Quanta is a vendor for HyperScaler companies. Meaning Google, Oracle, Microsoft, Meta, etc. This is typical of a quanta machine. 2. Are you absolutely sure these blades support AMD EPYC 7601 CPUs? Those are 180W TDP CPUs and are power hungry. 3. What wattage are those PSUs rated for? Typically, PSUs need to be 2-3x more powerful than CPU TDP (180 x 2 or 180 x 3). Then multiply that times 2. You have two nodes. Have you tried plugging in both PSUs? It's possible it wasn't configured for Single PSU use and machines won't boot unless both PSUs are present. 4. Quanta removes documentation due to the fact they mostly only sell to larger hyperscalers. 5. Do the nodes get IPs for IPMI? You have the BMC yes. But each node ALSO gets it's own IPMI as well. Try looking into that. 6. Do you have any storage plugged in? 7. Remove all RAM except for 1 DIMM. See if that makes a difference. That's all I can think of honestly. Oh. Last point. You get what you pay for....

I bought a Quanta Server T42S-2U on ebay and can confirm I feel the same pain :). I was able to install a skylake cpu with ecc and was able to boot the node and use IPMI to reset the bmc password. in my example I was only able to boot once I had both cpu sockets (of one blade) installed with a genuine intel cpu. ES or QS was not working as was only one cpu. If you want to have a look at something on that system let me know I can see what I can do. as for your blade I guess that usb and vga are disabled (which is something a hypervisor would request) so getting into the bios here would be ..... a challenge. As for documentation, the best option would be way back machine. also my system had different names for the blade (which had documentation) and the chassis which had less so try to see what the blade name is and google on that (tip might be a part list of the chassis part number).

A bat'leth on your server rack? It's a server blade to go with your blade server.

CLabRetro here on youtube here has a knack for getting into BMCs, but he's usually getting into older servers.

Always enjoy the videos, success or not so successful, they are all a good watch. I had John Coltrane playing in the background durring the vid. That kinda jazz fits right in to experimental homelab electronics. Cheers

Goddamnit Jeff, you beat me to this! I was researching this 3 months ago to house some of my extra 7451's. If you need, I ended up getting the Quanta QVL for this from the support team. It only shows support for the 7xx2 series, but it does have other information, so I never ended up buying.

IPMI on boot tried to register into dropbox DNS, that's a clue where this system came from.

Shoutout for the spongbob cards. little things like that make me lol

What a great video, sometimes the videos fail, but I am like you Jeff I love to experiment as that is how we learn! I really hope someone in the comments can help you figure this out seems like a ton of compute power to just let go to waste!

President Skroob : The ship is too big. If I walk, the movie will be over.

To reset the BIOS from the BMC command line on a Quanta server, you can try log in to the BMC's CLI and issue the command reset sp. This will reset the BMC module, which in turn might reset the BIOS to its default settings

If the fans are running full speed like that all the time, chances are its not posted. Re-seat RAM, CPU, and find a bios reset jumper. Or could be a quirk that both blades need to be running in order for it to post?

Love these kind of videos!

I have a similar purchase that is going slightly better. I found a SuperMicro 2U/4-node system of the X8 generation, so Intel, older, and slower, but each node has 2 sockets, so I actually have 8 sockets in a 2U chassis, but the intriguing thing for me was the boards all included 40Gig Infiniband (10Gig Ethernet) ConnectX-2 network cards. The down side is the onboard storage. No SATA connectors on the board, so no SATA-DOM. No M.2 slots. No SAS. Each node has access to 3 LFF drives on the front backplane and that's it. _But_ with the 40Gig IB, (I already have a Mellanox switch that supports 40gig ETH and IB) I figured I could probably work out some sort of iSCSI solution that would be plenty fast, but that has turned out to be a challenge. Trying to get a modern version of FlexBoot (Mellanox's PXE/iSCSI boot firmware) onto one of these older ConnectX cards (let alone one built onto a server board) has turned out to be a challenge. I'm probably going to abandon that and just boot from an iPXE ISO and let it handle the iSCSI connectivity, but that's a new development that I haven't had a chance to play with much. I did manage to wreck the pins on one of the CPU sockets, but I fortunately found a pair of motherboards on ebay for a reasonable price. They came with an unexpected surprise... proprietary BIOS. No interesting features, so I tried to flash to the stock BIOS but that was a no-go, so just swapped the BIOS chip from the damaged board and all was well. Eventually I hope to get XCP-NG running on all of the nodes and then mess around with some of the HA features, but I'll cross that bridge when I get to it.

Have you tried to put the cpu on the other Blade? Is bios the same? If each motherboard has its own bios maybe settings are different and you can have video signal on the other motherboard

I have a 7601 and a H11 board and it randomly decides to drop a single memory channel as well. It usually returns to 8 channels when I reseat the CPU. Happens a couple of times a year.

What was that big red button on the board? The one that reset?

Were you able to edit files in the serial connection? (I may not have been paying enough attention when you were talking about it) If so, would it be possible to hack the bmc webui and bypass the cloud login part?

Have you tried to put in a video card in to see if that would display video out and bypass the disabled VGA port?

The BMC has GPIOs and 5 serial ports, one is used by the BMC kernel (you connected to that one) but the other 4 are candidates for the BMC serial connection to the main computer. I would look at the code on the BMC and look for how it controls and monitors the base system: GPIO and/or serial along with the settings should be in there somewhere.

Some people online I saw talking about Dell blade servers said their KVM didn't recognize it. Have you tried direct connecting a monitor? Also one other post said something about using the print screen key to switch active blades, not sure if that's a dell thing only or if quanta used it too or if that even matters if each one has its own IO.

Videos like this reminds me why I run Dell.

howdy. great content. regarding the question you pondered in the beginning : you may have the same allergy I have. I found once I had the first drink, the phenomenon of craving occurred and I became unable to control the amount I drank

have you tried booting of sibling socket in case there is some weird topology with video output wired via specific PCIe lanes?

this is one reason im weary of enterprise hardware, there always seems to be some kind of proprietary garbage involved, sometimes its better, other times its worse.

I havent had experience with this server but perhaps you might be able to get a copy of the bios to download over the serial connection and edit options in with ami bios tool. But i would worrie about flashing it back as im guessing that its checksumed or something of that nature and turn it into a paper weight.but thats from some guy in his shed.

Is there anything preventing you from reflashing the EMMC for the aspeed controller? I had an HPE 12th gen storage server that had ilo crap out on it but I was able to reflash using a header on the board. I wonder if maybe you can't put an open source IPMI that supports that a speed controller on the flash for the out-of-band management card

maybe you're gonna have to figure out which chip on the motherboard contains the bios, connect to it with some sort of pin connector, extract the bios rom, and reverse engineer it, or try to use a flasher (like a SOIC16 flash chip) and try to flash a different bios on it (like coreboot), keep the original bios rom in case you need to flash it back (and perhaps brick the board). But since you have no docs to go off of, this will be extremely hard to figure out.

So glad ive hust standardized to supermicro. Alrhough i dont currently have a use case for blades ive been curious. I do run dual epyc 7742s in supermicro 2u servers for my cpu mining when im not teaching myself how to build ai/render nodes.

I had a server like this before. One think I did note. It will not work with any type of modern monitor and I need to use a old VGA one and once you did this the screen worked and you could see what was going on with the node server. Also type swapping the node around could be a faulty slot.

I’m kind of curious if this OOB manages the system similarly to how a Bluefield DPU works, where you can get into the host system via an rshim console. Have you looked for a running rshim service in the BMC environment? If the rshim service is running, then I would imagine that you can use minicom or similar to connect to the rshim device in /dev.

Not sure if it would work but I know some dell boards even when you didnt have the idrac unit in would still support command line IPMI commands. May apply here for getting passed the license issue

You could try to use ipmitool remotely to directly change settings with your credentials or to get access to SOL to get access to the BIOS of the board

What about adding a graphics card for a video output other than the disabled VGA or adding a NVMe that is already preluded with an OS that has a static IP set to see if it will boot to the OS.

I resemble the remark about being "crazy enough to have a server rack in the garage"

did you try putting a gpu in? Also kinda wonder how hard it would be to use the serial to modify the ipmi web portal to work locally without cloud subscription (or to reset the password directly in a passwd file or however its stored).

Dumb question, did you try with both power supplies plugged in, I know with some of our gen 10 HPE blade units they won't post fully with out both psu plugged in

You might like to try a pcie video card and get screen output from that but after getting into bios you can set prams for vga output. Good luck!

Thanks for informing me that Quanta should be 100% on my avoid list.

Thanks for sharing! Now I have similar problem with Brocade 1020 network card. I run it on Arch Linux but most of my SFP+ modules are failing on it, even 8G & 10G Brocade modules. So now I am digging and trying to find firmware for that thing :(

Well done 👍 you have brought this to everyone’s attention . I know this type of hardware has a life span 🤔😞 But borking it so it can’t be used. Right to repair is a thing Right to repurpose? Is that a thing? Those chip are awesome 😎

In similar servers, running with just one PSU can stop it booting. Try plugging both in.

Could you gain anything by adding another video output on the pci slot?

In addition to trying a GPU card or doing a clip chip reflash of the BIOS, could go old school and try shorting the BIOS nv chip while booting should go into defaults. Once you're visible/in the BIOS you can lift the short and save new settings

16:33 did you try something like a GT1030 in that PCIE slot, small chance but it might not have a GPU whitelist

I had a 1U server about that long from supermicro, what was weird is that it was a dual node, but it wasnt a blade, it used blade style dual socket boards, but instead of a power finger like a normal blade board, it used a 12VO 24 pin(and no CPU 8 pin) also, it had two of them, and while the manual said not to chain them, you could totally chain 2 boards from one PSU if you are using L series processors rather than X series, but keep an eye on that RAM, if you used ECC RDIMMs rather than UDIMM just 2 sticks could use more power than the L series CPU and with 12 memory slots, you could have ~250w in just RAM(edit on a single board) A rats nest of cables. odd considering how much room was inside this massive chassis with these tiny supermicro X8DTT-IBQF boards. It was also wider than a normal serve rack, so i ended up pulling the boards and PSU's out to put them in some long ITX cases. like the coolermaster 130

Try putting a live OS that will boot headless on various media USB, nvme and see if it boots and is visible on the network.

Probably already thought of this, but maybe use the Way Back Machine to recover is documents?

OK you are logged into the Onboard KVM that switches between the blade servers. So you have not gotten into the Blades.

I'm actually tempted to buy one of these and a used 64-core Rome CPU just for the (relatively cheap) compute power to replace two nodes of 2x E5-2695 v4 CPUs. But it seems like a gamble with the lack of support and documentation. The difficulty with using it is probably why it's cheap to begin with.

1) So I tried looking for the manual -- no cigar. I did turn up that the model of the motherboard itself is known as "S5H(F) MB". You see the s5h referenced a bunch of times in your dmesg output and console logs. 2) The other thing that I noticed was that right next to the big red reset button, there's a dip switch (SW1) which says "CONSOLE_SELECT_SW". The picture, in that area, doesn't have enough resolution for me to be able to read the silk screening on the switch itself (I think that I see a '4', but I can't tell if the other thing is a '10' or something else.) So, maybe you can try playing with that to see if that might change stuff. 3) The other thing that you might also try is do drop down to only one stick of RAM. That might be a challenge because without the manual, you don't know which is the primary DIMM slot, so there is a chance that you might not get very far with this. 4) This is why I buy Supermicro. Yes, they might be more expensive, but their stuff well documented. 5) I think that there's been discussions about Supermicro paywalling their IPMI behind a subscription, but it hasn't happened yet. But yes, this would be very frustrating.

Are POST cards still a thing for modern PCs? That would be the best indication of whether or not a system is trying to boot.

Very interested in the end of this story!

So Quanta requires that your IMPI network has access to the internet? That seems like a security risk waiting to happen. I'm not sure if this is standard practice or not, but if I ran a data centre and had the option of IMPI being intranet-only, that's how I'd do it.

Could that blade be bad or flakey? Maybe try the proc and mem in the other blade for funsies?

Great timing, i have one on my ebay watchlist but haven't pulled the trigger. Did you try reaching out to the seller? They might have "some" information. I'd try throwing a gpu in it, see if i could get some video on that at least. Outside of that, i'd go down the.. is the rest of the hardware good route. Do I have the right mem slots, etc. I saw a picture on the outside of the case which i think outlines some of that info.

Get a SOIC clip and pull the ROM, for posterity, and also open it up with a BIOS tool to change the default options. Then perform a reset and it will run with the options you set. Also as others have said, throw a vidoe card in it. Waiting for round 2.

I would try to install a video card in one of the PCI slots since it's a new device It wouldn't be disabled in the BIOS.

If/when i become God-Queen-Emperor, my first decree will be that beer shall be free. My second decree shall be that lights out management requiring a cloud login shall be punishable by surrender and liquidation of all company assets to charitable causes.

Also, try installing PCIE low profile GPU to output video and go into bios settings

7:07 do you know that "special" AMD screwdriver that limits torque applied to socket screws is actually made so that users won't strip socket screws, and not to apply the specified force to CPU? I've built more than a dozen of AMD servers using nothing but standard screwdrivers, you just need to bottom out all screws and everything will work fine

37" is long for a server. I thought my SuperMicro 6047 72-drive model was bad - but it's only 35". On the other hand, it's a 4u and a LOT heavier - even before I put drives in it!

Oh God, i found the same one AT AN EVEN BETTER PRICE!!! Man, $299 is tempting even with all the failure. Does anyone know if Quanta vendor locks their cpus?

Did you try the way back machine on their website?

So excited to see this vid drop. Been looking at these for a while given current epyc motherboard prices. Shame that they act like scumbags like that. They clearly arent interested in something they cant sell the service for anymore.

My simple thought, which is by no means a solution: use as few RAM slots as possible in the beginning to minimize problems.

pull cmos/bios batt for a bit?

I really wish they did these in a 2u form factor stacked so I can put them in a narrower rack.

*Thumbs Up* for a good Guy - i do same stupid stuff at home, so Fun to watch 🙂

Probably a custom board with a bios that supports only one or two specific cpu models. Ppl keep mentioning the 7351. Seems like the easiest way forward.

Maybe try an EPYC 7352 as those are on the list from the Qanta spreadsheet and have similar performance characteristics and prices as the 7601.

I might suggest contacting a reseller like "the server store" or something similar to see if they have documentation that they would be willing to send you. If nothing else you might get a hard copy.

Hello, I am not sure where you got this from but It could of been from our store. We have the original processor that came with this server. We have not found any other processor that will work with these besides the original ones.

you tried to put a nvme/u.2 with linux/windows installed, and configured for remote access. if post and boot mayby you will access via network.

Hardware locked behind a subscription should be illegal. There is no reason for it. Also it should be mandatory to open source your software/firmware when you end of life a product. If you aren't going to support it let other people do it.

10:14 klingon bat'leth ... just in case .. always be prepared..😎

Your tech mojo is Jedi level, when a Padawan is ready a master will be found. To begin my training, I will subscribe and ring the the bell 🧐.

3:13 That was a faaar fetched reference. One could call it a pizza plate. For the hut

what trackball are you using?

Did you try popping the cmos battery and shorting the terminals?

Did u try an add-on pcie gpu/vga card?

Thanks for the info. I'll never buy Quanta. Wow, imagine removing all docs! What are they thinking!

We've seen them opening a package with a Klingon knife, now we see that he also has a Bat'Leth (spelling?). Makes me want to ask him which series he prefers.... DS9 here....I have fond memories of watching it with my mom. A local station with cycle through TOS, TNG, DS9, and first season of VOY before starting all over. Came on sometime after the news went off at 10:30pm .

7:24 Not often you see servers with T-topology that doesnt matter if you use the furthest/closest slot in a channel, good to see. Though in all honesty, unless you're dunning DDR4 4000 it shouldnt matter much anyway even if its not T-topology

The login credentials for the BMC over Serial or SSH are NOT the same as you would use in the Web UI or over IPMI. And Quanta does not require special licensing or "cloud login" to access their BMCs. The default IPMI login is admin/admin, have you tried that?