Of course. It was not really the focus, but we wanted to show both a specific upgrade path (e.g. the most popular Xeon SKU Supermicro sold in that generation to a new upper-midrange part.) Still need to have the broader market context as well.
I swear to god, you are one of the best IT educators on RU-vid, along with Level1Tech. Very concise, articulate and inspirational. Thanks for these insights.
I agree, Patrick shows what its audience wants to see. Regarding server hardware, for me it's Phoronix for written reviews and ServeTheHome for video reviews.
Based on my experience trying to prototype & run generative AI services on AMX on a sapphirerapids chip, while it does better than AVX512VNNI on icelake, it did not feel like the AMX accelerated software was sufficiently matured and that the AMX acceleration was not being taken full advantage of, since when adjusting for disparities in cores vs the compared chips, AVX512VNNI on zen4 was... not far behind. Another VERY annoying problem with AMX is that it is not officially supported in VMs - and true to intel's documentation, it flat out doesn't work in most VMs, only some specific versions of specific VMware hypervisors released in Q3/Q4 this year have support for it. Does emeraldrapids solve the slow clock ramping that sapphirerapids is saddled with as well? When going from no load to max expected load, it would be overloaded for a while due to the cpus taking a while to ramp up to reach their sustained clockspeed (which is sort of the opposite of normal behavior, where normally clockspeeds will ramp down from an initial boost speed)
I have not tried using AMX myself, but QEMU+KVM does seem to report AMX support on SPR. FPU state also reports the correct size (with AMX Tile Config/AMX Tile Data being available). Though AMX does seem pretty half-baked for sure... (looking at all its erratum)
Great video. Disagree about it feeling like an ad. I think the comparisons are relevant for the organizations I’ve worked in. And you mention that they provide the hardware.
Yea not much you can do other than disclose. I mean it is not hard to figure out what 3-6 year old systems are deployed and coming up for refresh cycles.
@@ServeTheHomeVideo agreed. And if I’m cross shopping from AMD, I’ll go look at the STH website and videos to get information about them. AMD does have a max core cpu advantage but for the “mid market” segment it seems like intel has some competitive offerings from what I’ve seen. I think you guys took the right approach here personally. And if anything you guys err on the side of being extra transparent so I’ve never doubted your commitment to providing unbiased information regardless of what hardware is provided and by whom.
Thanks for the great review. EMR looks like a solid upgrade even for those coming from Ice Lake! (Though we're waiting for Granite Rapids for our next upgrade). As a current SPR workstation owner though, I really hope they manage to find a way to somehow make the reduced PCIe lane in EMR works in a workstation. (EMR only has a maximum of 96 lanes on die while SPR has a max 128 with 112 lanes enabled)
To me, the cool part about an EMR desktop would be the lower idle power consumption. 80-100w is huge. I do not think many folks are going Ice Lake to EMR just because Ice was launched in early Q2 2021 but started to ramp volume in Q3. 9 or fewer quarters is short when many are pushing servers to 6 years/ 24 quarters of use.
@@ServeTheHomeVideoCompletely agreed, that improved idle power consumption is huge for workstation. Which is why I I think it would be sad if they can't make EMR works on existing W790 boards due to reduced PCIe lanes (as they're physically wired)
There is unlikely to be workstation products, right? AMX might be the only affordable option to run really large language models locally. Can't afford several RTX 6000 Ada or A100. And Intel isn't selling their Gaudi2/3 or Max GPU for workstation either. (is this low power idle modes again?)
@@ServeTheHomeVideo yeah it was hard to pass up for 80 USD. It's gonna be a great project upgrading and tinkering. But it sent me down a rabbit hole reading up on all the differences the xeons have fro the I series or the older pentiums and celerons.
Great review STH. I love how the comparisons were between the volume SKUs most people will be buying. Great stuff! What Intel has done with Emerald Rapids is really impressive. Really excited to see what they'll bring next year with Granite Rapids and Sierra Forest!!
Thanks! I think most folks in the industry know that I am extra excited about Sierra Forest. That is part of the reason we have done so many E-core only mini PCs.
How about a Rome vs Siena comparison. As a home user having high clocks for single client access but also very low idle power draw are big things to me.
Diving into the fascinating comparison between the Gen2 Gold 6252 and the flagship Platinum Gen5 8568 is truly captivating! Notably, at the time of release, there was a substantial price difference, with the Gold 6252 priced at $4263.00 and the Platinum Gen5 at $6497.00. It's crucial to recognize that these models cater to different needs. For a more precise comparison based on price points, exploring the 8568 versus the 8268 or even the 9242 might be more fitting. However, the notion of relying on those visually striking 3 to 1 figures remains somewhat uncertain.
It’s kinda funny to me to see these just now being reviewed..I’ve been inspecting and refurbishing a ton of LGA-4677 dell boards. And I’ve been wondering why I can’t find a lick of documentation when I search the part numbers.. it makes sense now, and it makes sense why all the CPU’s say Intel confidential on them💀
Out of curiosity, if you were running on board AI accelerators to deploy generative AI, and had a hard power limit (absolute power, not efficiency), are there any server platforms that come to mind with support for around 356 to 512GB of memory, and a sub 200 watt total system power draw?
Are you just thinking AI? For EMR I think that would be pretty tough. EPYC 8004 can get close-ish, but does not have the onboard accelerators. With AMX, you end up wanting more cores since it is more like AVX-512 versus a standalone-ish IP block like a NPU or the QAT accelerator.
@@ServeTheHomeVideo Haha, that's a fair point, and thanks for the response. I'm specifically thinking about large languages models deployed in CPU (via cache-line optimized linear algebra libraries, similar to LLamaCPP), versus dedicated dedicated accelerators. My suspicion is that in a lot of cases you might be able to get to a memory bottleneck either way, in which case while there is a difference to an extent, you might actually lose more time paging in and out to memory than in dealing with CPU cores (particularly with AMX) *or* accelerators. One somewhat wacky option is a person could tensor parallelize the models (given sufficient RAM) on a consumer CPUs, (on a low power server platform) possibly via OpenMPI, and it would be fairly easy to hit a quite harsh power limit as I've listed here, but there are software limitations so I was hoping someone else had come up with an idea I had missed. Either way, this isn't a speed-critical application, but an accuracy dependent one, which isn't as common an issue in the space so I've been struggling a bit to find the right match. Regardless, cheers, and thanks for the content!
@@MarkBarrett Ryzen X3D beats out all current gen Intel CPUs, and not to mention using half the power consumption in Gaming. On the Gaming side Ryzen wins out, but on the Productivity side, Intel wins MT perf/dollar (AMD doesn't care enough to increase Core Count on R5 & R7 products).
@@rezaramadea7574 Intel still wins on 1 or 2 cores, which "key threads" are most important. I'll admit, nothing I know of can beat the Threadripper on multi-core.
@@MarkBarrett Nope, you're talking about Gaming, X3D will still destroy intel, on every Average Metric possible. 1 core or 2 cores on "Synthetic Benchmark" is not Gaming.
Thanks for including more color on the charts, it gets hard to figure out which CPU is the one you're comparing when they're all black and blue. The green really makes it pop and easy to see.
I lost you for most of it, that's ok I had no interest in the subject, I do like the enthusiasm though. Off topic question ,for your video editing, did you use some form of Vegas?
I love watching you guys and level1techs but the softest of kid glove treatment with this release left me a little disappointed. From all the sources I can gather information from (until I can get hands on) it seems that while these aren't bad and probably a step in the right direction given AI isn't going anywhere they aren't all that good either. Time will tell I guess and I otherwise appreciate the overview and comparison.
I am not sure what you would want different? We showed the EPYC numbers including Bergamo/ Genoa-X. This is a decent refresh upgrade to the same socket as SPR. Starting tomorrow, I would always pick EMR over SPR unless I wanted the Xeon MAX.
I really want an ai inference device but I don’t want to go the server route. It’s a bit expensive and too power hungry. The m1 ultra showed that it’s possible getting decent generative ai inference speeds using an APU. This is my backup plan if Intel or AMD doesn’t get their new products in order for ai. I’m waiting for Intel or AMD to catch up because I don’t want to be stuck in Apple’s closed eco system. I heard that AMD strix point will have 50 TOPS for ai calculations. I might get that or Qualcomm’s elite x (if available with software support) and pair it with a lot of ddr5 ram and maybe a Rtx 3090 worst case scenario. I reckon I can get decent response time for very large language models this way.
@@ServeTheHomeVideo I saw benchmarks for the m1 ultra and I’m really tempted to get it. My main issue is that I can’t provide more memory to it. Although they claim it’s 128gb, not all of it can be allocated to the gpu. There’s also no upgrade path for this device. After AMD strix point there’s apparently another one coming out called kraken. I’m not sure if they will come out for the desktop market but I hope they do. Either way, there will definitely be some Chinese Frankenstein computer coming out with these chips for the desktop market. Edit: I also worry about repairability with Apple. If one thing is broken then the entire thing is likely wasted. Whereas the diy market has replacement parts for itself.
Every new Intel release "oh look they're catching up", next AMD release "oh wait no, no they are not". Sure, if you are lazy and bad at job then you will keep picking Intel and get stuck with worse performance year after year hoping for a miracle before the next platform change.
I believe Xeon 4th gen MAX series has 64GB of HBM2 cache, you dont even need to have ram in the system, you can load the OS straight into the cpu’s cache
nobody should have any confidence with intel at this point - full stop - intel has been losing mkt share in server mkt segment for good reasons - the trends will continue - they fail by large margins in both efficiency and density metrics and these are important - btw: i was not paid to say this unlike patrick who was compensated
@@ServeTheHomeVideo yes good point. Intel is obviously desperate to keep those customers seeking to upgrade from the stagnation era of intel (2017-2021) from switching to ARM or AMD. AMD has an advantage in that they can offer a complete ecosystem, with X86 based servers or clusters with GPU acceleration for AI / massive parallelization with MI300 or FPGAs with Xilinx. Intel has failed to make their GPUs competitive with AMD and Intel.
This video seems like a paid ad. Comparing it to the same old chips and no mention of their direct competition Patrick? Really? Seems like you should disclose if you are getting paid by Intel/Supermicro
We addressed all of this in the video AND have EPYC numbers in the performance section. In the 3-5 year refresh cycle, 97%+ of folks are upgrading from 1st/ 2nd Gen Xeon. We already did Genoa and Bergamo/ Genoa-X videos. Also, in every video we say any company that provided hardware or anything is a sponsor, but nobody gets to see these videos before they go-live.
Not only that. We also have to disclose that Intel let us talk to their CEO and one of their VPs. All disclosed, and we marked it on YT as being sponsored because we got so much help. We do not let anyone review these outside of our team before they go live, and that is true for every video for STH.
We have multiple generations with Genoa, Genoa-X, and Bergamo in the charts so I am not sure what you mean? We have videos on those CPUs as well for a deeper dive on them
@@ServeTheHomeVideo You also did disclose this sponsored and made it clear what the focus of the video is. So I won't say there is any deceptive intent going on here, but I must say despite all that and while I generally love your content and your passion, in this video it felt kinda like looking through Intel marketing material to find something to be passionate about. Pitting this gen against a multiple gen older one that was, let's say on the more mediocre side, is certainly a valid perspective for those on an Intel upgrade paths - but also a very convenient choice for a first look, as it puts this product in the best light possible with big looking multiple X increases, while giving an excuse to ignore the competition. Yes you did show the Genoa/-X numbers. But that just made it more awkward to watch. Because you show on screen that the fancy new parts you're excited about are outperformed left and right, but don't even comment on that. Instead you go on praising how much consolidation you get - not wrong, but pretty weird when there are unaddressed numbers on screen that kinda show the competition had even better, denser options for that half a year ago already.
@@Hugh_I I guess the challenges are that a lot of folks look at it like "who has the most per-socket performance." Really, how the majority of the market buys is they buy a much lower core count than max. Also people tend to buy Xeon if they already had Xeon. To be clear, in 2019 AMD took the max performance per socket crown for general purpose compute and never looked back, but they do not even have 50% market share over four years later. Just so everyone is clear, I pitched the idea of doing Cascade Lake to EMR because it is such a big migration cycle right now. Nobody gets to review our content before it goes live (other server review sites send articles to vendors before publishing, even when they do not say it is sponsored, as an example.) Still, these days, we cannot be buying $50-100K of hardware for reviews so we need support on getting the hardware.
Sorry for changing topics, but I had a quick question regarding Project TinyMiniMicro. I've seen that these HP Minis only support 64GB while you fitted one with 96GB. Did you run into any problems with it?
