I worked on this CPU! Specifically the bridge dies between the CPU tiles. I figured I'd share some fun facts about those CPU tiles here for you guys: Each CPU tile has 15 cores. Yes, 15. The room that the 16th would occupy is instead taken up by the combined memory controllers and HBM PHYs. There is not one continuous interposer. Instead, each CPU tile sits on top of EMIB "bridge" dies as I call them. this strategy is more similar to Apple's than AMD's, or even Meteor Lake's. This is because Sapphire Rapids is so enormous that it exceeds the reticle limit of the machines that make normal interposers. There are 4 CPU tiles, but and 10 bridges. The tiles each have 5 connections, 3 on one edge and then 2 on the neighboring edge. 2 of the tiles are mirror images of the other 2. You can get a diagonal pair by rotating one about the center axis 180 degrees, but the other 2 have to be mirrored to keep the connections in the right place.
@@DigitalJedi thanks for you insight, always nice to see engineers talk about the work they do ;) I'm busy playing Starfield and porting it to my C128, why because I think that Z-80 will work as a nice co-processor to the 8510 CPU, ha :p
Considering new AMX instructions and all that bandwidth afforded by HBM, it would be very interesting to see benchmarks for AI tasks, like running stable diffusion or llama models. How would they stack up against GPUs performance wise, or power and cost efficiency wise? Would be very relevant in current datacenter GPU shortage!
One of the nice things about "Serve the HOME" (Emphasis on HOME) is that we get to have a glimpse to see what we'll be running in our HOMES as low end servers in 30 years.... I'm 5 minutes in and I can't imagine the cost of those things when they come to market, not to mention the REST of the hardware costs.
Without the RAM slots taking up width, you could pack a HBM-only server incredibly dense - maybe 3 dual socket modules across a 19" rack? Not many data centres could handle that power density, but it would be pretty neat to see
This one took a long time. Partly due to the complexity but also moving STH to Scottsdale and doing like 40 flights over the summer. I was hoping to get this live before Taiwan last week.
Damn, that localized memory is incredible for SQL instance/shard, web server cache and so much more. HBM memory runs at lower wattage than DDR memory, with significantly higher bus width and lower frequency required to achieve high bandwidth (afaik). p.s. Didn't show the bottom of it even once =\
On the topic of 1000W power draw, I believe these are the same CPU power delivery topology that Intel showed a while back during some of the lab tours (e.g. I believe one of der8auer's videos in the extreme OC labs showed this off), where you have a relatively small number of VRM phases on the motherboard providing an intermediate package voltage, followed by a massive number of on-die power stages (100+) parallelised into a huge segmented polyphase buck converter, which helps reduce ohmic losses and PDN impedance by moving the regulation closer to the point of load on the die. The combined continuous output current of the on-package converters appears to be 1023A, logically limited by the number of bits in the relevant power management control register. This kind of current delivery would be unworkable with a traditional VRM, but since the phases are physically distributed around the package the average current density is heavily reduced.
I hope this is something that filters down to consumer parts. Especially for APUs with integrated graphics we are pretty clearly getting to the point where they are being limited by memory bandwidth. The Z1 extreme with 8 CPU cores and 12 GPU cores is only about 5-30% faster than the Z1 with only 6 CPU cores and 4 GPU cores. These two chips are meant to operate in the same power limits and are running the same architectures. Given all that you would think that something with 3x as many GPU cores would be much faster but that just isn't the case and it is my guess that it is probably due to memory bandwidth. GPUs are bandwidth hungry and there is a reason GPUs pack their own specialized memory. I wonder if combining this with an APU couldn't let that iGPU stretch its legs to its full potential. Here is hoping.
This issue is that HBM is very expensive, and doing HBM right means a pretty much ground-up design for your chip to not only fit in the PHYs for the kilobit+ bus, but also the differences in controllers and possibly dual controllers if you still want DDR5 options. I've worked with HBM, and when you get to the class of connection density it requires, you need to spend the big bucks for a silicon interposer. Radeon Fiji did this, Vega and VII, and The Titan V come to mind. That is a whole massive die you need to make and then stack at least 2 other dies on top of. An HBM APU sounds awesome I agree, we even saw a glimmer of it with the i7 8809G, which had a 24CU Vega MGH GPU and 4GB of HBM. The more practical approach for right now though would be something with a dedicated GDDR controller, even just 128-bit 8GB would be plenty, as that is already around 288GB/s of bandwidth you aren't fighting the CPU over.
@@DigitalJedi Yeah I know there are definite issues. HBM has a 4096 bit bus which is gigantic compared to anything else and is why you need the complex interposer. Intels EMIB looks interesting and may help in that respect but we will have to see. Personally I would not have the option for additional DDR5. This would be replacing it. Many systems already use soldered memory so this would simply be an extension of that. I would dare say 90% of consumers don't bother upgrading the RAM on their computers anyway so if it is balanced properly it wouldn't be much of an issue.
Some of us remember when CPUs had L2 cache external to the CPU. Then the Slot 1 had the cache integrated onto the same card as the CPU, and when the Pentium III came out, L2 cache was completely internal to the CPU die. I don't see external RAM going away any time soon, just because of how useful it can be to just add more RAM, but this seems to be following the same evolution, and the performance it brought. Perhaps one day we'll see internal RAM on consumer CPUs as well!
No the original Pentium III (0.25um Katmai) does not have on die L2. It only comes in the 0.18um Coppermine version, which was super cool. The 500Mhz coppermine can OC to 666MHz without a sweat.
8:53 I always figured HBM was the endgame for the entire Optane thing. Too bad it never really panned out since it had mad potential and could have changed how we think about, for example, database servers all together. Intel sometimes is so far ahead of themselves even they can't catch up to them (and then something like Arc happens 🤦♀)
@@TheExard3k It's not about persistency. But when your persistent storage is so fast and low latency as Optane was supposed to be you can get away much much smaller memory pools, hence you can use faster HBM. The entire promise behind Optane was that it is so fast (especially IOPS wise) that you don't need to keep your entire application's data in memory.
@@sehichanders7020 Well, it is a bit higher level in a sense, tiered pipelining is not _either_ HBM _OR_ Optane, correctly used both will be a boost just at a different level, Optane has been used as RAM lower tier and HBM here as higher tier, so from the CPU looking for data it would go L1, L2, HBM, RAM, Optane, mass storage if it needs to go that far with each tier being progressively slower and larger. It would work very well most likely, just be hella expensive and a bit of a bear to backend manage. But it would boost performance a lot for almost all data intensive types of applications since you'd never loose more clocks than absolutely necessary to get the needed data, so the CPU would be waiting a lot less time than it does now, which is very advantageous. Now if the data is not in L2 or L3 you have a immediate hit of 10+ clocks to check RAM and if you draw the short straw you have to go fish in mass storage SOL.
I remember you telling me this episode was coming a few weeks ago! The idea of memory-on-a-chip would be sweet for the consumer audience. It was worth the wait. :)
Similar to how L2 cache used to be external to the CPU, then moved adjacent to the CPU with the Slot 1 and Slot A, and then moved completely internal to the CPU die, gaining performance with each evolution.
Compare it to GPU VRAM - sure top of the line GPUs have slightly more but H100 is pretty industry standard and has 80GB. Considering CPUs are definitely going to have way lower throughout than GPUs it doesn't seem like capacity would be the issue.
its a HBM2e also works like VRAM, its superfast btw my P40 24GB tesla with GDDR5 gets 2,50 sec in stable diffusion, while P100 16GB with HBM get 0,8 - 1,5 now imagine i use double P100
While I work with virtualisation a lot compared to specific high performance workloads, this has always begged the question for me, even when playing around with a legacy Xeon Phi 5110p CoProcessor, how would a chip like this handle memory failure? Nowadays whenever we have memory failure, ECC kicks in as a first resort and then you have options such as Memory Mirroring so your workloads can continue with a reduced amount of available memory. How would a chip like this handle it, say, one of the HBM packages was defective or outright didn't work, does the BIOS of the system have any form of mirroring? Considering this is four seperate packages working as one, would this prevent the chip from booting up at all? Great coverage though, always fun to see what new products in the HPC sector brings to the table.
Good call out of the Intel Developer Cloud there at the end. It's so important to try these kinds of systems out to see if you'll even benefit from these features before you go out and drop a massive bag of $$$ on procuring one.
Interesting CPU for sure. All about finding the best use case. I was thinking this CPU might also be used to drive faster networking if it is using the HBM memory. Not sure if that was tested.
I'm wondering how it would do running Prime95. With multiple cores per worker, it can hammer the memory pretty hard so the throughput of HBM should significantly boost the per-iteration speed, assuming the clock rates of the cores are decent. Tuning the worker threads to stick with the NUMA nodes would give the ideal performance (4 worker threads, each using all 14 cores on the same NUMA node). We did some similar tests way back when on a Xeon Phi and it was pretty decent although the HBM on there was much smaller so it still had to go out to "regular" memory quite often which slows things down. I've found that going over regular DDR4, it only takes a couple of cores in a worker to saturate the memory bus, although you do still get marginal improvements as you add cores. By the time I got above 10-12 cores per worker though, you can actually see a degradation as the individual cores are just sitting there waiting for RAM so the overhead can make iteration times drop.
Honestly it feels like once AMD did their monster sized CPU chip everyone stopped caring about keeping things conventional like how it took on couple to make everyone start dancing at the school dance.
8:52 My thought is why Apple didn't try to aquire the Optane ip and patents. Then we wouldnt have to worry about write endurance, and also an even lower latency SoC in combination with the massive amount of L2 Cache Apple has
Optane drives have failed to write endurance being exceeded. Being used as DRAM extensions IIRC. Its best placement is as a large swap space or cache for tiered storage to preserve endurance and power on time of other tiers. Intel stopped production / development and sold it due to it not selling well enough. The purchaser IIRC was a company primarily focused on memory. Enterprise and high end prosumer SSDs serve sufficiently where it fits best for a tiny fraction of the cost per cap.
I use to work developing telecom servers that was ultra efficient. Just run on normal intel i series CPU:s. The one we had go down to 10W for the whole board with a full intel Xenon CPU if the memory was removed. With the memory they draw like 40 watts. (This was quite a while back, like sandy bridge era)
Isn't that also the kind of Xeon where you pay to "unlock" some of the accelerators and frequency curve? Also it's not really a workstation part sadly. Intel is marketing their Xeons for workstation, while I want a GPU Max 1100 (PVC-56) as a workstation card. I got hopes for announcements next week. Intel is demoing it on InvelDecCloud and I had a chance to try it. I believe my workstation will still get a i9 14900K with custom look cooling (slight chance of tec)
Ram in a APU would eventually also greatly reduce cost. HBM ofcource is expensive and such. but it might become normal to see APU's become the new general CPU's and make APU's more like SOC's, essentially it allows to add in many more features and ram in the cpu allows for much simpler and cheaper motherboards and such. meaning that ram integration in low end chips allows to make super cheap and power efficient chips(more normal memory modules). that said despite HBM being much more expensive on these high end systems it is great actually many years ago when HBM and HBM2 where still cheap to make(cheap enough to be used in mid tier gaming gpu's) I also recommended doing essentially the same using something like hbm directly in a cpu.
Is this comparable to the Nvidia Grace ARM based CPU with 144 cores that Linus tech tips showed 3 months back? Or just Intel trying to catch up? Would like to see a video about comparing the server against server.
This has been in production and is being installed into the Aurora supercomputer which will likely be the #1 in the world in November. Grace Superchip you cannot buy yet (we covered it on the STH main site) despite the hype.
Enterprise and personal chips will continue to be even more tightly integrated and they'll mimic more a motherboard than chips we see today (also with the size). Just check Cerebras chip... memory system is still way behind the compute.
I think CXL memory in the PCIe Gen6 generation will have more bandwidth and be more interesting, but some applications will still like locally attached. More interesting is if there is optically attached memory.
sleep states probably are a early version problem, since it likely has to do with the memory needing constant power. in the future with a motherboard which supports 2 seperate cpu voltages at the same time(based on pin groups) or if the cpu's have some added in logic then it should probably work. ofcource they might not have given it priority since honnestly a cpu like this right now makes most sense in a server. while it is also great for videoeditting and 3d moddeling and rendering and simulating, most such softwares likely don't support it well enough yet, and while good and well maintained FOSS software like blender might support it quite rapidly and quite well. many companies who have shown to be very slow and ignorant in adopting new tech like adobe(even though they seem to accept AI pretty well now), and things like solidworks which still don't understand modern computers have more than 1 cpu core.
I just don't understand where in the current Marketplace it makes sense to buy xeons. You can buy a AMD Epyc with double the core count and half the power consumption and usually 3/4 the price
Hmmm Nvidia should take a stack of that HBM2e for a new shield console. And they are sorta hybridizing the next consumer cpu with on-die ram like apple did with the M1-2 SoC's interesting times ahead I can get a frequency Tuned Epyc with enough cores and cache to build out a nice fast NAS.
I wish Apple would adopt this memory style for their Apple Silicon SoCs. No current Mac has upgradable memory. You buy the SoC configured with a memory capacity from the factory, and that's it... Sure would be nice to have off the factory floor fast RAM, and _user expandable_ memory expansion slots for future upgrades. I really am liking the direction Intel is going with these!
Everybody thinks Apple is being stingy or playing games with RAM. Memory of that type can't be slotted. Because of timing and signal propagation, the LPDDR memory has to sit close to the CPU and be soldered. Which in a way leads to HBM memory. I think that will happen and Apple might do that in the consumer or PRO space at some point.
@@billymania11 what are you even talking about. Numerous laptops and desktops have slotted RAM. Your high sleed RAM remains factory determined, as part of the SoC, and "slow" RAM can be slotted in at a later date by the user. Many computers have used Fast/Slow RAM configurations. Every modern computer already does this, to andegree, with Cache. This is merely adding one more later between. SoC fast RAM, and slower socketed RAM.
@@billymania11 i am literally describling what is inside laptops. _today..._ I work in a PC repair shop. I have been building and repairing computers most of my life. My first computer repair was in 1989. Look up how Cache memory works. Computers have had different amounts of different speed memory on and off die for decades. Most CPUs have at least 2 or 3 levels of cache memory, plus the external RAM accessed through the memory controller (also on die with modern CPUs). Some computers (mostly long ago) had both fast and slow RAM, accessed directly by the CPU for the fast RAM and through a memory controller for the slow RAM. The Amiga did this. Even many modern PCs can do this. If you have a matched pair of faster RAM modules in a pair of DIMM sockets on one channel, and a slower matched pair of RAM modules in the DIMM sockets of a separate memory channel, then many CPUs will be able to run each channel at it's best speed. There is no reason you can't have a high speed memory controller with some channels directed to on SoC chiplet RAM (HBM or HBM like), while _ALSO_ having some memory channels reserved for slower slotted RAM (either in SODIMM or the newly developed CAMM socket). There is literally no reason a computer manufacterer can't do this, particularly in lower factory memory configurations, where less high speed Factory installed chiplet SoC RAM is installed. You say "sure", like it's something unbelievable... I work on laptops every weekday. More have slotted RAM than don't, and some already solder some ram on board, and have a secondary slot for expansion. No reason you cant have some higher speed RAM on the SoC, as configured from thenfactory, and use other memory channels for slower socketed RAM. I'd LOVE to have sockets in my Mac Studio, so I could add to the already present 32GB of high speed RAM... But YES, Apple is being stingy, because they are profiting on people buying the RAM they _expect to use someday_ right now, while it's still expensive, rather than just buying the RAM they know they need to be high speed, and adding slower RAM in the future to aleviate usage for miscellaneous tasks, freeing up the high speed RAM for more intensive tasks.
What kind of place would be using something like this and what would they be running on it? This kind of tech is fascinating to me and I don't even know what it's used for.
@@ServeTheHomeVideo It is amazing!! 2 billion billion calculations per second. One thing that interests me that was mentioned being done by Aurora was Dr's studying neurology and mapping out the brains neurological pathways. What does the program running that even look like and also that it needs such mind bending computational power? I know I'm in way over my head, but to me it's such awe inspiring work.
So what I’m seeing here is apple complaining they couldn’t add ddr5 slots to the Mac Pro cause of unified memory was just their engineers not being allowed to do it
Imagine if AMD started adding HBM2E or HBM3 (that samsung connection they have) onto their Epyc.. aswell as the 1152MB of L3 cache and the 96 fast cores.
Well, Phoronix published reviews of the Newest AMD Epycs inluding Bergamo and they literally destroyed even the HBM version Sapphire Rapids chips....so apparently AMD doesn't need HBM
@@tomstech4390 their MI300C has no GPU cores at all and is 96 Zen4 cores with HBM, but it's unsure whether they will release it since there might be not enough demand for it since their V cache already gives them a lot of memory on-die
@@whyjay9959 HBM2 has got a Bandwidth from 420 GB/sec. There is quite some way to go for PCIe to allow CXL Ram Expansion at that speed. PCIe7 x16 only manages 240 GB/sec. PCIe7 isnt even out yet, and HBM3 is already beginning rollout 2024 with a whopping 512GB/sec Bandwidth. Even the latency on the Bus would be way too high, even if the Bandwidth would be reached. With HBM memory expansions die out. CXL only helps for "slow" DDR5 and DDR6. The HBM standard even states that RAM must be on the processing logic die.
@@Jerrec I think you mean bytes? Found a chart showing 128 gigabytes per second for PCIe gen6 x16. But sure, it's all a tradeoff. CPU-integrated chiplets get inherent performance advantages from having the shortest simplest connections but cannot be changed, so they will probably continue to be combined with slower, more flexible types of memory as preferred.
@@whyjay9959 I dont get your point. You are right with PCIe 6 x16 and 128 gigabytes. I wrote about PCIe 7 that comes out 2025. You are right, I mean Bytes. Sorry about that. If I can correct it, I will. Anyways, not considering HBM3 in 2025, it means HBM2 runs on 25% or maybe 50% speed. Thats not a tradeoff, that is ... unusable for such a memory.
I wonder how long it will be before the system boots up with just the cache and then a ACPI message tells the OS when the main memory is online. This would help with the long DDR5 training time.
Something like that would be valuable in the consumer market I reckon. Servers are already notorious for long boot times so I don't think there is a lot of incentive at the moment to enable a fast boot.
What is so difficult about the integration that Intel does but AMD does not? Why is this harder to do than AMD Instinct HBM or Versal HBM? If HBM is used as cache how many sets does it support and how long does it take to search 16GB of cache for a hit?
It’s "3D" Stacking, that makes it much more expensive. It’s similar to HBM Packaging (but still different) and not just a couple of Dies glued together on the same package. AMD could so it but they want lower costs. AMD uses these packaging techniques but not in Ryzen/EPYC CPUs
This Xeon's will Serve The Home - all homes at once 😎 But who needs this power? Usually the storage is the slowest part of a system and you better invest in faster storage than in faster CPUs. Most of the time several cores are idling. But i'm sure there are some strange physics-simulstions as a usecase... simulating earthquakes, weather or nuclear fusion... or simply having the fastest minecraft-server of all 😉
Question on video quality, everything looks fine except Patrick's face is super red, but everything else looks fine, and pictures in the video of Patrick looks fine.
We are getting more bandwidth using on carrier mounted direct attached HBM than with off carrier DDR5 somewhere on the board - crazy! *** Ehm no, if you DIDN'T - it would be crazy, and not just that, it would be a failed design that should just be binned and never released, since anyone buying it would be certifiable and should be put in a padded room for their own and others safety... It is a cool design, following the roadmap/wishlist/path to glory that was laid out many years ago of which Optane is also part. Intel has been at it quite some time, they have explained it and reiterated it many times for those who stop and listen. They are building systems to allow for multiple tiers of storage where you'd have L1 - L2 - HBM2 - RAM - OPTANE - SSD cache - SSD - HDD sort of, all in one system where data is promoted or demoted between the tiers depending on how *close* to the actual CPU cores it needs to be since closer = faster. It's like a Guru and disciples in a sense, with the disciples being the memory tiers sitting in ever larger circles around the Guru dispensing the wisdom and word of god. The closer one sits, the easier and clearer it is to hear and understand what the Guru says.
I'm usually an Intel hater, but man Threadripper and the GPU mining boom destroying Phi completely messed up superscaler development, and AMD just never filled that market niche back out again for Epyc (they had this nice locked down with Opteron), so there was just a pit there ARM was slowly trickling into like groundwater. Database hosting apps really needed these bulk core chips ages ago, but it's good we're at least getting something comparable now in the form of Xeon Max.
I called it! Less than a year after I asked when we would be able to boot servers without even needing to add RAM right here on one of your videos, and here we are! Somebody saw my comment and made it happen!
Hate to burst your bubble, but the CPU design timeline is such that they would have been actively working on this CPU design for *years* prior to review samples being available.
@@bradley3549 be that as it may, a lot of things, like the ability to use the onboard cache like system ram, are minor revisions that can be made in firmware or opcodes, the kind of tweaks that can happen at the end. The extra cache was planned for years. Booting from it was my idea.
@@aarcaneorg You're definitely not the first to think of integrating ram and CPU and then booting from it. That's been a feature of CPUs for a LONG time. Just not x86 CPUs. Sorry.
In a way, yes. But think of it more as caching slower/ higher latency/ higher trace power far DRAM to faster/ lower latency/ lower trace power close HBM. There is a big difference between access over a few mm on package and going out of the package, through the socket, through the motherboard, through the DDR5 socket, onto the DDR5 module and so forth.
Is it possible to apply such design to GPU? A bit like HBCC for AMD but you can install DIMM modules on the GPU to give extra RAM for various purpose, such as running large AI models, running heavily modded KSP and try novice shitty program that memory leaks for no reason.
3:14 "thats why intel does it but AMD doesnt." Wrong, AMD started doing chiplets back in 2015 with the... uh... i forget the name of it. Ill edit the comment lol. It was one of their GPU's. And also starting in 2023 with their 7000 series GPU'S theyve gone back to doing chiplets. Still no HBM to replace VRAM entirely yet but its still a chiplet design for the core.
We have to give credit to AMD on this one, regarding chiplets. The danger though is the approach AMD chose to implement chiplets. Later designs like Intel's might be superior in a range of functions not initially considered. I do expect a pendulum swing in favor of Intel as their approach gets validated.
No sleep states are needed for any platform, though preferred when running on battery. A workstation or gaming PC benefits from disabling them. Except for power choking of unused cores to boost those heavily used. Or cooling is insufficient, so sleep states are needed to help for that. Lacking them is not a reason for not using the same CPU as in this video for those workloads. What is always relevant is the performance per cost. Or just performance if cost doesn't matter.
you keep talking about memory bandwidth. but what about latency? i think this would help applications that need low latency memory a lot more. ddr ram latency is sooooooo slowwwwwwwww. bandwidth is fine
It is AMAZING to me that for CFD, the AMD Genoa X still TROUNCES this Xeon Platinum Max, even despite the HBM2e memory at nearly DOUBLE the Xeon Platinum 8490H baseline, whereas the Max, running in caching mode, actually performs WORSE than the baseline, and with the HBM2e only, it performs slightly better, but nowhere CLOSE to what the Genoa X is able to do. That, for me, is a much better marketing slide for AMD than it is for the Xeon Platinum Max.
