About the "power wall". Intel chased this with the Pentium 4, assuming GHz would go up forever, and at the time Gigahertz was marketing. The issue is, power consumption is roughly a function of frequency times voltage squared. As frequency increases, more voltage is required to keep it stable, but since this causes power usage to scale exponentially, it is a problem. Also, as frequency increases, the voltage required starts to runaway. So there is a balance of frequency that is optimal for different CPUs. Generally the limit is around 3.5GHz where it starts to runaway, making the transistors flip on/off faster becomes increasingly difficult. So, it is much more efficient to run multiple cores at a lower frequency, than to try to chase Gigahertz. As transistors get smaller, you can increase complexity to increase instructions per clock, raise frequency, or both, but frequency is always a balance for power consumption. Modern CPUs may boost to 4.5Ghz or so, but not much frequency has been gained in 20 years. More effort goes into increasing Instructions per clock, aka IPC, multiple cores, cache, branch prediction, SIMD, etc. Laptops are far more efficient, running perhaps 3/4 frequency can easily cut power usage in half, or even lower, due to the exponential nature of voltage and power consumption. And any reasonably modern CPU automatically varies its frequency based on load to reduce power, so idle cores may run at say 1 GHz instead of full 3.5-4GHz max all the time, and partially used cores somewhere in the middle. There is much more that goes into CPUs, like pipelining, and branch prediction, which have their own tradeoffs and balances.
6:57 There is another technique, called “pipelining”. Or, in industrial terms, you set up an “assembly line”, where one person just butters the bread, and passes the slices onto the next person to put in the cheese and slap them together. Modern CPUs use this technique as well as multiple processors. And then there’s also caching.
In essence a modern computer is a parallel environment, typically both symmetrical and asymmetrical at once. Since the CPU is typically having more then 1 core, and it offloads tasks to the GPU that as well typically has more then one core (sometimes even having different types of cores for different tasks, so the GPU can be asymmetrical as well!). Not to mention the fact that there also is many other processors in a modern system. Since HDDs and SSDs have memory controllers in them, even network cards at times have an on board processor.
You'll probably lose money mining Bitcoin on a GPU. Even if you have free electricity and Internet, it's just not competitive with ASICs, by a long shot. If you want to mine on a GPU, pick some other coin.
@@fllthdcrb I'm pretty sure all cryptocurrency is a bubble. And given how much of an arms race it is the issue only gets worse. I wonder how much of Global Warming can be directly blamed on Bitcoin miners.
I remember the "transputer" and occam (which in combination allowed parallelism) being "the next big thing" and if I remember correctly it was available as a side computer (add on, like the second 6502 shown in the vid) for either the BBC Micro or one of the next gens Beebs such as the Archimedes. I saw one doing some amazing real time graphics at a computer fair in london as a kid and was stunned at how much more powerful it was compared to the BBC which even in the day had some stunning graphics and games for what we'd now consider a hugely underpowered processor.
What a beautiful HP Z800 (?) dual Xeon system at the start. I have a couple of those under my desk at home. Bought them a couple of years ago (used of course).
Some people separate "multi processor" which share the same memory from "multi computer" where each processor has its own memory and use some network technology to exchange data with the others. An example of a multi computer would be any system built with the Inmos Transputer, which should be mentioned in any talk about parallel processing (specially in the UK).
I'm waiting patiently for Computerfile to do an episode on transputers. Apparently, and forgive me if this is completely wrong, chip manufacturers had hit what they thought was the hard limit for transistor size in the mid-eighties and transputers were seen as a way around that. I'd love watch a 'historical' episode on that. :-)
My impression is the opposite: Inmos expected transistors and chips to become cheaper and cheaper so at some point in the future a whole processor would be the basic component and a system would have a very large number of them. You certainly wouldn't build a consumer product with 30 Transputers at 1980s prices.
“Multi computer” would imply entirely separate physical boxes connected by cables. These might be regular LAN interconnects, but custom high-speed ones are also used (e.g. Fibre Channel, I think). One also talks about “tightly coupled” versus “loosely coupled” multiprocessor machines. Typically “loosely coupled” ones don’t share RAM, so they communicate by sending messages over communication channels of some sort. “Tightly coupled” ones share (almost) their entire RAM space, so they can communicate just by reading and writing shared memory. A small step back from full tight coupling would be NUMA (“Non-Uniform Memory Access”), where each processor has some local memory, and can also access memory belonging to other processors, but the latter accesses are slower. So this logically appears just like tight coupling, and is almost as easy to program.
nother reason for multiple processors is dedicated task performance. Some tasks are timing sensitive. You want the frames to be coming constant - not pausing every time you click a button. And so, you can have dedicated processor for sound, video and other time critical operations (moving a motor without interrupts for example).
Such tasks don't necessarily require a CPU - every console has a graphics processor but we don't call it a second processor if all it does is spew frame data to the TV.
@@thewhitefalcon8539 yeah. I was meaning things like custom PWM for motor in PID... when you change the position - it needs to process the incoming message. So, a single core processor will disturb the PWM pulse... sure, can cover up by including the processing time into the pulse width but there's a thing... Also things like OS interupts can disturb, and so forth. don't wnat to interfere with my PWM each time a mouse moves or clicks. Obviously, if you deal with a massive data stored on a hard drive that merely needs to be "flushed through" for pixels, a GPU can do magic there, fully independently. Just specify the start of the data series and the end and it will just "chunk it" in a mass parallel FPS... not even causing memmory conflicts... so to say... in a hevily over simplified way.
It's actually more common than you'd think really, and how leniently you want to judge a processor. Many video subsystems were implemented in a way that made them essentially dedicated processors. The SNES, rather famously has a Sony produced sound processor subsystem And another obvious one is the Megadrive, which has both a 68000 and a z80.
One great use for multiple processor systems is in "Monte Carlo" statistical simulations. I just completed a pet project this year of a tool that simulates all of the possible outcomes of our fantasy football league to determine likelihood of each team making the playoffs. Even starting at week 7 (of a 13 week season), and just considering wins and losses (not ties), that meant my program had to consider over 4 _trillion_ possibilities. With fairly optimized code on an 8 core system this took just over 24 hours. The original, single threaded (single processor) version written in VBA would have taken close to 130+ _days_ to complete the same task. I'll put a video to the result in a follow-up comment for anyone who is interested.
The CPU in the PS2 was a MISD when you put VU0 into micro mode. You pair two instructions pr. clock cycle, upper and lower instruction running from one data stream.
@@jc_dogen ah, that makes more sense :) VLIW architectures were being experimented with a lot at the time, although I only ever read about them in passing. Now I feel like diving deeper into the tech that went into the PS2. I always thought VLIW was relegated to some niche subtask of supercomputing.
It is somewhat trivial to attach more processors to a system, but like what Dr.Bagley said, it's dividing up tasks so that you are utilising them in an efficient and useful way that is the tricky part. It's something I've been fascinated with since the mid 80's.
10:35 Another kind of SIMD was introduced back in in the 1970s with the Cray machines. These had instructions which could act, element-by-element, on arrays of up to 64 elements with a single invocation. However, the operations were not performed simultaneously on every element, but were pipelined so that, after the initial instruction setup, the result elements would be churned out in succession very quickly. Contrast this with PowerPC Altivec (which I think predates Intel’s initially rubbish MMX), where operations are performed on all elements of the operand vector simultaneously. Note also that these vectors tend to be shorter than the Cray ones: a maximum of 4 elements (maybe 8 for small integers) is typical.
@Michael Hansen I think the earlier ILLIAC IV research machine was also SIMD, with four processors executing the same instruction set on four data streams. Seymour Cray looked at the difficulty of programming that before creating his architecture for the Cray-1.
8:16 Case in point: the Atari Falcon had a Motorola 56001 DSP, because in those days general-purpose CPUs were not fast enough to do processing of CD-quality sound data in real time. Nowadays we don’t see DSPs much in regular PCs (except maybe embedded inside a controller for some peripheral). But new, more compute-intensive tasks have become popular, which is why we now have programmable GPUs, for example. Will the wheel turn again? Yes, I think it is possible.
See thats the interesting thing, what is the minimum processing power required to count as a "processor" ? because most computers nowadays have more than one processor, the other processor is just processing dedicated things instead of anything. Multiple other processors actually. ADC, DAC, etc etc.
In my opinion, it's the ability to independently execute a sequence of instructions stored in some sort of memory. The keyboards that Steve mentioned as having their own internal processors count, if they use a general-purpose processor rather than a hardware keyboard scanner chip. However, in computers from the '70s and early '80s, I think it's more likely for keyboard electronics to be of the latter type. As far as I know, any USB interface will have its own dedicated processor to handle the lowest layer(s) of USB protocols. Modern GPUs count, since you can program them to do things other than just draw graphics on your screen. Older graphics displays may have dedicated chip(s) to repeatedly get data from display RAM to refresh the CRT, and not do much of anything else. Those wouldn't count.
@@kc9scott Heck, GPUs are arguably more than one processor of different designs. Also, the IBM PC had a microcontroller for its keyboard, so it'd count. USB does require separate circuitry at least on one side yes
The Commodore 128 computer has 2 main CPUs, a Z80 and a 6510 (or 6502?) in it, but unsure if it counts as a multiprocessor since it seems to run only one CPU at a time. Also, the Commodore disk drives are built like computers with a processor. Some do mention 16bits and 32bits Sega consoles which yes they have multiple CPUs inside like one usually kept for sounds, one for graphics (not yet a GPU back then), etc.
butone of the cpus (on both systems) were used to process audio, both of these are Asymmetric or custom hardware for specific tasks, genesis and arcade boards is a perfect example, they used multiple cheap off shelf components rather than designing a more complicated multi core on single die (very expensive at the time), IBM main frames used the later.
And a SNES had two CPU's already but relatively frequently ended up with a third in the cartridge. Actually, the programming reference for the SA-1 specifically mentions how it's presence forms a multi-processor system, and how you could handle it as an 'accelerator' or a true multi-processor system, or some mixture of the two depending on how much complexity you were willing to accept. it's pretty weird when you think about it... Then the n64 has a GPU that technically owes a large chunk of it's functionality to containing a second CPU core with the same instruction set as the main CPU - yet you wouldn't typically be inclined to think of this as a multi-CPU system either. Really quite odd when you think about it.
Commodore were always an exception because they owned MOS technology and, thus, manufactured their own custom chips. So, for example, the C64 had the SID "Sound Interface Device" sound chip and VIC-II "Video Interface Controller" graphics chips (the VIC-I was in the VIC-20, after which the computer was even named). (But I guess, on a technicality, in terms of this video, it wouldn't count as a genuine "multi-processor" system, as these chips did not accept instruction streams. You programmed them by changing hardware registers memory mapped into CPU-accessible RAM and the VIC shared memory with the CPU for the video framebuffer. Not that the video went into such a fine detail, but from the way Dr.Bagley was classifying his collection of machines - and how he considered the Amiga an "arguable" multi-processor machine - and the discussion of "SISD / SIMD / MISD / MIMD" then we're not fully counting chips that don't take their own instruction streams here. So by the same argument as the Amiga, the C64 and VIC-20 were "arguable" multi-processors. Down that way headed, with custom chipsets for specific purposes of graphics and sound, but not fully there to qualify as proper "co-processors", as these custom chips did not have their own instruction streams and were very much peripheral slaves to the CPU, which actually did all the true "processing" work.) And, yes, the C128 - though it had two fully-fledged processors on-board - only enabled one of them at a time, so it wouldn't really count as "multi-processing", in terms of there not being any true parallel operations going on at the same time. The system was "multi-processor" but did not do "multi-processing". Subtle but important distinction. And also Commodore did just simply have another 6502 with a small amount of RAM, inside their 1581 disk drives. Because of this, it was actually possible for the CPU to "upload" new software to the disk drive and execute it. There are many C64 demos that actually do this, uploading their own routines to the disk drive to more finely control the disk loading and, thus, could format the data on the disk however they liked - straying from the standard disk format, save the main index, as you did still need to start the loading process to get their code into RAM in the first place - and could utilise tricks for quick loading. It was even conceivable that you could upload a routine to the disk drive to have it not only load data from disk, but do some decompression as well - though this was limited by the drive not having much RAM and only a 1MHz 6502, so you couldn't do a massive amount with it. But, yes, the disk drives were their own 6502-powered computers unto themselves. Though, again, we'd have to check definitions here, as these were separate and independent physical units, so this arrangement would count more as "networking" between two machines, I would think.
Just nitpicking: If you count the keyboard CPU of the Atari ST(E) you have to also count it for the Falcon, since all members of the ST/STE/TT/Falcon-family use the intelligent keyboard desing with the 65xx (or 68xx?) keyboard processor. So, Falcon has 3 CPUs: 68030, 56001, Keyboard.
MIMD would be running multiple instructions on multiple data (like sending it to some arithmetic unit and float unit), not diving it up to do single instruction on single/mutliple data. Parallelism doesn't change what type of register and operation you're using.
Well, the first bit when asked "which systems are multiple processor systems".....they all are actually. There is a chip for all the disk drives that runs separate to the main board chip, and in some instances, has dedicated RAM and ROM for its own programming. It was a long time ago (I was about 10), but I remember being able to solder a couple wires on a 1541 board and had the capability of switching the 6502 ROM pointer to the RAM index, allowing the chip to run editable code on-the-fly separate from the c64, giving you effectively 2 cores. Just saying.
@@BrianMelancon Fairly sure the Atari floppy disk drives were much the same. the Atari 1050 drive certainly has a CPU and it's own memory. Not sure whether it's technically capable of standalone operation, but you can upload custom code to it for sure. There's likely a lot of accessories for these systems that technically qualify as computers in their own right.
@@BrianMelancon ... you can now turn your "fairly sure not" into a "definitely has" a processor. I personally never worked with the system on the internal level, but I know the chipsets were there on the board above the drive unit. sundhaug92 - yes, the c64 used a 6510, basically the same chip considering the RAM/ROM limitations on the 1541 board anyway (didn't need 6 IO pins)
I'm pretty sure that Apple II floppy disk drives don't have their own processor. The disk controller board (that plugs into the main computer's bus) is hardly anything more than shift register(s) and a boot ROM. The main system CPU does all the work of group coding, waiting for the proper sector to pass under the head, reading/writing the sector, and running the stepper motor to position the head. This was one of the design decisions that forced them to keep their CPU clock speed unchanged for many years.
The BBCs (including Archimedes) are the only computer listed which the second processor did not need to be the same type as the main one. It is possible to get a Z80, ARM, 30216, 86 processors as well which.
You probably already know this, but If I recall correctly, one of the primary purposes of the "Tube" interface on the BeeB Model B, etc was for Acorn to evaluate / test the processors for their "next generation" machines - so ultimately, the test bed for first ARM or Acorn RISC CPU they built went into one of those side boxes, from which they interacted with the new CPU. The other thing I thought was cool was that Acorn prototyped the RISC micro-code on the humble Beeb as well as the circuit design :)
In the late 70s I worked with a CDC Cyber mainframe that had PPUs that would perform tasks asynchronously from your program on de CPU, for instance for performing IO. Come to think of it, why would you call something a CPU when there are no PPUs?
So is anything with additional dedicated ships a asymetric multi-processor system? What about security/cryptography modules (like TPM) or even the northbride ship on the motherboard?
The original IBM PC was a multiprocessor system, the keyboard had its own processor. Before that I worked on a mainframe with smart terminals (VDU), they did also have their own processor. In my opinion you do not rate a computer "multiprocessor" before they all do serious work, all the axillary processors do not count.
I supposed if you loaded user programs on it that the performance of the computer would double. Both drive and computer uses a 1MHz 6502 CPU (C64 CPU was called 6510, but is mostly the same).
The original PC used a microcontroller in the keyboard with a fraction of the computing power of the main CPU. The AT added a second microcontroller from the MCS48 series on the computer end of the keyboard as well. It even used the keyboard controller to drive the A20 line to memory, but I wouldn't really classify it under multiprocessing, even not asymmetrical. Fun fact: the situation is similar to how the PIC microcontroller came into existance. The original PIC1650 was designed as a programmable peripheral interface, to offload the cpu in dealing with devices.
So as soon as I break my problem apart and execute it in multiple threads does the OS (Windows 7+) then automatically run that on a separate processor? Also, what about Intel's "virtual" cores do I get any speed benefit when running threads on them?
I think SoC can be consider as Multi-Processor too isn’t? Of course they can’t handle instruction set by itself and they work more of like a hardware acceleration which still need an input from CPU.
I have a peculiar MP-system made by Sequent which contains basically all microprocessors of the era in one box: 2 Pentium 60 and 2 486-50 (these four sharing 160MB RAM), a 68030 handling Ethernet, a i860 handling SCSI IO and an i386 handling 32 Serial Connections. It runs dynix/ptx. It seems like a desperate try to remove all unnecessary load from the CPUs because they were so slow. Does anyone know it?
Did the Atari Jaguar have a Blitter chip I know the STFM and STE did and wouldn't it count as a processor. Or maybe it was the other way around it's been a few years since I've had one apart. The last time I was inside my STFM was to upgrade the TOS Chips which I ended up not doing because I discovered it was an STFM with an STE prototype board in it and that's how it shipped from the factory. The board is marked STE so I decided not to flow solder on it. May have been common I don't know.
Many Telephone switches would run three CPUs with a single data stream and the same instructions. It worked in a voting system. Majority win. Another system is to run two CPUs and throw away the data and restart if they mismatch. Rockets do a similar system. Multiple cpus with the same instructions and data but must match.
Do modern rockets still do that or have they moved on to some other form of error checking? I heard about the multi-processor approach being used in the Apollo days (if memory serves they used Super Crays) but would think they'd have a better way of doing things now.
@@grn1 probably. Rockets that fly regularly will be conservative about true trying new things. If something worked in the past there isn't much reason to change.
But if the Atari is multiprocessor due to the CPU in the keyboard, then there's no way you can say the Amiga isn't. There is a 6502 in the Amiga keyboard.
Problem was, dos and windows couldn't deal with more than one processor at the time. You had to run windows NT, when it came out, to get multi core use.
1980 era video games used up to three processors : sound, video drawing and game logic / ui. Semaphore flags in shared memory are used to keep processors aware of what tge other ones are doing.
I used to have a dual processor liquid cooled Apple Power Mac G5, the only dual processor system I've owned. Been wanting to build an X86 dual processor machine for years, but there's little reason to these days.
@@johnfrancisdoe1563 I've found it just isn't worth it these days. The on-package multi core systems we've got now are probably more efficient than two separate CPU packages, and the type of workloads I do in my machines don't warrant such massive CPU multi-threaded compute power.
Yeah, you get things like that. I wanted to build a beowulf cluster for a while, but as much as the idea of it amuses me, I can think of no practical purpose whatsoever to me actually doing so...
I would assume multiple processor means multiple CPU to RAM sets. i.e. multi core wouldn't count because it all accesses the same RAM modules unlike server boards which have multiple processors each with distinct RAM slots. Though of course even in that case anything with a graphics card is arguably still a multiple processor computer unless you count graphics cards as computers in and of themselves... There's a lot of grey areas here I think.
well today must people have smartphones, which from time of Nokia N95 is very much AMP and now it is infused which SMP so now we have HMP (hybrid) so we have CPUs (sometimes more then one type of cores, sometimes even different flavour), we have DSPs (oh they do make huge come back into phones - image and sound are processed and sometimes even calls) and we have GPUs, which do sometimes nie then just graphics (most of them are GPGPUs - even N900 had one) all those elements are available and actually used on all flagships for years already
I have a question GPGPU have been around for a while and alot of math intensive tasks have been made to run on the gpu's like the boinc? gpgpu where you process distributed datachunks . But i have not seen a single emulator made to run on gpu why i s that, as i understand there is C,C++ like languages. I am not sure about what CUDA and AMD close to metal was. But i know language like Brook gpgpu was borrowing alot from C? The applications that i thought were coming out of gpgpu never really took place just fancy demos? A modern GPU has its own memory, and streams audio maybe even process it. So why are there no GPGPU emulators of old computers, would they still utilise to much of processor for th e I/O work?
0:14 Oh, that's an HP Z820 (or similar), very nice dual processor workstation 3:01 That looks like a Nexus 9 Tablet with Folio keyboard in a sleeve And that's a Yubiko Type-C security key in Macbook!
If you have a Commodore 64 with a 1541 drive you have two 6510 processors as the floppy drive has the same CPU. Some games and quite a few demos actually use it for extra compute power.
I wonder how much can be offloaded from the CPU and GPU into specialized chips at this point while GPUs can run in SLI or whatever i wonder if it would not be cheaper to have Cpus for GPU that further specialize like when humans do that kind of thing. Did you ever talk about bio-computing?
There are actually a significant amount more chips in modern systems. Yes you have the CPU and the GPU, but even within those there are several different chips with different purposes. Most IO devices these days have their own controller chips (these are especially important in SSDs as they are where the real improvement in speed is coming from), the motherboard has a collection of chips.
And also more expensive. Proprietary OSes, and the proprietary server-side apps that run on them, tend to be licensed based on the number of CPU sockets in your machine. A special exemption was made for multiple CPUs in the same socket, just for the sake of keeping the customers happy, because such a thing was unknown in the 1990s.
@@lawrencedoliveiro9104 It wasn't that much more expensive, and NT licensing was fine with only two processors. Didn't have any other software that wasn't part of the MSDN. I believe the motherboard was a Tyan Titan Pro. Funny socket design on that model of processor.
You talk about multiple processors and blew right past the XP machine as a single processor. Where does the 80X87 math co-processor fit in? Is that a bolt on part of the main processor, or is it indeed a separate processor? I think one could lean either way. The CPU passes information and waits on a result, therefore making it a bold on addition to the main CPU. An 80X87 cannot function by itself (again, my feeble understanding). If it is considered a separate processor, the next question is arr all processors 486 and above multiprocessors? I have no clue whether there is a builtin math coprocessing slice of silicone, or of the 486+ cpu capable of doing FP math itself?
Even with a single-task OS, if you use interrupts for anything, the main CPU could service interrupts as they occur (and thus be able to do something useful) while waiting for the AM9511. Again, I don't know whether the same thing would apply to the 80X87.
14:28 Threads are separate execution entities that share a common process context. It’s not the only way to parallelize a program: there are also programs that run multiple separate processes in parallel. In fact, one might argue that threads are about the last form of parallelism you should resort to: look at other solutions first if you can, then fall back to threads if nothing else will do. This is because of their propensity for the most difficult sort of bugs: the ones that are timing-dependent, so attempts to track them down can often cause them to (temporarily) disappear.
The definitions get pretty complicated huh. Aren't modern GPU's nowadays basically near fully programmable? That makes them SIMD multiprocessor clusters in their own right (given they effectively have core counts measured in the thousands.), but then these graphics cards are coupled to a system where the main processor is also a multicore setup. So you have a symmetric multiprocessor feeding data to a second symmetric multiprocessor that collectively forms an asymmetric multiprocessor system composed of symmetric multiprocessors... OK, I'm probably abusing the definitions of those terms at this point, but it surely gives you some idea of how messy this has all gotten? XD
GPUs are mentioned in the video. In the 90ties graphics cards had chips with fixed function units. In the 00s, GPUs became more and more like CPUs because they allowed for shaders to be programmed (but they are more like SIMD, not SMP). See also GPGPU on Wikipedia.
Graphics card is more like a small computer in a computer. It has it's own GPU, VRAM, VRMs. Card is dependent on computer to function, but almost isn't. Graphics accelerators used to do very limited amount of things, but they got more powerful, got reprogrammable shader pipeline instead of fixed shader pipeline. Now nVidia's RTX cards even got some ray tracing processing and deep learning processing, effectively allowing it to do more things.
It’s called “homogeneous coordinates”. Sometimes the extra component isn’t a 1; a value of 0 can be used to represent a “point at infinity” (having a direction but no actual position).
@@aarondavis5386 It means that all linear transformations in n dimensions can be expressed uniformly as multiplications between vectors of (n + 1) components and matrices of (n + 1) × (n + 1) components (e.g. 4×4 matrices and 4-vectors for 3D graphics). If you try to use n-vectors and n×n matrices, then rotation, scaling etc are represented by multiplications, but translation (change of location) has to be done by addition.
There is one more type of multiprocessor system. What arm call little.BIG. back in late 90-tys I was figuring how to build a system like that using a low power 486 in a PCI slot with a k6-2 or a k7 main processor Two major points. Get windows the second thread to hang up. (Back in win 98 time windows time out was quite common). The second reason was to make the major processor micro sleep during work. Reducing energy consumption quite a bit
