Running Neural Networks on Meshes of Light 

A big Thank You to Alex Sludds too (from grateful audience)!

I almost commented a few videos ago that you have single handled staffed all US semi conductor fabs with engineers in the next 10 years just by posting. Happy to see you grow so much even in your own niche without clickbait.

As a ECE sophmore in college, I just wanted to say that you're playing an amazing role in developing the next generation of semiconductor engineering! :)

I work in a research group which develops simulation tools for these photonic circuits. This video was very well explained. I can't wait to see what photonic circuits will be used for in the future. Thanks for making this video!

FWIW, that observation that IO consumes more power than MAC operations in an AI accelerator is pretty universal across problem domains. I often quip that it's a silly accident of history that we call the metal boxes "computers" since almost none of the power, gate count, mass, etc., is actually used directly for computation. Most of computing in-practice is about getting the right data to the right place at the right time. I have a patent on internet-scale CDN configuration. But it's all the same at every scale. Pushing configuration data across the globe to the right server. Pushing weight data across the chip to the right IO pin. The memory/storage hierarchy instantly becomes the constraint as soon as you try to scale compute at any scale, in any domain. The ideas driving photonic compute for AI will be directly applicable to more seemingly mundane use cases.

Electrical signals are essentially sent at the speed of light (because it's not charge carriers that transmit the signal, it's electrical fields), so it's not the signal propagation speed that allows high throughput, it's ability to distinguish signals. Electrical fields get "smudged" along the way, but so does electromagnetic signals. But second one does it remarkably lesser. Also, electrical logic gates takes some time to transition from one state to another, and that's the major factor in limiting throughput. If there would be faster switching transistors - higher frequencies would be available. I don't know about photonics, but it seems that for it transition is either much faster, or architecture is completely different, like, instead of switching state, light is split along the way and goes through preconfigured logic gates, so processing is faster while it goes through same transformations, but takes some time to switch from one configuration to another. But there's possibility that same results could be achieved with using electronic components.

Wow, your videos just get better and better. As I watched this I kept having flashbacks to my university math/physics discussions on matrix mechanics of more than 50 years ago, and realizing that those concepts remain important in today's world.

You did a really good job on this one, man. That's no small feat, bravo

1) Electrical signals also travel at the speed of light (speed of light inside the conducting material), the signal is transmitted by photons. The main limiting factor of electronic computers are the capacities inside them. The most basic one is the capacity of the FET gates. In case for a FET to function the gate needs to reach the desired charge and this although getting smaller and smaller with the new nano transistors is still there. The same applies to discharging of those capacities which still takes time and also dumps all of their energy into heat. 2) the speed of light is a limiting factor even on photonics: A 4ghz chip, something that might be in a modern computer has a 4ghz clock and a period of 0.25ns between clock cycles, light can only travel 75mm in such a period and this is in the best case scenario (in vacum). A theoretical 40ghz photonic computer will have a 0.025ns or 25ps period and light will only be able to cover 7.5mm. This means that even in a 40ghz chip, the maximum distance for the datapath inside a computation core is at 7.5mm in the best case scenario. Having photonic computers working at teraherz is almost certainly sci-fi. And ofcource this type of cpu, with such a small distance covered between clocks will have very big memory bottlenecks (time it takes in cycles for data to be stored-recovered from the memory) and will require the memory to be very very close to the chip.

Silicon Photonics represent a quantum leap in technological speed and power efficiency. One major issue when dealing with light is the fact that you're reading the probabilities of the light waves. You run into quantum mechanics at this level. Light is sensitive to interference of the environment through quantum decoherence. I believe there will be a solution to this problem as our understanding of quantum systems evolves.

I watched a you tube called "The next big step in computing" by Anastasi, she mention how they are trying to use light in a analog form, different intensity's as a new way to compute. Not as in depth as here but still over my head.

At 5:04, did you mean to write picojoule instead of petajoule?

im really glad to see this tech being mentioned more and more.

Great work Jon!

So there an analog aspect of these calculators as well? Very cool...exactly what was wanted. Can't wait to see how this tech works out....cheers

Excellent work! My company is one of those working on photonics/quantum compute InFlight as optical networks transit the world. Though quite different, great progress has been made.

i have the idea to do computing with light 5 years ago, but have no clue to get it done. glad to see a big step in computing!

"Photonic Neural Networks" That a yummy combo of words. I hope this video doesn't disappoint.

I have recently seen a great video on why our end/or gates will always dissipate energy. The answer "boils down" to entropy. Depending on how far into theory one wants to dabble this might be pretty interesting content.

have you heard of Brainchips Akida chip? Currently in production. Akida is a neuromorphic system on a chip designed for a wide range of markets from edge inference and training with a sub-1W power to high-performance data center applications. The architecture consists of three major parts: sensor interfaces, the conversion complex, and the neuron fabric. Akida incorporates a Neuron fabric along with a processor complex used for system and data management as well as training and inference control. The chip efficiency comes from their ability to take advantage of sparsity with neurons only firing once a programmable threshold is exceeded. NNs are feed-forward. Neurons learn through selective reinforcement or inhibition of synapses. Sensory data such as images are converted into spikes. The Akida NSoC has neuron fabric comprised of 1.2 million neurons and 10 billion synapses. For training, both supervised and unsupervised modes are supported. In the supervised mode, initial layers of the network are trained autonomously with the labels being applied to the final fully-connected layer. This makes it possible for the networks to function as classification networks. Unsupervised learning from unlabeled data as well as label classification is possible.

Thanks for metioning the paper! I knew I recognized this and when you showed it I realized it was 4 years since I read it.

Awesome video. Another reminder of why I’m subscribed. 👍🏼 This technology is really cool. It seems like the use case to make this commercially viable is training massive neural networks rather than inference. It’s the training that is computationally expensive and requires stupid amounts of computing power. That’s a challenge that needs to be solved. Inference on the other hand is trivial by comparison. Almost every smartphone these days has a built in neural engine that can run inference in real time at less than a watt for relatively simple problems, and even moderate to large problems can be run through inference on a traditional modern CPU with no dedicated matrix multiplier.

Quick posts! Really enjoying your silicon rabbit hole.

Great video as always! Keep them coming, love it!

This was a very informative video. I am in fact working on a time-multiplexed SiPh matrix multiplication design like the one you mentioned towards the end of your video.

6:28 High bandwidth is not due to the physical transfer speed, in fact, electricity also move as speed as light. Bandwidths usually determined by how many bits per transfer and how many transfer per second. A normal GPU transfers couple hundred bits at few GHz.

Thank you for this! I have been seriously wondering about Lightmatter and I just checked up on them recently. Looks like they’re hiring some powerful folks and hopefully going to be able to offer real products soon!

Great video! Thank you very much for making it! I’m currently working on a research project with ultra low precision neural networks. I wanted to ask if reducing the number of bits in the activations and/or weights to about 2-3 bits each (using state of the art Quantization methods) would help with the issues with photonics accelerators raised in this videos accuracy and scale? In general, most neural networks these days can be quantized down to 4-bits with almost no loss of performance, using the latest Quantization methods. So 8-bits might be a bit unnecessary, if these methods are used.

Excellent video! Learned a lot. Well done! 😃

From what I've read (ML is my main field), even AlphaZero (and definitely MuZero) run on a "high end PC". The training was done on TPUs and simulation on CPU servers.

I can't thank this channel enough. Good job.

Such a fun video ! Great work guys !

In my favourite si-fi movie Bicentennial Man they kinda show photonic brain, even though it's called positronic. I love it now...

Man, you are remarkable. Btw, do u do financial consulting for tech companies? Or plan to do in the future?

Good that you started looking at this. More presentations in this area should follow. Talk to more experts.

Welcome back on Asianometry. Thanks for your answer. I’m not so ready! First my family. See you soon. I hope! DV

When I was getting my chemistry degree I noticed multiple labs working on materials for things like this. It's cool seeing it hit youtube.

You mention that the 2016 AlphaGo was ran on 48 TPUs. Were these required for the Inference step used during the matches? Or was the final trained version running on just the laptop we saw in the documentary? Thanks for the great video!

This is very interesting. Excited to see what the future brings in this space.

Cloud Tensor Processing Units (TPUs) Tensor Processing Units (TPUs) are Google’s custom-developed application-specific integrated circuits (ASICs) used to accelerate machine learning workloads. TPUs are designed from the ground up with the benefit of Google’s deep experience and leadership in machine learning.

Great Video really interesting to see how other fields are tackling the issue of power consumption. From what I understand, it is not a fair comparison to make between conventional neural networks and the human brain. The human brain works on a completely different mode of computation where data storage and computation are unified and signals are carried through spike potentials. Hopefully photonics can be applied to designing analogs for spiking neural networks.

Amazing and insightful video. I take solace in the fact that brain is much more efficient if not the best at specialised tasks. Let's hope the photonic innovators are able to get a product market fit, and who knows, due to efficiency reason we just might be able to simulate quantum computers before we actually design quantum computing at scale !

Excellent channel. Objective, serious and extremely informative. Channels like these are what make RU-vid great. Not those bonehead vloggers.

This is one of my favourites don’t know why it doesn’t have more views

Thank you for these fascinating videos.

Was wondering when you would cover this.

When Deep Blue beat Garry Kasparov, some wag said: "Sure, but how did it do in the post-game interview?" Probably wouldn't be hard to train a neural network to give trite answers to trite questions, with a few quips thrown in. "Mr Deep. Can I call you Deep, or do you prefer Blue". "Whichever you like." "OK Deep, how do you think Mr Kasparov played?" "Pretty well - for a human.". "Why didn't you take his pawn at move 35?" "It wins at depth 6, but loses at 16. Humans are so slow."

The caliber of this video was very impressive!

Do Photonics require extreme low temps, such as q-bits do currently? Quantum bits receive noise from temperature, so those chips work most reliably in a very low temp, close to 0K. It ends up with a machine thats mostly multi stage cooler, with a chip on the tip. Are photonics the same?

Hi there, I really appreciate your content. Just a side note: I believe you meant 20 or 1 'pico' Joules / MAC and not Peta which would be about 278GWh = 19k households/year?

12:35 - That's really clever.... Easy to get bogged down by the "right and wrong" ways to use tools.

12:41 LET's GO, literally called it because rate coding is how our neurons are organized!

the part about alpha go and how many TPUs were used. it's no wonder i cant find anywhere to build AI for StarCraft on my PC at home. ambitious but just not gonna happen, it seems. nevermind that, this talk/video was spectacular and incredibly informative. thank you.

I wonder if it is possible to have multi-stack-layer of LED film to do the similar task, since LED can both emit light and also photoelectric?

Light doesn’t travel faster than electrons, but it’s the frequency of light that allows more information to be transmitted and processed faster.

love this video! cheers from California

I saw that analog computing could have conversion to digital and back between a few steps to recover accuracy, with some circuit tradeoffs

A few mistakes: 1. ML typically consists of many matrix-vector multiplication steps, not matrix-matrix multiplications. 2. At 5:02 you meant picojoules, not petajoules 3. As I understand it (not an expert in photonics, though I have worked on an ML accelerator), for a given level of accuracy a photonic matrix-vector multiplication circuit will consume more power than a digital one, mostly because of the digital-to-analog and analog-to-digital steps. So I think it's somewhat misleading to say that power is not the problem. 4. I think the last point about replacing one of the axes with time is also misleading. That can be done for any circuit ("time-multiplexing") and will proportionally decrease throughput. So it's far from a solution to the density problem.

Woah this is blowing my mind! It's amazing, the things nature has provided for us humans. And the human scientific collaborative effort never ceases to impress me. Thank you for this video.

I saw an article in high school in the early 2000 on photonics research for replacing buses on motherboards from MIT. The idea was to reduce heat loss and latency.I wonder if this is an offshoot of that research?

Fascinating. Thank you!

I created a Neural network I trained to work as a Binary ALU. Even better for this I trained "Cells" which act as logic gates and I would love to see my data encoded into glass such that it could function as a full on ALU in light.

Great video. I enjoyed watching it.

You are producing some excellent content, never a dull video…thank you!

It would be interesting to know some numbers. So far as I can find out, Google's TPUs use a little bit off-standard 16 bit floating point format for all of their data. You don't need the high accuracy of a 32 or 64 bit float, at least for inference. If the silicon photonics ADC/DAC has an effective end-to-end precision of an 8 bit float, then the gap between them and what is useful for AI is very, very large. If its equivalent to 12 bits, then its not as much of a problem. The other thing that would be nice to know is how much process variation there is in individual interferometers. One nice thing about digital electronics is that you fabricate a chip, you test it at speed and if it gives you the right digital answers, the chip is good. With analog electronics, you might have an interferometer with a reliable 32 bit equivalent signal to noise ratio, but non-linearity and variation between interferometers on the same chip might push the effective precision way down into the single digits, especially with a light path passing through multiple optical elements, testing every possible light path may be functionally impossible. With digital electronics, all you have to know is that one device's output falls within certain bounds to know that you can chain together an unlimited number of them with no loss of accuracy. With analog electronics, chaining them together always compounds the error, whether its RMS error of the noise floor being added or the multiplicative error in the actual signal. Anyway, I don't expect answers to these questions, but they are questions that I think the answers to determine whether digital photonics will be a thing in the future.

Are these light matrices forming AND, NOR, OR, XOR gates, etc? Or is this a different type of computing that isn't "Turing style" ? in other words, are neural networks different from these logic gates?

In the challenges section what does John mean when he states that the photonic chips aren't used for training, but only for 'inferences' due to their lower accuracy? Great Presentation btw !

Paralell operation can be achieved using different frequency lights on the same chip simultanously.

excellent summary. thanks

with what you said at about 10:10 with analog not having the accuracy, could i theoretically use multiple streams of analog to convey greater resolution, like if i want more accuracy, i can have a 1's channel and a 1/2 channel for 2x the accuracy like how binary/any other number system has powers. id imagine if the error is on the reading side and not the light multiplication computer part then that could work(i dont really understand light, spooky stuff like magnets and electrons), however in a op amp implementation id imagine the lower places would leach more noise as they have heavyier weight.(could be reduced by differential noise reduction, but emf is unbeatable) another solution would to fill up the smaller place with value, then as you fill the first place, use the second as an extension to allow you to stack the channels into one massive in depth channel made up of many streams of light or whatever the medium. Idk how any of this works whatsoeverbut i would love to know if this method is of any use for improving accuracy at the expense of complexity

Lightmatter has an upcoming talk about doing this at wafer scale coming up in 2 weeks at Hot Chips. I hope you can tell us what they're up to! (And Ranovus).

8:23 - I'm not very surprised. I figured it could be done. That kind of manufacturing isn't in my wheelhouse. But, architecture is, haha.

I think there is a bit of an error in this video. the MZI is a passive device which uses a half-silvered mirror to create interference patterns, so there is no voltage applied to the MZI. What I suspect you may be referring to is the Kerr effect where the reflective index changes with applied voltage, and if you used it with an MZI then this is likely to be what gives you the desired properties.

5:03 Are you sure it's petajoules? That's the energy equivalent of about 1 megaton of TNT.

Holy crud.... That is insane. This reminds me of the x64 jump, and I think it will be as, if not more, significant.

I'm far, far away from this area of study, but I am an EE nonetheless... could they replace the "thin film heater" with a Piezo element on each of those interferometers to slightly deform the one leg? This stuff is so cool.

Wow...this is incredible!

I wonder how this compares to the analog circuits that are being used to run neural networks.

I see this being a much more viable path to future computing than quantum computers. Even if the chips are substantially bigger, they'll use far less energy and won't require cooling in the same way as traditional transistors. I think it's really exciting and I hope to see this continue to grow and advance!

OK. It seems like old video but also just released. It will probably be fantastic.

Small detail, but electrons don't move through the chip/wires. They just wiggle around; the energy is transmitted over the electric field around the conductor, not by the electrons. Doesn't really matter as your point is still valid, just a technicality.

One thing that seems to be overlooked in the video is the use of a nonlinear activation function as part of the computation. I do not think matrix multiplies all by themselves give the desired effect.

What would be pretty wild would be using both time offsetting and wavelength multiplexing to increase throughput. If I understand it, it would be like light based hyperthreading, except you could do 3, 4, or more threads all independently. I guess it would just rely on how passive the structures would actually be.

There's some inconsistency in the argument. At the start you note that most energy is lost on data transfers, yet these are untouched by the photonics, they tackle the multiplication instead. And I can't help but notice that important part of the photonic circuit is a heater, presumably to affect length of one of the paths, adjusting the interference. So while there seems to be obvious advantage in speed of the multiplication itself, it's not clear how much if any energy does it save.

would you consider investing in which company making some of the axes for the photonic/silicon era?

Hmm I get the hunch that these chips will go the way of transputers, as per nice idea at the time, but I feel something far smaller is around the corner. There is a Chinese researcher who is doing it on the molecular scale using crystal lattices and the doping of them with different atoms in order to change their topological properties. The light therefore behaves according to the structure of the lattice. You can us entangled photons to send signals and the photons themselves are in a squeezed light state. I understand the doping is done using a femtosecond laser. I'm a little unclear on the details but it is the leading edge in photonic computers. The work is published in respectable journals but still highly experimental.

"old" silicon can still compete well into the future, well, if we replace silicon for another material (yes that is in research since +10 years) we could get the same result with less electricity used, thus faster and more efficient computers. The "Real" fun begins when scientists achieves room-temperature superconductivity, that would enable computers running Much, much faster than current computers while using close to zero in electricity (as superconductivity would allow electrons to flow with no resistance, thus using electricity solely for the calculations/data movement through the material)

If accuracy can be improved and scaled then it can be used for inference?

This is some real Metamorphosis of Prime Intelect shit. Also this chanel is great keep it up 👍

Excellent channel

Wait, there is a 2-months old comment but the video was released like 20 minutes ago ? what kind of magic is this ? 😀

6:19 voltage potential is also transmitted at the speed of light

Thursdays I fry my brain with First We Feast in the morning and then educate myself at night with Asianometry.

Uhmm, that photon will be generated by a LASER that are known to be a big power churner, so how does that make them better?

Have they tried encodings that use transitions rather than levels? Id guess probably so but going off the graphics in the video it makes me wonder. For instance, no transition = 0, transition = 1, and the level is never really looked at just edges. Or pulse of light = 1 and nothing = 0. Probably both cases have a start byte for Synchronization. But the pulse idea has an issue that switching speed must be way way faster than the previous techniques... more used for low speed power line communications e.g. like DALI (the same wires that power the device can short the wires together instantaneously for communications, while the power rails filter out the communications with L and C. There's also positive edge = 1 negative edge = 0 ... that also makes you have to be able to switch much faster at TX side but also easy on the Rx side less error prone. I'd assume error correcting codes were used??? Also possibilities like QAM constellations can boost bandwidth exponentially but SNR must also increase. You could even construct more constellation points in more dimensions than 2 with multiple light frequencies on the same line Maybe a trial and error thing after these results figuring out what schemes make the systems fastest I suppose

Congratulations for the very interesting and informative video. However, I guess that probably you meant femto rather than petajoule per MAC. Furthermore, the speed of light is invoked inopportunely both for justifying the very large frequencies and the short time of computation. In optical fibres light propagates 1.5 times slowlier than in empty space, and in SOI waveguides it does even 2.8-3 times slower; by contrast, the RF or microwave signal in a modulator travels faster. And in general, electricity propagates to a speed comparable to c, because it's the electromagnetic field to propagate it, not the electrons in the metals, which, as a whole, drift by cm/h (when applying DC). The point is that in photonics you use dielectrics like glass, silica or intrinsic Si, therefore absorption is much smaller than in a conductive material; this would be evident if the same circuit were implemented with microwaves on a microstrip. However, the problem with metal connections when using high clock rates in digital circuitry is that you have to charge and discharge the parasitic capacitance of those lines. About the second misconception, an electrical circuit would be much slower with respect to the MZI mesh because of its RC time constants: it's not that the electric signal propagates slowlier, it's that the transient is much longer. Regarding the 1980s Bell Labs research you mentioned, I guess that they made an optical computer, I doubt (but I should check) that it was based on optical transistor as that technology is still at the proof of concept phase. However, this does not change your point, that announcements about silicon photonics neural networks replacing TPUs must be taken with caution.

"at the speed of light" has nothing to do with bandwidth. Electric fields also propagate at the speed of light.

Correction: high speed transmission lines of electricity also travel at the speed of light... electronic circuits are physically electromagnetic waves, just much of the time circuit theory is an adequate tool and very simplified. While the overall movement of electrons themselves, ie, electron drift, is incredibly slow (shockingly slow if you didn't know already) Typical FR4 dielectric on a pcb slows the waves by a factor of 4. Fibre optics also slow the light. Depends on the material how much

6:20 - Wrong. It's not strictly because the signals move with the speed of light. Electricity travels at about 270000km/s, really close to light speed. What is really holding electronic processors back is the fact that electricity generates heat. Exponentially as you increase the throughput whereas light does not.

I have no idea how any of this workd, but maybe something funky multiplexing could happen to this sector, similar to how DWDM re-uses fibre optics and multiplies their bandwidth?