Light Fields - Videos From The Future! 📸 

This is absolutely incredible. Imagine a movie where you can actually look around the scene! What a time to be alive!

This is one of the few papers that really made me think "this is the future". Feels like wizardry to rotate and pan around a video, rather than a still image. Not a super practical thing, but it's dang cool!

Moving the camera while seeing paused motion blur is such a cool effect.

"Hold on to your papers" that joke always makes me smile 😂

Imagine People walking around with that Monster taking Selfies 🤭

This was exactly the sort of thing I was looking for after the last video with the video selfies. This looks like an incredibly promising way to make VR video more immersive than ever before.

I could see this being used for movie production; imagine being able to film once and then tweak the camera movement in post production.

Download the VR version from the website! I realised this was the first time ever that I could experience *true* 3D video (not just stereo). It is something else! I kept wanting to pet the damn dog! :D

I just tried their demo in VR and it was very impressive. Things felt so real, like I was there (I know, of course it's VR, but still!). Now I am sad, that I cant watch every video like this. Also the one sitting at a lake or at the ocean were very relaxing. This has definitely the possibility to be used for virtual tourism. AMAZING!

it feels so weird for humanity to be so perfectly preserved for the first time ever

at each video, we get closer and closer to magic.

1:27 That animation was so cool!

I first saw this tech used in vr with google lightfields (on steam), if this becomes affordable I would be really happy

Imagine watching back your memories played back in that.

Wow, add tobii tracking device and this will revolutionise the media!

My jaw dropped when I saw the dog behind the fence! It was almost perfect, such an amazing implementation.

This feels way more enjoyable to me than 360 VR. I like the idea of a 3D video space where there's parallax if I happen to shift in my seat or tilt my head. And more importantly, the viewer has an actual sense of scale and depth- They can lean in to get a closer look at something in a Maker video, for example.

This could work to accelerate VR-as-a-service platforms, where the remote game engine renders an entire light field for the approximate location of the VR headset, which is delivered once every second or two with extremely high compression. The client's PC decompresses the light field and maps the actual head movements and rotations onto the now-old data, simulating a highly responsive remote rendering service of a semi-static environment.

watching these videos only for "What a time to be alive!" words

I really hope this leads to better predictive image in-painting soon. Having something of this quality on a phone with minimal hardware changes would be absolutely incredible

For VR, it's not just about rotating your head, it's 6 degrees of freedom. The image is still correct when you move your head and this is why lightfield images feel so real in VR, it's like the realism of a photo/video combined with the freedom of real-time 3d.

There's so much development in this field, please, cover them more *waiting room for 5 papers down the line*

I watched this entire video with my mouth gaped in awe. I cannot wait until 2 maybe 3 papers down the line where the artefacts are less noticeable and the content may be optimised in a way so that it could work over a limited connection such as mobile data. Hopefully another 5/10 years down the line we will be able to achieve the same camera angles all available on your smart phone. who knows! truly what a time to be alive.

Having both played Cyberpunk and just seen these videos on my VR Headset, it is almost frightening how close this is to a supposed Brain dance Experience from the game. The indefinitely repeating glitches on the side of the view field make it even more realistic. Awesome how close it even is from being actually there in the scene. Just want to up the resolution a notch. Now you have to interpolate space from only two cameras and their movements in space, get them small and fit them into eye implants. Voilà, Kiroshi eye implants from the game.

I really can't wait for this to catch on. True VR videos? YES PLEASE!

I've been having this idea for years: Take a huge concert in the daylight. The lead musician asks everyone to take a short video at the same time, for example of a jump he does or something cool. Then these 30,000 people upload their video on a given platform (RIP 4G antennas). => You get a messy, inconsistant, but free and unimaginably huge dataset you can perform 3D reconstruction on! After a lot of cleaning and reconstructing, you could walk through the crowd, turn around the stage, etc. I know some stadiums are equipped with 4D Replay or other technologies that allow for similar experiences, but only for a very limited portion of the venue.

Wow that was amazing.

For anyone wanting to view Light Field photos (not video) in VR, Google released a free app for SteamVR in 2018 that's pretty awesome: store.steampowered.com/app/771310/Welcome_to_Light_Fields/

I think lots of people on here are confused about the difference between 360 video, photogrammetry, and light fields. It's not just about stitching multiple videos together and slapping it on a 3D model. A true light field is the 5D value of every point of light within a volume. They are capturing a few 2D slices (where the sensor collects the photons) and the algorithm is filling in all the rest for the (hemi)spherical volume. Then it's compressing that down into something that can be streamed, which is probably a 1,000:1 ratio. It's like compressing an IMAX feature film into the size of a JPG file!

Light fields are amazing - that's the field in which I'm doing my PhD. There is so much promise in this technology, and we've come a long way since the brilliant Lytro camera.

This technique with 360 degree viewing looks like it would allow for really immersive VR movies

This needs to get promoted big time. It needs a web platform providing content from spectacular places around the world. It needs a dedicated app for the wide spread use with mobile devices via browser and cardboard experiences. Go to rallies, fairs, TV studios, landmarks and great vistas to create a bulk of content. How odd that the relatively young concept of wide angle stereoscopy already seems to have found its match. I'm curious to see what's the future will hold for this technology.

Imagine now how Walking Simulator games, would be! WHAT A TIME TO BE ALIVE!

What's weird is that the camera of the future looks a lot like the human eye. What a time to be alive!

The Google lights fields demo on my vive left me wanting more amazing light field content. I'm happy to see the amount of progress being made in this fascinating field.

There is a Demo for this in VR, which makes it even more impressive, as the content can be rendered in stereoscopic mode, because all information necessary for that is included in the dataset.

This is actually the future. Can't wait to watch these kinds of videos on my VR headset.

I just realized that it makes a lot of difference in watching VR. It is the small head movements that do not show in the image that cause nausea and break the illusion of reality in VR. So yeah, much more relevant than a silly 3d effect. Remember the head pivots around the neck joint, not the eyes, so there is always a parallax effect that needs to be handled even when the viewer is sitting still in the vr scene while just looking around.

Imagine they implement this technology in VR movie experiences where you become a character inside a movie like a horror one for example, and than you experience the entire spectacle first hand.

This domain always has been fascinating to me. I once reimplemented on GPU the method of the paper "Soft 3D Reconstruction for View Synthesis", and it made me learn so much about light and depth.

Virtual theatre plays! Live streamed in VR! Potentially a great way to keep theatres open during a pandemic, if only this were commercially available

This is so magic… I was think about this yesterday and don't know what keyword I should search in Google, then you creat this video today! WoW!!

360 degree cameras can accomplish some of this, which is ok, but we definitely can improve.

I can already imagine how much more immersive this will make VR videos. No more of that jarring/nauseating feeling from having the world shift with head movement.

Here is how to do it with a single market-ready phone camera: create a camera app that when you click the button to take an image, it starts filming a video, allowing you to make a verticle circular motion with the camera in a 2 foot diameter, then you click the button the second time to take the photo. Then the camera app constructs the 3D image using the algorhythm and the images from the video. You can fight artifacts and blurring by letting the program draw detail information in a single 10x10 pixel area from a single frame that most closely matches the surrounding pixels.

To those who like this idea - take a look into Lytro camera. They managed to produce their first light field camera model available for purchase and then they went under. Not too many people interested in the technology. With this special camera you can capture more information and store it in a file that needs to be reconstructed in their software, where you can chose the angle and focus of your picture and then export it in a normal photo. This is mainly why nobody was interested. Other than that - this technology looks like magic compared to what we are used to. If 3D TVs didn't die out - this would have complemented them very well. But nobody wanted to invest more money in 3d. Companies that worked on light field on 3d TVs with no glasses needed also went in oblivion. If that didn't happen - today instead of fighting over hdr- we would have be looking at TVs with popping picture you can look at from all angles. In VR this light field technology allows for removing the bulky heavy lenses that sit on your nose and thus making VR glasses as thin as regular glasses. Also - this technology allows for your eyes to move around and focus inside the image, making the experience much more realistic and less vomit inducing. However - Nvidia - the only people behind it - also backed out and stopped developing their 3d and VR glasses. How about that?

I thought light field photography was an entirely different technology - one where you capture, well, the light fields, and can do things like adjust the focus after the fact.

I do think we'll see bitrate for these formats decrease in the near future. All current codecs including H265 are made to compress a series of nearly similar 2D images through time, not to compress multiple parallel streams. Just like how there's similar information from one frame to the next, there will also be very similar information from one stream to the next in this case.

The moment you mentioned reflective surfaces I caught myself pretty much only looking at the mirror in the shop. :D

Reminds me a bit of the Lytro Lightfield Camera. Tiny box capturing all incoming light and lets you change the focal plane as well as your point of view by a few degrees. Imagine what you could do with an array of Lytros and VR+Eyetracking (incl. pupil dilation to calculate the eyes aperture to adjust depth of field) for replay. Total immersion

Can't wait for the Dall-E paper from OpenAi (and its video)

There is a studio in France that uses a ligh stage and shoots 3D video. So they do a photogrammetry reconstruction for every single frame.

That's an old stuff that's going with it's roots up to Lytro cameras from 2006. But still the stuff doesn't stop to impress and be "the future" for 15 years straight in a row.

AI is going to teach us something about light we didn't already know existed in real life

Can't wait to see a paper that defines a new compression method specifically for light fields. H265 is good but I seriously doubt that it's designed with light fields in mind.

Combining image inpainting with the data of two or three extra cams seems like a natural next step, imho.

I know that Cyberpunk has been slated - but this reminds me very much of the braindances from that game - you can explore previous memories & events in a similar way to find out additional clues around the scene!

This would be amazing on a flat screen with a head tracker. Or even better one day if they can make a light-field display, i.e. a two dimensional array of tiny projectors that project directly into the eyes of any number of viewers.

Your channel is amazing man really make it enjoyable to learn and it’s interesting and needful concepts!!!

Fecking amazing! Tutorial videos (and all videos/movies) of the future will have so much more!!

I can see this being useful for video/movie editors to perfect that camera angle / pan after the fact!

Can’t wait to watch VR movies with this technique in the future.

This, in VR? Imagine watching your "old photos" this way. Popping a VR headset and just being there.

I can see this becoming a new method of viewing content... like video over books.

Something I'd be really interested in is AI video and image compression. Imagine being able to compress videos much higher than normally possible and still retain most of the detail. Stuff like that is kind of happening with NVIDIA DLSS (which upscales gameplay resolution from like 1080 to 4K), but I think that making a system dedicated to compression could be very cool. I'm not sure if this already exists, but I think that something like this will definitely be used in the future. Imagine being able to watch videos and movies while using 25% of the data. That could save a lot of money in ISP bills.

This is looking incredibly promising! I still remember seeing a demo at a conference of what LYTRO was doing with lightfields (2016-ish), I was blown away at the time. Too bad they went 📉 but I'm glad to see lightfield technology still going strong

This totally fucking rocks! Just awesome to see where this will go for entertainment and teaching/instruction.

We may need a custom entropy encoding format that look across all view ports for these highly coupled videos. Most of the image across different physical or logical cameras are essentially the same. Only some highlighted area shown in 1:44 are actually different so we can skip other part and only keep a key frame of it.

This is essentially what Lytro did years ago, but they fucked up the business when they tried to "sell" it to photographers as after-shot re-focusing and changing DOF when instead they should have pitched their product to the 3D-related communities and show that you can do more than re-focus or re-bokeh. It's like inventing time traveling and your strongest selling point is that you can go back 2 days in time in order to buy the t-shirt at a lower price. Not to mention, photographers would never trade their smooth bokeh fast and sharp lenses and the full-frame high res and high iso sensor to a Lytro optical system. My lenses focus so fast, that I never need to re-focus. Instead, the ability to capture light fields, essentially 3D scenes would have been praised in the game dev world or the vfx side of cinematography. They even had a video camera capturing 3D videos in which they've shown how they could easily insert geometry that fits with the scene, re-light the captured scenes and more. And this could have extended to even more amazing use-cases. Now facebook and otoy are doing it, intel is doing it and seems more and more giants dig into this tech. It's a pitty what a massive opportunity Lytro lost, they could have been market leaders by now, perhaps even creating glass-free 3D screens to preview their captures and push this tech even further.

The next step has to be doing this with fewer cameras. Being able to do this with 5 cameras would be a HUGE step.

Soooo, photogrammetry applied to each frame of a video scene captured from various angles to reconstruct the 3D scene?

my thing is, yeah we HAVE 3D videos. but the perspective is fixed, and often completely wrong. (at least for vr). but with this, the perspective is never wrong! and everything is correctly sized.

yeah I checked this out in VR a few months ago - absolutely the future!

Light Fields aren't new. But I have to admit that this is some outstanding work. Even so a camera sphere of half a meter is some what impractical. Usually small high speed light field cameras are used in the industry for quality control in fully automated systems.

The cameras and original raw footage are huge impractical for any form of wide consumer adoption but perhaps for the largest of film crews. A different approach that would solve these two issue is to crowd source cell phone photos and videos of popular sites to create a cloud based 6dof 3D model of the background scene in any direction and then to insert any live action into that model from the cameras already in the form factor of our phones. When 5G networks become widespread you might see the creation of something like this to relive your memories in fully walkable 6dof VR, etc.

Now THIS is something I am genuinely very impressed with - I was already impressed with the Tom Grennan video on the PSVR, but that was a lot of work. This does it completely automatically. I'm definitely going to keep an eye on those papers.

Still love hearing the word "doctor" at the beginning of his new videos!

Our world contains a lot more information than regular person can think of. Imagine capturing all of it :)

This is sort of what you would need for "glasses free" 3D TV. You cannot transmit 20+ parallax views in HD (or better) given the limitations of network transmission. You would need to transmit some images and some 3D geometry and then re-construct some of the parallax frames.

Being able to capture (or capture some and fill in the rest w/ AI) such footage is important for when VR tech is ubiquitous

This is like Photogrammetry but for video, very impressive

in VR this is mindblowing.

just amazing. I loved Google's lightfeild demo, I wonder if ai volumetrics is less computationally intensive to play back live than Googles implementation

yes

I'm convinced that I have seen a paper which combined multiple video streams into a light field video. And it was made to use standard video decoder hardware in creative ways to accelerate this process. I do believe it was presented by Disney, but I have not been able to find it again so I might have dreamed it. :-P

Absolutely stunning work.

It's like we have the real world on your computer

Would be cool in an interactive detective movie, pause time, rewind to inspect etc...

That's awesome stuff. I look forward to the hardware for it becoming cheap, commonplace, and integrated. I'm also looking for some more solutions to data compression. Have the AI scientists designed something for general compression of noisy data yet?

This tech seems like something, that will consume lots of energy, if whole movies on Netflix were streamed like this.

I bet within 15 years, someone will make a movie like this, so that you can watch 180⁰ with the 3D-feeling of being there. - Perhaps with haptic feedback? Imagine that.

Omg the reflections updating blew my mind the most

If the video encoder can treat the different perspectives as the same video, the compression rate can be much better. There must be so much similar information.

This channel is basically a window into the future

Check out "Welcome to Light Fields" on Steam for VR

Two Minute Papers is a bit late. You were able to download and watch these exact same lightfield videos in vr since Siggraph 2020 in August, via the github link in the description. That said, they are extremely impressive! They're the only 6DOF VR videos you can see today.

This reminds me of a kickstarter camera where you could change the focus of an image After taking the image. No idea what happened that tech but its very similiar.

*braindances* seem much closer now than 2077 🤩

Of course this will be done in the future with a bunch of tiny drones with wide angles lenses that position around the subject and follow. I give it 5 years top until it is commercially available.

So if I had eye's like a fly this is what it would be like to see through them, amazing