Layered Depth Light Field Volumetric Video 

Thanks!! I hope you're right!

@@JoshGladstone Maybe with AI it will be possible?

@@rubi-w- This technique already uses AI, but it's a fairly active area of research and there are advancements every year. The main issue isn't really the capture technology, it's demand. Volumetric content requires a different technology to view than a normal screen, so there's not much demand for it right now. Maybe as VR/MR headsets get smaller and more popular.

@@JoshGladstone Thank you for the clarification ^^ Gonna force my friends to buy a VR/MR headset. Already own a Quest 3 but am planning on switching to the Quest Pro. (Maybe it’s better to wait or save up for something else?)

@@rubi-w- Unless you specifically want the face and eye tracking, I think Quest 3 is better than Quest Pro

Great, so glad it worked! Thanks for checking it out! :)

Thanks!!

I appreciate that! Thanks for watching! :)

Thanks!!

Glad you like it!! :)

I think you meant "Post a comment if you have any thoughts or questions ...... that I may or may not answer"

Yes! It is very similar to the Google Layered Mesh representation, although the implementation and playback part is pretty different. You can PM me on instagram if you want more info

That wasn't the part of my comment with the question mark 🙂@@JoshGladstone

Thanks! At the moment any movement whatsoever requires a completely new colmap solve to yield good results, so ever shot does need its own solve. Still working on that though. My earlier rig did have a wider baseline between cameras, and that did seem to help with objects further away, since there was more parallax differences, but it then struggled with closer objects. This rig is my attempt to split the difference and also have something more portable. Ideally sync would be at the sensor level, but really it's only an issue with fast moving subjects at the moment. If I'm able to somehow get moving camera shots to work, it could also be an issue there too, but for the moment sync is alright for what it is. I also need to experiment with shooting at higher frame rates to try and get the sync closer.

Check out lifecastvr.com, I believe they're using rgbd + stable diffusion to inpaint a static background layer.

Thanks, I appreciate the kind words!! Actually, my first rig was a 1ft by 1ft square, so the cameras were much further apart, and it was able to get good results at some further distances, but closer captures were challenging because a lot of the image was out of frame and there was no overlap. This rig is an attempt to be able to capture some closer subjects. I even had one version that had the cameras spaced even closer, so this is sort of the middle ground between the two. I'm still optimizing, but yeah for distances it's definitely better to space the cameras out more. It would probably improve things to use more cameras, but without a better way to control all of them, it's sort of a balancing act between capture quality, usability, and portability. It would really help if an action camera company got back into the wired array game, but that hasn't been a thing for years. It's an extremely niche market. That's one of the reasons most camera arrays are studio only.

Yes, Gaussian Splatting is very cool! In fact, the footage of my GoPro rig is a Luma AI gaussian splat! One of the issues with gaussian splatting and nerfs in general are the number of views needed to get good results, which is a challenge for video content. There has been some work in nerf video (Neural 3D Video Synthesis, DyNrf), but real-time playback is a challenge. It would need to be baked down into another format that can be streamed/downloaded and played back, and then you can lose a lot of the view dependent effects, i.e. the nerfiness. It is an active area of research though!

This project is specifically for planar content. Google's light field project was able to do 180º content, but they had a much different setup and different neural network. Planar content is simpler.

I haven't, but it wouldn't work with this technique. It probably would with my previous stereoscopic multiplane project, though. But the depth sensor wouldn't factor into that. I can't think of any projects that take stereoscopic images + depth as input.

At the moment, it needs more than two cameras. But my previous project was based on stereo pairs, check out my older video "Multiplane Video - Volumetric Video with Machine Learning". I even converted some historical stereo photographs to volumetric, which is also available on the channel. It's certainly possible that future projects could produce similar results from stereo images. In fact, there are projects currently in the works to produce similar results from a single image. Hell, there are even current generative AI projects that aim to produce similar results from 0 images! Crazy!

Splatting tends to need a lot more views to get good results. It's also an open questions on how to play back gaussian splats in real-time as a video. This can be wrapped into an mp4 and streamed on mobile hardware.

I did use guassian splats for the video though! The shot of the camera rig is a Luma AI capture.

I submitted it, but it still needs some fixes before it's able to be published, unfortunately.

The Lume Pad 2 version is now live in the Leia App Store!

Thanks! I haven't seen any samples yet, but Apple's spatial video is just stereoscopic 3d wrapped into hevc. There's no volumetric or 6dof aspect to it, so once the format is parsed it should be relatively simple to view on any VR headset.

@@JoshGladstoneThat was my guess as well. But rumor has it that it will incorporate depth from LiDAR and/or some AI trickery to give it some 6dof flavor. It fact it would be rather disappointing if Apple would end up with only good old 3d video.

@@Naundob The frame it's displayed in has some 6DoFiness to it, but it's definitely just 3D wrapped into hevc in such a way that it's backwards compatible with 2D playback. There's more info here: developer.apple.com/videos/play/wwdc2023/10071/