That’s mind blowing. If we can manage to get AI to develop on the fly meshes to manage the motion and keep the textures coherent, you could automatically convert most pan shots into a fully 3D rendition of any scene captured…. Allowing you to walk into a movie, as your watching it and see it from any perspective shown in the footage.
It’s actually even better.. but yeah. And I did something similar over a year ago using LiDAR and pointclouds ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-UKAguEUb2Lc.htmlfeature=shared
It didn't even occur to me that we could use this for footage in the future so you could fly around a real world location in "real time"(probably offset due to the required rendering).
Gaussian is "gow-see-ann", even though it looks like "gawsh-ee-ann". Just remember "Gau" rhymes with "Tau" (which rhymes with "cow"). The name Gauss rhymes with house.
Loved the video! I wasn't aware that we could do the training with old gpu! I've read at some github that the minimum required was 24gb! I was completely disappointed hahahah Please do the tutorial you mentioned! Subscribed =)
I'll have some time this coming week! Which GPU are you running? SIBR viewers was where I was getting stuck. I had to use Nerfstudio because SIBR required my CUDA Compute Capability of 7.0+ whereas my 1080ti was only a 6.1 (developer.nvidia.com/cuda-gpus).
@@RealityCheckVR I'll upgrade it to a 40xx series, I'm currently with a 1060 6gb... I wanted to upgrade to a 4070, which would be really cool with my current setup. if I had to go with a 4090 I'd need to upgrade the power supply, the case and most likely a water cooling system... It really adds up. But I really wanted to dive into this new tech so I'm really uncertain about the investment.
SUBBED!! Caleb, how do you think the video-making will progress? By using multiple cameras and generating a gaussian Splatt for each frame of synced video, and then somehow "playing the 3d environment forward", or some other way!!?
Surely there will be advancements that will change the workflow up as things get better and ideas are shared. Still, as I thought exercise it's fun to think about how it might work. For example a basketball game could have hundreds of little cameras embedded around the arena as well as Lidar sensors making the process even more accurate. Each camera system would be mapped out virtually so that the software would understand how the images overlap and where objects are correctly in 3D space. Basically, the "colmap" process would be done ahead of time. The system would have to be able to account for moving objects with a simple timeline feature. It takes my PC about an hour to do 30,000 iterations of a good-sized area. I can assume that you wouldn't need to do this many iterations and could probably get a pretty decent-looking picture after only a few iterations if the tech is advanced enough. The static areas could all be baked in and the processing power could be spent on all the live-action movement. As long as the mainframe is powerful enough it could hypothetically power everything in real-time and offload proxy data for everyone to jump in and fly around on their cellphones. I'm sure I overlooked somethings in my explanation, and maybe it would be much simpler. The point is, we can do this right now with enough time to simply integrate it!
Very cool. Would you care to walk us through the process of capturing (smartphone?) and processing the capture media ? have you made a tutorial on installing and using Nerfstudio?
very thorough video! Caleb, how do you see the bandwidth requirements for end users will change with this type of volumetric imagery? Where do you see the render will happen, can user devices handle it?
Thanks for checking it out! I think we will have service providers that will be able to have a master feed which users will be able to "plug into" and control wirelessly. All future decisions are always based technical feasibility combined with "LET US HAVE ALL OF YOUR DATA!" 😅
I think what would get rid of all the fog is if it was able to capture the full volumetric video and not have to stay steady all the time and going around the subject
It can be used with UE5 now but there's long way to go to efficiently make actual game with it ( collision, having a competitive result when responding to lighting, etc)
I believe this will be automated very soon, there is still a lot of refinement to be had and right now we have no standardization. Exciting times for sure!
Currently the training as well as viewers are coded for still images. They will need to create a timeline viewer that can update and store data points so that you can view video in real time. We have all of the building blocks to make this tech possible right now, we simply need a venue with the right person to get the first use case up and running and others will surely follow suit. Or more likely some company will perfect the tech and run around installing it as fast as possible before the copycats show up. EDIT: I realize you probably meant just simply using movie footage to recreate certain areas, and YES, you can absolutely do this especially if it shows off a scene without cuts and people are still throughout. It would work great I imagine. Exciting times!
@@RealityCheckVR This requires a really beefy computer and GPU, right? Or not even run on a PC but cloud computing? Trying to get an idea about how far off this is from everyday use.
@@gblargg I run this on a 1080ti GPU from years ago in combination with an i7 4790k CPU. I can create a Gaussian Splat in about an hour for 7k iterations.
