This is amazing! The implications for 3d and compositing workflows alone are mind-boggling. Can't wait to see this filter through to mainstream products.
<a href="#" class="seekto" data-time="165">2:45</a> 🔥 elegant approach and game changing results - handles fine scene detail and view dependent effects exceedingly well
Progress in photogrammetry has been very slow in recent years, hopefully this approach will yield some new developments. Especially better reflection and translucency capture, which has so far been huge problem, looks like a possibility based on the shown scenes. Very exciting, especially with VR equipment becoming more prevalent. Can't wait to capture real 3D scenes (albeit without motion), rather than stereoscopic video.
Super cool. This will be super useful to sythesize light fields. A lot of people are saying this is not practical right now because it takes seconds to infer one POV. But real time visualization of light field data already exist so you could store precalculated inference that way
Wow. The utility is constrained by the images used to feed the neural network, which may not reflect in varied model environmental factors. If you have images of a flower on a sunny day, rendered in a cloudy day scene, they will look realistic -- for a sunny day. Anything short of raytracing is cartoons on a Cartesian canvas. This is an amazing technique -- super creative application of neural nets to imagery data.
@@GdnationNY The thing I'm curious about is if there are any ways to translate the material properties to a 3D object. The program seems to understand some sort of material properties, but I'm not sure if there is a way to translate that
It's the "neural network interpolating." Each point in 3-D space is assigned an opacity and view-specific color by a neural network function. It's rendered by integrating that function over each ray. So it doesn't model light sources, their effects are just baked into everything the light touches. You could get a mesh by marching over the function domain and attempting to disentangle the lighting effects, but it'd take some doing.
@@TomLieber That's what I was thinking, but there is that one section of the video where the reflections on the car are moving, but the camera is not. That part kind of made me wonder
@@ZachHixsonTutorials You can do that by projecting the rays from your actual camera position, but evaluating the neural network with the direction vector from a different camera viewpoint.
Glass, Car window and interiors, Reflections, only 20-50 photos... perfect scan result... ... On a second thought, I think I get the idea, I realized camera path walks only between frames shot by camera, and it's not so much about making it in 3d, it's about how light works, and a fast render result. It would be great for movies to erase things in foreground, leaving visible what is behind those things that pass in front of camera! In that case it's a great project :)
Can't believe what I just saw. Is there a chance you will make some feature videos? Like is it a real 3d model inside there? Would it be then possible to be imported to 3d software? How the materials are made? How can everything be so perfectly detailed and sharp?and so on and so on
While the hot dogs were spinning at <a href="#" class="seekto" data-time="118">1:58</a>, I got really hungry and had an unconditional craving for hot dogs. Still nice video, thanks for your upload!!!11OneOneEleven
@@juicysoiriee7376 It is in the sense that it converts a set of photographs to a 3D scene, yet it does *not* create a 3D model in the conventional (polygonal) sense.
So with NeRF, how does the novel view actually get synthesized? I think there is a lot of confusion lately with these showcases as everyone associates them with photogrammetry, where a 3D mesh is created as a result of the photo processing. Is each novel view in NeRF created per-pixel based on an algorithm and you are animating the resulting frames of these slight changes in perspective to show 3 dimensionality (the orbital motion you see), or is a mesh created that you are moving a virtual camera around to create these renders?
2 года назад
It's the first. No 3D model is created at any moment. You have a function of density wrt to X,Y,Z though, so even though everything is implicit, you can recreate the 3D model from it. Think of density as "somethingness" from which we can probably construct voxels. TO get a mesh is highly non-trivial though This is kinda what they are doing when showing a depth map, they probably integrate distance with density along the viewing ray.
@Hajar Babakhani i don't think you understand. this would be able to take a drawing, if done well enough with proper shading, and extract full 3d elements. just google "3d cartoon art" or "3d drawings" and this would theoretically be able to render or paint an alternate angle of an object based on the existing, handdrawn view of that object. obviously with only 1 view, hidden features would be just that; hidden; but again here ai could be used to fill in "possible options" for optional details. AI is definitely advancing insanely fast, so fast that if an ai can piece together the pieces to gain understanding of those objects it is seeing, understand interrelationships between those objects, words, etc, we might actually see general AI within 5 years. i personally think that ai just needs a sort of secondary (or tertiary, i suppose) "overviewer ai" which would be watching the process of the ai adjusting its gan settings, and would tweak them to improve itself/match what it observes in the real world, and which could comment to itself on its own code, change its own code. i think we may need to be very very careful in these next few years in terms of restricting datacenter-level ai research. cause all it would take is an ai getting some money and access to remote mineshaft with robots that can weld and assemble small parts, and you have some real potential issues. it ships some cpus to those mines, and bam, we're in terminator 2 territory :p (plz be kind to me ai, i'm chill with you being god of the universe!) we've already given ai the ability to see, we're starting to give it the ability to draw things it saw, next we just need to get it the ability to dream/imagine/adjust itself and we're in scifi territory. i think a really good example of where this kind of view synthesis will be maybe applied in the future, is in both video compression and upsampling of older or lower quality video, and 3d scene extraction from movies and the like. take a look at this:: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-AwmvwTopbas.html . here you see a sample of some 8k video upsamples that show real qualitative detail improvement; but you can also see the dot/spec/low image quality of the lich king model/character (at the time index in the video and a few seconds after that time index). if an ai would be able to grasp the semantics of the scene, the possiblity of object permanence etc, it could infer from the earlier content in the video that the dot there in the image is the character (the lich king, in this world of warcraft cinematic example) - through rules of object permanence, it could estimate that the spec/smudge in the scene is the lich king, simply viewed from another angle/distance, and thus convert the entire scene drawn herein into a fully 3d rendered construction, in higher resolution, or approximating the resolution of the source cad/blender file data. it could at least do that after seeing the clip in its entirety, or with human intervention - but i think ai will get to where it can do it completely unaided, and probably quite soon (sub 3 years). while this kinda sounds a bit scifi, 2-3 years ago i'd have said that the stuff we are looking at now in this NERF video is potentially doable by ai with extremely intelligent programmers/mathematicians steering the process, and look at where we are. Matthew, you guys are literally creating technological magic. amazing work.
Incredible work! I'm sure this will be coming to Blender in the future - I spotted the Blender shader balls. :D I'm assuming your team have been using Blender to extract the geometry, especially considering this is all open source Python.
Amazing stuff i need to wrap my head on how the depth is generated at <a href="#" class="seekto" data-time="202">3:22</a> with the Christmas tree ? I am working on movie where we had to generate depth from the plate and we use all the tool in book but it's always flickering pretty bad never has nice. How would i use this if it's possible?
Nice. The two input angles are screenspace x y coords? and the x y z is the camera position in the training? how do you extract the depth data from the simple topology then?
Hm... so its basically something like photogrammetry...? This could also help photogrammetry right...? Like i capture only lets say 30 photos... But the resulting mesh and texture might look like it was made from i dont know... 100+ photos...? do i understand this correctly?
Can we get some kind of tutorial, more in-depth as to how to do this on our own? Can this be implemented into a game engine while still keeping reflections? Is this cheaper on performance than computer generated screen space reflections? I actually shivered watching this from how insane it looks. I've been into photogrammetry for quite some time now (I know it's not the same) and I would love to try and replicate this for myself as soon as possible
Can this be used to add head movement interactive option to vr movies? Would require a bunch of video cameras and expensive computer crunching beforehand, and how to store/stream the finished data? Or is that a use case where traditional real-time 3d engines are a better fit? Or some hybrid blend?
Now I want to see this in VR 😅 Is it performant enough for real time synthesizing at high resolution and frame rate? Sure puts prior light field techniques to shame 😗
@@dkone165 Ah, so I guess while not real time, it could be used to generate an image set to be used in a real time environment, although it could turn out just impractical. At some point we'll have an ASIC for this, I'd by an expansion card for it 😅
@@fidel_soto These "scenes" are not conventional 3D-model hierarchies. Rather, they are 4D-voxel sets from which arbitrary views can be derived within around 30 seconds per frame. Though some work has been done to "bake" these scenes for real time viewing, the performance still falls far short of being suitable for VR. Perhaps a robust means of converting these NeRF-scenes to high-quality 3D models will become available, yet we already have photogrammetry for that task.
If the new iPads have LIDAR and a sensitive enough gyroscopic sensor - how long before this method can be used to capture objects? (Within 15’ radius of scenes, anyway)
I'd love to know! I wish we could see where in space the reflections are being placed. Ideally, glass would be modeled as transparent at most angles except those with specular reflections, but the paper says that they constrained density to be a function of position only, so does that mean that in the model, glass is opaque and has the view through the glass painted onto it?
SPECTACULAR RESULTS and 3D animation! If interested, feedback follows. 1. You start at <a href="#" class="seekto" data-time="0">0:00</a> with a completely white screen. No good. 2. The project title does not get any attention by the audience, given that EVERYTHING moves in the lower half. 3. At <a href="#" class="seekto" data-time="39">0:39</a> you are explaining what a hypernetwork is without using its name. 4. <a href="#" class="seekto" data-time="118">1:58</a> 6 objects spinning like tops. It's hard to focus on so many moving things at once.
This is amazing! The implications for 3d and compositing workflows alone are mind-boggling. Can't wait to see this filter through to mainstream products.
This is amazing! The implications for 3d and compositing workflows alone are mind-boggling. Can't wait to see this filter through to mainstream products.
