I can't see why the two can't work off of each other. NerF still requires images to be taken like photogrammetry, just far less, and generates a full 3D view in high fidelity. Why not, then, extrapolate the additional reference points from the generated view to aid in creating meshes without artifacts? I don't see one replacing the other so much as them working in tandem for maximum results.
come to think of it, gaussian splats can be used directly for other things like rigid body physics simulation (easy to calculate object intersections or estimate center of mass - don't need meshes for that). Once animation is solved, i can totally see meshless gaming engines pop up in the next hundred of years
would be interesting to see how this may be able to take 2d video content and make it viewable in 3d in vr. Like imagine being able to watch a sitcom and feel like youre there in the room with them.
That would be awesome. I believe the biggest issue there is that *dynamic* scene reconstruction itself is already incredibly difficult with conventional methods (such as photogrammetry). It's new to me that NeRFs could do that now.
There is no way NERF can replace photogrammetry .... But Neuralangelo or Neoralangelo 2.0 most likely will :D ... I've been making nerfs for about a year now since it first came out and the quality is just not that useful in most typical 3d pipelines if you want high quality 3d models and texture accuracy. But Neuralangelo looks like it can pump out some nice geometry + textures. I am very excited about getting my hands on it.
@@merseyviking No, the image quality that you would be sending to photogrammetry would be much worse than just using photos. And NeRF does not create objects out of thin air. It must be able to see the object to construct it. The cool thing that NeRF does that Photogrammetry cannot is reflections. You can record the ocean, windows, cars even reflections in mirrors if you are smart about it. However perhaps you mean combining the tech to make something better. If so the NeRF math and Photogrammetry math can be combined potentially and that is something that is already being researched.
@@DirkTeucher I see a lot of people saying that reflection is something NeRF can handle. But it only seems to be the case for the NeRF view, not if you convert it to a mesh. I used LumaAI alot recently and the reflections were all nearly as bad as a photogrammetry approach.
Now I need a deeper dive into the subject. Mostly what I got out of this is that Nerf is AI enhanced photogrammetry that uses a different file format. It has to be more complicated than that. There must be a reason the people that developed Nerf, didn't use an object mesh as output. Isn't the situation similar to a RAW image format like DNG vs some other proprietary format like CR2 (Cannon) or RAF (Fuji)?
A NERF is a fundamentality different way to represent a scene, it's not just a different way to import a scan. To convert a NERF to a traditional mesh you need to go through a process of similar difficulty to processing a photoscan in the first place.
Many of the issues with photogrammetry mentioned here can be pretty easily overcome. Reality Capture, which Epic games recently bought to allow game developers to create assets for UE, has some great features to model shiny objects easily. And the latest version is MUCH faster now. What takes Meshroom 8 hours to do, RC can do in under 2 hours.
I would say this woulld be the evolution of photogrammetry because it saves time and will get better overtime allot faster. Where the other is at its peak and cannot get better as fast over time. Only things it will be able to do is create more detailed models but it requires more work and better tech. Nerf would also be great for indie companies and creators so i would say if this is accepted by individuals and industry. This would be a better option especially since nvida and others are already working on this tech.
As long as the data exists in the training databases, or even real time updated data sets for scientific purposes, the generative transformers can recreate, using standard models, the attributes and effects of the missing parts, or by direct request of the user. For gaming industry, it means that if any part of our cosmos has been measured and scanned with various methods of source collections: images, lidar, telescopes, microscopes; these generative transformers can manifest your requested objects or subject immediately at run time. A combination of many sets used in the training data is good, but it requires strong ai models to drive it until science can optimize the systems further.
It has the potential sure. But just like any type of emerging tech like this you always just see the best results of the absolute best material. In most real life cases the results are not good enough to be used or a real headache to get there. It's intersting technology for sure, but I am really, really hesitant to call it anything but potential for the future. It may be future tech, but the future is not now.
Glad you are addressing the issue of ethical data gathering for AI training, it's something A LOT of people just gloss over when it come to talking about AI technologies.
In 3:15, you show the use of depth sensors with real time capture, and label that as photogrammetry. This is not photogrammetry because it does not calculate depth based on parallax pairs.
I would take a guess the reason might be that easily accessible lidar tools (for instance, the lidar option in Polycam) do not yield particularly useful results. The resulting mesh is lacking in detail and overall "blobby" looking. Compared to photogrammetry where one can use anything from a phone to a dSLR to capture source images and the result is generally as good as your gear, and patience, can handle.
Lidar is used, but they are very expensive. Example lidar on a iPhone is not a great option to make something very detailed. It will be close, but nothing that could be used close up in a scene. The good lidar guns you could have it close up, they are made for that (I think they start around $4,000). Where taking photos, basically everyone has access to a decent camera. You can take a bunch of pictures to get the details you want. So that's why it's popular. It is very cheap and basically gets the same results as the expensive lidar ones. It basically just comes down to cost. If a really good lidar gun was like $50, lidar would be more popular and used way more than photos.
Price mostly. You can hack something together with a Roomba LiDAR and some electronics skills, but the high-end ones used in engineering are much more accurate. As someone else pointed out, reflections can be an issue with LiDAR, but they are also a problem with photogrammetry.
"Scary and exciting". Yes, that is what it is. It's not good or useful for anything other than the creation of fiction. It's scary that anyone might think this is useful for recording reality, or attempt to use it in such a way.
It's several processes chained together. But essentially it figures out where you took each photo from, then takes pairs of photos to generate a stereo image just like your eyes do. From there it can calculate the distance of each pixel from each photo. Next it creates a point cloud from that depth information. Finally it turns that into a mesh amd textures it from the photos. That is an oversimplification, but that's the general workflow.
Other terms: Sorcery, voodoo, magic, and witchery. What it does is compare multiple images of different angles of objects and extrapolates geometric data from the results based on tracked reference points. It basically already uses a less advanced kind of AI in order to work. Nerf just adds another layer of AI to it that allows it to 'fill in the blanks' based on a trained data set of similar objects and stores that information in shorthand.
Imagine if researchers put any time into AI models that can take garbage photogrammetry point-clouds, which are basically useless as 3D object, into useable, reasonable, geometry. NeRF doesn't look to be helpful in this respect, 'radiance fields' are even more esoteric, fragile, inflexible, types of data structures.
No, NERF will not be the end of photogrammetry - Gaussian Splats will do it. Oh, and they capture motion too, if you shoot the subject correctly... you can even bring them into Unreal Engine.
All depends. It seems all a bit better visual trickery on worse models. Bear in mind the fuss about ai looking at images made by others. Stealing real world objects to convert to a digital form is the same.
eh, photogrammetry is here to stay... with photogrammetry you can do precise scientific measurements... you can't do measurements at all with NeRF... and that's especially the strength of photogrammetry, which is why it will never go away... but, NeRF is fun, so... it's also here to stay, but in completely different application fields...
Is any work being done in this general field that would allow breaking core photogrammetry rules? I'm referring to changing shadows, the object moving amid the light sources instead of being fixed and the camera moving around it. Example: a museum has 20 pics of an artifact but they weren't taken with photogrammetry in mind, all the info is there but it can't be combined in a conventional way. I recently became aware of one-2-3-45 and the one pic to 3d object concept. The dream program is one step beyond that, where instead of AI filling in the gaps with guesses it references angles from other photos. Alt approaches? Pipe dream?
@bricaaron3978 If you walked around an object and took 50 pics, photogrammetry will work. If you move the object around or change light direction, it's unusable. I'm no tech on this but it works on how light reacts with the object, how deep the shadow go, where it fades showing depth. When everything's constant it can match & overlap the photos using shadows and reference points. If it got bumped a little bit it throws everything off and the photogrammetry will fail completely or give very distorted results.
@bricaaron3978 Unfortunately not. NERF was new to me and I thought it could be what I was looking for, but it relies on the same photogrammetry base rules.
this was so annoying to watch. im new to the topic of nerv and was looking for some insights on how the technology works, but all you did was give a surface level overview without any explanations, and whenever you attempted to give an explanation it was just very general and not insightful at all.
The baity title makes me want to say that yes, photogrametry will be forever useless and should be criminalized, for long shall live the NERF, the one final solution for every field of science. 3d? Nerf. Animations? Nerf. Cooking? Nerf. Depression? Nerf
Pretty sure Gaussian Splats are the end of NeRFs. I think you're a year or so behind the times! Maybe there will be a transformer-like advancement in NeRFs but for how to-the-point and efficient that Gaussian Splatting is, I am not particularly inclined to believe that it's possible.
No cus u cant change materials and manipulate surface shaders, cannot make something burn etc. But its ok for visualization, a good replacement for traditional point cloud
