NVIDIA’s New AI: Wow, Instant Neural Graphics! 🤖 

I hope so, imagine being able to take a few images of a place and instantly be able to walk around virtually in this space. You could couple it to image search services and just type a location to find third party images, and be there.

Google Earth can be very enhenced with this method.

@@ET_AYY_LMAO imagine if vr headsets used this to finally make ar truly be able to interact with the environment. It's so fast that just a quick walkthrough would make a 3d map. Plus you could have it run in the background if there's any new positional info that needs to be added as you move

I really hope so!! I hate how long it always takes to process my inputs

In fact with a few cameras something like this could make a 3d video phone call finally work

Introducing ai vision to settings in fps games 😂

Imagine how much this will streamline workflows for 3D graphics designers! You’ll have updates to adjustments in SECONDS

It also begs the question and possible reality or realities that if an AI can generate graphics on demand in this contextual sense how do you know what you see is actually what is represented before your eyes compared to something that's shown to another entity? Not taking any position just wondering what if?

@@lorimillim9131 Yeah, that has pretty frightening implications for media/the legal world.

@@lorimillim9131 I assume like we saw with deep fakes that a counter-ai trained specifically on the pitfalls of the technology in question gets rolled out at about the same time to counteract misinfo

You are too kind. Thank you so much! 🙏

Yeah i have been thinking the same lately, that i would love to dive deeper on some topics. @Two Minute Papers i think this is something you should seriously consider as a second channel - big fan of your work :)

just saying, I'd subscribe to that.

@@TwoMinutePapers Karoly, we expected you to tell if you'd do it!

@@guillermojperea6355 a second channel with in-depth analysis of papers, which is a whole new huge project with hours of work for each episode ... yeah let's decide in a few seconds and announce it in a comment's reply 😂 Btw check the chanel's playlist, he has courses !

That was what I've just elected to name "classic photogrammetry". Back then, there was _no other,_ and AI was still barely something more than academic curiosity. Unless you count stuff like OCR as AI. I'm talking about the distant past of four years ago.

Neh, its amazing for rule 34 artist, imagine the possibilities. ( ͡° ͜ʖ ͡°)

This is an incredibly important point. On that note. Let's also not forget that this means you can hyper-accelerate drug design from first principles based on receptor shape. There's already algorithms that can design molecules with a certain shape and algorithms that can search for a synthesis pathway. So you could theoretically feed the machine a series of images of a receptor and get a recipe for a drug targeting it out the other end.

It’s not just in our dreams, our brain creates a 3D environment while we are conscious too. Our eyes are 2D sensors. It’s our brains that combine that information into a 3D experience.

100% FMRIs can already get enough data to discern if someone is thinking about a building/person/ animal etc. Until now the resolution was bad. If AI fills in the extra data reliably , we are not far from mind reading/projection :P

" which is VERY similar to what our brain does" - Not really. Our brains probably don't need to "reconstruct" a 3d environment. It's already perceived to be a 3d environment, no need for "reconstruction"

This is the paper singularity

There are already neural networks that can learn by reading papers. And other neural networks that can compose videos. Not much more is needed.

This is funny but could also end up being true WHY NOT

Very Noice 👍

Two Minute Papers: a better paper in every 2 minutes 😲

As I understood it, it takes a couple of seconds to create the "render" of the scene but when it's done it is in fact running at 60fps. So I guess you can just have a loading screen or something before you get to see anything

Imagine ditching the equipment too and having the AI embodied?

could even be next month lol

This NOT creating a 3d environment, or even a single 3d object. This is creating a smooth track of 2d images. It's a very cool technique, but if you're imagining a game where you can do anything more than ride a roller coaster or something like that (and not change the direction of your camera) then this won't be applicable to that game.

Yes, with one remark: it will be done with ONE camera.

But there is better laser scanner now to scan whole building. I have seen an episode about that in net geo

@@Adhil_parammel lasers are cool. Getting a near-perfect render with a phone camera & 2 minutes is objectively better. Especially if the site is hard to get to or not very secure. even if the laser thing is higher quality the lower cost & higher accessibility of this technique would still be mighty usefull.

Careful. Remember that what isn't in the source photos has to be made up. Great for many applications, but not so great for archival and historical research. Is that the actual gargoyle , or invented from training data from all periods of history?

@@virutech32 Also one could use small lightweight drone to take photos of places where using laser scanning is impossible.

@@stub42 It's a valid argument, but the same is true for a laser scanning, and it requires human clean-up. After photo scanning with this net there also would be validation process.

And then turn it into a photo realistic image. What a time to be alive!

What exactly would be interesting about that?

@@Ginsu131 A lot of time can go into making the details of a map in table top rpgs it would be awesome to have an AI be able to fill in that detail as it would speed up production and take a lot of mental energy off the GMs.

nothing because new cheap GPUs won't exist anymore, they're all going to keep costing $700+

@@SnrubSource 3090's will be cheap

@@ChuckSploder trust me it'll stay the same and newer gpu's only get more expensive

Mining bitcoins I suppose.

My gt 1030 can run tetris! In 10 FPS... I meant 10 Seconds Per Frame

the are so many applications to this, in VFX too

If that‘s the case, we‘ll soon be able to walk around in movie sets and possibly assume the role of a character, just like the book Ready Player One predicted.

@@cbuchner1 bruh Imagine being able to see boobs from other directions like that 🤯

@@Danuxsy and it it's gonna be the most rentable application of it

I'd argue that machine being able to interpolate between photos is still far from understanding the actual geometry and synthesizing the scene from ANY point of view, as VR does require.

As cool as this tech is, you'd have to be very careful if you wanted to use it like this. The AI doesn't know the truth, it's just guessing one possibility of what could be true, based on the limited amount of information it was given.

@@EVPointMaster I assume you mean the CSI idea. Yes that is a joke. With regards to the 3D printing functionality I assume you'd still need to do some work to get a useful mesh. However it looks really close to something that would be useful.

@@EVPointMaster youd like to think that enhanced footage was never used in criminal cases, but just look at the kyle rittenhouse case, the prosecutor submitted a frame that was an interpolated and upscaled image from a video. in order to try to make a very specific claim of kyle rittenhouse doing a specific thing. for one frame. it was allowed as evidence and even enhanced it was a blurry mess that showed nothing specific, but the prosecutor was allowed to claim it showed him aiming a gun at someone.

Sorry, havn't dive into the paper bcoz am totally noob in AI, any idea which graphic card they r using in this paper? Am bout to get a new rig with RTX 3070, but had put onhold due to the progress in AI for photogrammetry is too fast..

@@danielng1765 rtx 3090

@@danielng1765 the RTX30xx series cards are for private house holds, nvidia also makes graphics cards for datacenters (commonly used to train ai models or for things such as sorting long prioity-lists for the internet). The server-rack architecture is similar, but the bandwidth and parallelization is much higher, and it scales the price to "millions per unit".

@@danielng1765 for the common high-tier-gamer (or indie gamedev and ai-code-learner), the playstation5 has a gpu, that compares pretty well to the GTX 2070, This is 2020 technology, significantly slower memory access, significantly worse for motion blur than 30xx cards of 2021. GTX 2070 card is currently still a relative good value (damm all the cryptocurrency scammers/thiefs) to put in a new pc, that costs up to 1100usd new. A GTX3070 is significantly better (>2,5x of a 2070), and commonly fround in new PCS that cost over 1700 usd. the "ti" suffix makes a significant difference and is not to be overlooked (commonly means: 24% faster memory, +20% power consumption +50% more cuda-cores.) This in general seems to appeal more for higher resolution, displays (up to 4k)

@@ollllj thks for the advise, I initially decided to get RTX 3070 due to available performance comparison based on agisoft metashape. RTX 3070 has the good cost/performance balance compare to others based on their standard samples. Shall check 3070ti as well if the result is available.

How did the results look in the end ? Would love to see that.

I remember Microsoft had a program that did something similar for popular tourist destinations, forgot the name

@@DonC876 It kind of ended up looking a lot like how google maps does their 3D models just a lot more manual and a lot more buggy but it did kind of work

@@StolenPw do you still have that program, and could you make a video of it?

@@URB4NR3CON Photosynth

It can take several years to ten years. No matter how great the performance is, if it is a specialized single function, there is little motivation to offer it as an easy-to-use application. If this feature can be further developed and used for all kinds of surveying on construction sites, people's productivity will be visibly improved.

@@B0A0A Why the society cannot simply into Kickstarter some developer?

@@brexitgreens Nvidia don't need Kickstarter they will loan the tech to game devs to earn royalty

The method is already available on GitHub. For commercial use, the authors say that Nvidia should be contacted. So it is a matter then of going through a sales process to generate interest between stakeholders, agreeing on pricing, and going through legal. Only then would a commercial agreement begin. And if the goal was a graphical interface you'd have to give ample time to develop this, bug test it, and run perception tests to ensure that it is user-friendly. It altogether takes a while and can explain the lag between the newest results and easy-to-use graphical programs like Topaz Labs. I work on a sales team for a commercial scientific image processing software. I suggest new ideas to our R&D regularly. Maybe 2%-5% of ideas are accepted and end up in the product. For these features (which are mostly low-effort high-reward due to cost considerations), it often takes 2-3 years. And that's when we already have an application and team to build the new feature in to.

@@mustardofdoom Thanks for the detailed explanation, it really puts things into perspective

great idea! It would also be easy to generate data. What is the state of the art on this?

Nvidia maxine exist as neural video compression

I believe this is what they are aiming for. Extreme compresstion for photos and video while preserving the important details. Say a video of a footballer kicking a ball inside a stadium. The important details such as the facial expression and the actual movement are preserved. Non-important details such as the field, the spectators, the roaring sound can be compressed. During playback, these non-important details are then procedurally generated. This is somewhat anolagous to how we stored information in our brain.

that's what OpenAi Codex and Github Copilot are doing (not at this level, but they are quite good)

I can already tell you: - Video games production will just be... wtf-level of workflow improvement ; - Google Earth, which already have bazillion pictures, will either be banned or transformed into a life-like simulation ; - Military analysis and transport will make use of the above (Google Earth) ; - Urbanism, real-estate construction and sales of real-estate ; - Back to Google Earth, the applications are just crazy: imagine a traffic application where you can get a life-like visualization of the traffic, making it seem so much more reliable to the viewer than a red line on the road? - I'm going nuts thinking about all the crazy ways this will affect us in our day lives. ... aaaaah, just the VR-Google Earth life-like simulation is utopian/dystopian enough.

@@themore-you-know Im working with google earth studio.. combine that with blender -and Video editing .. you can create a realistic 3D scene. The problem is close up... the textures/models are so broken.ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-UrvKsuDSaNE.html

The AI created the detail from that one single grainy, almost black and white image. It is so good at creating the next layer that it can procedurally generate ever increasing depth of image. Pretty neat stuff.

From my understanding, and i dont think in this case he explained it vary clearly... An AI seems to take a large image and train another AI with it in order to reduce the size of the image since the new AI will then be able to fill in the details based on its training. So each image would contain its own AI or be paired with one specific to it. So long as the end result is smaller its a win. Though the explanation like i said is lacking and it would have been great to see the original file size VS the end results file size.

@@jessiejanson1528 Well thats dissapointing in a way. But it could be used for compressing data immensly.

Yes retopology and UV mapping is the most annoying part of 3D design. We are close though 🔮

same here. Imagine how good GTA Earth could be!

I find this with most of his videos

It interpolates between given photos of a scene.

Basically they developed a multiresolution input encoding that simplifies and allows to highly parallelize the task, taking full advantage of the GPU. They apply it to different techniques (NeRF, Neural Gigapixel Image, Neural SDF, and Neural Volume ). From the paper website: *_Instant Neural Graphics Primitives with a Multiresolution Hash Encoding_* _We demonstrate near-instant training of neural graphics primitives on a single GPU for multiple tasks. In gigapixel image we represent an image by a neural network. SDF learns a signed distance function in 3D space whose zero level-set represents a 2D surface. NeRF [Mildenhall et al. 2020] uses 2D images and their camera poses to reconstruct a volumetric radiance-and-density field that is visualized using ray marching. Lastly, neural volume learns a denoised radiance and density field directly from a volumetric path tracer. In all tasks, our encoding and its efficient implementation provide clear benefits: instant training, high quality, and simplicity. Our encoding is task-agnostic: we use the same implementation and hyperparameters across all tasks and only vary the hash table size which trades off quality and performance._ _Abstract:_ _Neural graphics primitives, parameterized by fully connected neural networks, can be costly to train and evaluate. We reduce this cost with a versatile new input encoding that permits the use of a smaller network without sacrificing quality, thus significantly reducing the number of floating point and memory access operations. A small neural network is augmented by a multiresolution hash table of trainable feature vectors whose values are optimized through stochastic gradient descent. The multiresolution structure allows the network to disambiguate hash collisions, making for a simple architecture that is trivial to parallelize on modern GPUs. We leverage this parallelism by implementing the whole system using fully-fused CUDA kernels with a focus on minimizing wasted bandwidth and compute operations. We achieve a combined speedup of several orders of magnitude, enabling training of high-quality neural graphics primitives in a matter of seconds, and rendering in tens of milliseconds at a resolution of 1920x1080._

@@Pixelarter Thanks! Are the inputs to all of these tasks the same (2D images?)? Cause the tasks sound quite different, so it's cool that their encoding works for all of them

@@tobiascornille No. Some are 3D, some are 2D. From what I glance, the hash they developed just encode the position of the input at different resolutions, concatenates them and apply some transform, and feed as input along regular information.

Deep learning abstraction... first layers do the stuff, just plug your application on the top.

The same way your brain does, evolution.

Yeah they give descriptions of a scene or draw primitive zones like this blob is water, this is land, this is trees, this zone is sky etc and a.i paints a picture. That is magic indeed. One upping that is telling a.i to do a task or write a program and it does it in a matter of minutes or hours.

I got that from this explanation. The layer that it produces is good enough to use as input. So, yeah at some level it is creating images based on pixels that it generated.

i imagine its trained based on images that exist, and learns what detail should exist in such an image, and then fills it in. so you can zoom in on a city because it knows what buildings, cars, roads, people look like. but it wont understand that zooming on on a persons face would reveal skin cells, unless it was given high resolution photos of skin, that transition to the microscopic level to reveal cells. also, if its given context that its a painting, at some point you would want it to assume its zooming on on oil, not assume that the artist was able to paint in individual cells in the painting because its a painting of a face.

The code is on GitHub! Dunno how hard it would be to actually get it running, and the GitHub page says you need a Nvida graphics card, but it is available to the public if you know what you’re doing!

@@thegeekclub8810 I got it running just now. I'm on a GTX 1080 and it runs quite slow, but I can train the fox one and look around in low res. Works pretty well. Setting it up is quite impossible if you don't know what you're doing. But you can always try. I tried training one of my own datasets but it throws an error. Still working on that. edit: The GTX 1080 is probably much slower because it is not RTX . the Geforce RTX line of cards have tensor cores which are more optimized for this kind of job and my 1080 has none.

@@daanhoek1818 can you please share the link?

@@thegeekclub8810 Thanks for the information!

There is a link in the description to it.

It takes 5 seconds to train nn, and evaluation (rendering) is real-time. Acording to paper this animation (5:02) can run 133fps in 1080 on rtx3090

that would be quite heavy to replace your model on every frame of animation.

@@magen6233 Idk I did not red whole code yet. From what I read, I understand that synchronization is done with m_training_stream (for training) and m_inference_stream (for rendering) (these are cuda's streams and are used for runing kernels asynchronously). Whole magic happens in Testbed::NerfTracer::trace() function and train_nerf. I think that they are coping sth but for sure not every frame (update_nerf_transforms function, copy sth every training step).

Dude you can create 3d models from photo alone

I'd love to use this on Google Earth. Imagine you can slowly walk down the street instead of zipping down a few meters at a time

Photogrammetry, self driving vehicles, and, an interesting and maybe not so feasible one, pre-rendering a complex scene and simplifying it to be displayed on lower end hardware using this technique. From what I read, it produces an SDF, which is super awesome because they're cheap to render and can offer a lot of mathematical meaning, e.g. with self driving vehicles

@@Hexcede yes it creates 3d models, but how would you integrate it into existing software would be a bigger challenge. 3d models to be used in applications need to be highly optimised as well (topology , different maps and stuff) I get that this has amazing and varied applications but I fail to see how it can be seamlessly integrated into existing software say blender or reality capture, unity, etc

@@SYBIOTE well obviously it would need other processes added to it. for say if you wanted to use this to create cgi characters. youd probably start with a person in minimal clothes, then add the clothes on top after. if you want it for map making, then youd remove everything you want to be interactive and model those objects separately, so the level, the walls, floor, etc is created using this technique, and you dont have to cut things out of the model. combine this iwth teechniques for removing the specific lighting condition and being able to use an in engine lighting, which weve already seen in other papers. these things are all literally just research papers. how they get applied to software that is end user friendly, even for professionals using complex software, is years down he line, and will likely require companies like weta to create them.

Yes, I also immediately thought of that when I saw the video. Like GTA in the real world.

I'm wondering about the storage difference. Images are huge (there are a LOT of them), but this is probably way bigger. Or is it...?

@@mikiqex maybe the neural network could generate the intermediates in real time someday?

We are entering Dr Strange realms here 😂..

Thats currently easy.