NVIDIA's Image Restoration AI: Almost Perfect! 

By the way, there are now algorithms that can...'uncrop' images.

STFU 😂😂😂😂 Can you link the papers or articles about it artman40? please I can't stop laughing this is so unreal but also amazing. ps: pls don't be a bait/troll!

seems like we humans are less useful

Most assholes watermark their fanart soo.. i don't feel sorry for them.

its already possible in PS but for semi-transparent watermarks.

Thank you so much for the feedback and for your support - noted, and highly appreciated! :)

They do, its called adobe sensei

There is already an open-source implementation if you are interested: github.com/yu4u/noise2noise

That's great - thanks

wow :O I can't believe how few lines of code this is. Mind Blown.

useless, not exe, non programmers won't be able to use it.....

224 to 176!

Who knew the CSI memes would come true?

Isn't than an Angry Joe meme

i used to get annoyed at those tv programs for doing that... now it turns out its actually possible

They day you can use something like this to extract a readable paragraph from 6 pixels is the day I die of shame at how I repeatedly mocked and said its physically impossible

Where do you see reflection of the photographer?

You can see black shorts and white legs inside the big orange box at 2:40 That clip is used a couple times in the video, but that's the spot where it hit me.

I think that is just pareidolia.

@@user255 It may very well be, but that kind of detail is atypical at best in eyes, and the neural net decided to put it there. I find that concerning.

True, but this is also true for any denoising algorithm that I have ever used, with "artifacts", "glitches", "smudging", "waxy look" or whatever. The alternative is to use the noisy image and let the human brain to denoise it with an "algorithm" that depends on what he drunk last night or how much coffee he had this morning, if the room is dark or there is glare on the screen (and his personal eyes, brain, and experience of course). Is this technique an improvement over what we currently use? Maybe, let's do some tests to find out.

Now its possible a student restored a 1896 video footage to 4k and 60 fps ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-3RYNThid23g.html

There are a lot of videos that remaster old videos using AI

Yeah, this channel rarely explains how this stuff actually works, it just presents it. Anyway, what I gather from the paper is that they train by adding noise to the already noisy training images and then use the original noisy image as the target. Because of statistics, the noise on the targets average out.

The channel name is Two Minute Papers. Two minutes is enough time for an abstract and brief thoughts. The paper is generally linked in the description, though.

So the algorithm ends up removing both the added noise and the original noise?

Read the paper ;) but yeah explanation in short would be nice

So for any set of images that are noisy you can over a large enough sample of images of the same scenes or shot with the same devices gather a noise floor for those scenes and devices. This you use to perturb so that you can characterize the skew from base noise to skewed noise....turns out that this skew is merely an extension of the noise added to a hypothetical "clean" image. It sort of is doing an extrapolation backward in the noise space of the images knowing a relative noise profile between the input "noisy" base images and the extrapolated "noisier" generated images and then works backward to predict what a "clean" image would be based on this extended correlation. One idea that comes to mind as to how this may be improved is to expand across a range of devices (cameras/sensors) for creating the noise to noise skew factors ...to see if the back extrapolation for inferencing of new images could be improved (to reduce noise while preserving real detail). NOTE: This method relies on and assumes that the noise to noisier extrapolation is stochastic which makes it to large degree deterministic over a temporal window for a given data domain. In signal processing parlance the noise must be "stationary". If this is not true of the real noise generation profile then this method would be less effective. There are domains where the noise is of a "shot" like nature and thus is unique for that moment...such randomized inclusion of shot noise into a noise profile would generate a non stochastic profile and thus be harder to uniformly generate and apply to derive consistently noise reduced results. As it appears it is working astonishingly well ...so any "improvement" would be icing on this already delivered cake but this is the age we are in....this same method can be done for ANY application that relies on noise processing ...so expect it to be useful for denoising a range of data sets that have nothing to do with image data... I could imagine a temporal version of this applied to a digital delivery channel in order to infer an optimal coding strategy to allow that channel to approach the Shannon limit for a given level of noise for example. We are in a wild west of potential applications for all sorts of use of DNN techniques.

Figure 4 and 5: Wow! (ok, all the others too...)

Making such comparisons is very risky. You are able to do that because you have tons of related knowledge. You know how it's supposed to look like since you've most likely seen less noisy images of it. Maybe you've even seen it in real life. And if not, you've at least seen similar stuff.

Schreddermann our eyes have probably never seen a clean image before. The retina has nerves in front of it which produces a noisy image. The brain has to figure out what is a normal image by only using the noisy images provided by the eye.

skierpage That sounds very interesting, could you post the link please?

Lord Lima Bean this is exactly where my mind went. Denoising texture maps.

Their code is available: github.com/yu4u/noise2noise Image denoising is a really popular machine learnin task because there is basically an infinite amount of data available. So you might find an already trained denoiser in a model zoo.

RTX is likely using something like this paper that was already on the channel, by NVidia research ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-YjjTPV2pXY0.html

It's not usable in production yet because a) it reduces fine texture detail (pay attention to Koala's fur) and blurs it a bit, enough for you to still want to render it normally b) there is no temporal coherence and you will experience flicker if you try to denoise an animation. Nvidia denoiser available in Optix suffers from same issues. Most likely denoisers need more high level understanding about textures they see, like that Koala's fur has a noisy patters if hairs or how fabrics look. Unfortunately, wood and everything else gets microblurred as well, if you look closer.

Thank you so much for the kind words! Happy to hear you've been enjoying the journey so far. :)

now we can even do it in real time lmao

green magic

here it is ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-tjf-1BxpR9c.html

They are, it's in the description

If you mean by pixelated low resolution then I would suggest you to look up Nvidia AI Up-Res

The paper has a discussion about real-time denoising of noisy ray-traced images: "Figure 8 shows the convergence plots for an experiment where we trained a denoiser from scratch for the duration of 1000 frames in a scene flythrough. On an NVIDIA Titan V GPU, path tracing a single 512×512 pixel image with 8 spp took 190 ms, and we rendered two images to act as input and target. A single network training iteration with a random 256×256 pixel crop took 11.25 ms and we performed eight of them per frame. Finally, we denoised both rendered images, each taking 15 ms, and averaged the result to produce the final image shown to the user. Rendering, training and inference took 500 ms/frame."

This is a problem you have with any technology. What if the MRI machine itself is not properly calibrated and the original output is wrong? What if the doctor is not at 100% of his/her capabilities? You can only add multiple safeguards to minimize these mistakes. A denoising NN is just another tool (not much different from the ones already used).

Yeah probably wouldnt work well that way then, but what if they used the AI to spot diseases based off pictures that had no business revealing that disease. Personally I would rather have a properly trained AI looking over my MRI knowing its checking every pixel has probably the whole history of medical literature at its disposable and will never forget, something a human doctor no matter how good could ever do, it also could see things and make connections we had no idea about. Need blood work? no problem, smile infront of this camera. Done. That being said, this wont make doctors obsolete, it will make the drudge work obsolete and Doctor's will be more creative problem solvers, using AI's as tools. This will put a lot of doctors out of jobs but on the plus side healthcare will be many more times affordable to everyone.

exactly. The AI will start making up random brain tumours and such... really dumb idea.

Denoise, upscale, colourize... Maybe even make it 3D and then we can view it in VR headset, this will surely be possible in the future.

I believe they're using two noisy originals rather than one... I just replied to Damien Reloaded's comment with my current understanding of the technique.

Instead of supplying noisy image + clean image and train the network to convert one into another, they do the opposite. Supply pure noise (aka grain) + noisy image and train the network to remove the "noise component" from the noisy image, leaving clean image. Basically, instead of training the network to match (best case scenario) desired result, they train it to avoid the (worse case scenario) undesired result.

@@KohuGaly Holy! Thats clever! ^_^ thanks!

I wonder if it could be applied to sound too??

@@KohuGaly beautifully explained in 2 sentences.

Ok, I think I understand after reading the paper: the key is that they need two *independently-corrupted versions* of each clean image. In such cases, they find that you can train nearly as well using the second corrupted image as a training target as you could by using the ground truth. (Note, I believe you can't simply create the second corrupted image from the first by adding noise since it won't be independent... you need two separate samples.) Their conclusions are quite significant for training in situations where you'd normally derive the clean image from an average of many random samples... they give monte-carlo raytracing, astrophotography, and MRI as examples. In such cases, it seems capturing lots of samples only to compute a clean average *is a waste of effort* - you can train about as well by capturing only two noisy samples instead for each training pair, and you can do considerably better if you devote the same effort to collecting a higher number of such noisy-pair training examples. It makes so much sense in retrospect (averages discard information!), but it's not something I had thought of before, and I'm surprised how simple it is to take advantage of! A very nice result overall, albeit not the one I initially assumed.