coding a really fast De-Noising algorithm 

there is also a technique called manifold exploration, if you really hate your sanity.

Cool! An interesting thing to try: - Render just the edges of an image (or do a highpass in Photoshop/Krita to extract the edges) - Run this to fill the cells for a cool looking filter

I really like where this channel is going! Edit: training a neural network to do denoise might be interesting.

I love "funny Eve Online moments" as a description for intergalactic alien space war

bro made me cry bc this video's good af 😭😭😭

Brooo this video looks and sounds mesmerizing 😵‍💫😵‍💫 loove it

The look produced by this algorithm reminded me of how shadows look on rtx games. That made me wonder if the denoising algorithm in some real time raytracing applications is somewhat similar to this one.

Damn, that is quite the denoising work you did 😳

thanks a lot for the explanation, im working on my own 3d renderer without an api like opengl or vulkan and i searched a lot for an ideea for an algorythm but every webside explains it in words normal people can understand but no exact step by step guide for how it works

Go go gadget pixel enhancer

now I know why image sensor on dark places is noisy

I'm creating a Python raytracer that simply uses PIL to generate photo realistic image at AAA pocessing speed. However... looks like I will soon need Ai to create the precalculations since I have to also work on a whole lot of other Python projects by contracts. Basically how it generates the images is by double layering images... the first buffer RGB images are 128x128 tiles PIL blurred at 50 of 9 main colors... and the 2nd buffer is RGBA which is exacly like the first buffer except resized to 32x32 to sharpen the 1st buffer image... each combination pre-rendered to be sorted in a Python dict() to be laid out by the Wave Function Collapse.

Hello, Really interesting approach ! Maybe you could correct the unwanted blurring effect by applying a sharpening kernel convolution onto the noisy patch areas after the denoising alorigthm part ? Idk if that would help to get better results ...

Ahh my ULTRAKILL neurons are firing

I wanted to fork this a month ago, but got tied up with an emergency work project. I'm not sure if you're still working on it, but I definitively want to have a look at the code and see if I can contribute.

It looks almost dreamlike. It could be used in a stylistic way, rather than in a realistic one. Also, it wouldn't have been difficult to generate some noisy images with blender, and I'm kind of curious to see how well it works "in the field"

actually now all you need is a sharpening algorithm and then youll have comparable quality to the nvidia dev channel

Are you applying teh blur on top of everything? You should only have it fill in the dark pixels in an additive manner.

until we have AI filling in the missing pixels

I am edging to this rn. I was never this close to Bussin... Looking forward to the continuation, that edging session would be wonderful

Awesome video! Maybe some feedback: the audio volume is really low :)

I've used scratch for 3 years and it's VERY slow. but yes its a lot faster at certain things than python is

I can confirm that Scratch can indeed often outperform python

Cool video, I wonder what would happen if raytracing engine, instead of rejecting a ray that ran out of bounces, took the colour it gathered but with lower intensity, maybe use that as a base for the blur filter so the black islands aren't so black. After all it carries some information.

Thanks, that's was very educational.

Edit: I should have watched until the end of the video before commenting. Silly me. You even said “before you ask”! [strikethrough]Something I wonder if might make sense, is, if the rays that time out and don’t reach a light source, instead of being black, are instead a not-a-color value? Because like, that way you can distinguish between pixels that are black because showing a completely absorbing surface, and pixels that ran out of bounces.[/strikethrough] Another idea: what if each surface could be treated as a light source, but only if running out of bounces, and where the light-source properties of the surface was based on how it was generally illuminated? (Like, maybe based on the total brightness of the rays that hit that surface first in a previous frame, divided by its surface area?)

the voice is so quiet

Couldn’t you use an anti aliasing style method?

I didn't catch it, how long does the final algorithm take per image?

Cool!

🤍

holy shit youtube fkd the volume of this video

The problem is that black pixels are not the only cause of noise in path tracers. Path tracers sample multiple rays per pixel. Some of these rays are finding their way to the light and some don't. This results in pixel colors calculated using multiple averaged ray samples to be sometimes dimmer and sometimes brighter compared to the neighboring pixels. They may look like black pixels in some circumstances, but they are actually not most of the time. Just look at the sides of the spheres where they are well lit by the big light.

Im really hype for the ultrakill bot

The raw raytracing output looks like salt-and-pepper noise. A common way to remove salt-and-pepper noise is with a median filter: read N pixels around a center pixel and output the median pixel. Wikipedia also says that when there is only "pepper noise" (that is, random black pixels), it can be removed with a contraharmonic mean filter.

0:10 a common optimization in modern path tracers is Next Event Estimation, which means that a light source is sampled at every ray intersection point before proceeding to the next bounce. The idea is that, in most cases, a point on a surface will either 1) be exposed to at least one light source, or 2) be indirectly lit by a nearby surface that is. It's in a class of techniques called Multiple Importance Sampling, and it's a deeeeep rabbit hole if you wanna fall into it at some point lmao. Not sure how pertinent/interesting this info is to you but I figured I'd toss it your way.

Wow, this was so much better than I expected it to turn out!

Next video you should train a CNN to take the image and output the denoised image. Training would be really simple, generate a bunch of scenes with noise, save the image, then shoot more and more rays to denoise. Then train a CNN on the low vs high amount of noise. You can do it all on the GPU as it’s just kernel functions and dot products!

Interesting that Nvidia's denoiser was able to completely change the imahe with the horse!

Next time you can use something called Voronoin't. Does what you do on the first iteration but only goes thru the image once

Hmm, what you're describing sounds a lot like the voronoi cell pattern over the nonzero pixels, with the Manhattan/taxicab metric. Reading up on OpenCV's document, what you've been trying to implement can be done with the `dilate` operation, together with some simple masking. Alternatively, you could have taken any blurring convolution filter, then compute `image = image + blur(image) / blur(mask) * mask`, where `mask[x, y]` is 1 when the pixel is black. In both cases you wouldn't need to then normalize the entire image, which I believe is the cause of the weird artifacts you were getting: a black pixel surrounded by white pixels anywhere in the picture would cause the normalization step to divide the brightness of the entire convoluted image by 8.

interesting approach, but your gain is too low, please put your voice to at least -3 db

looks sick, thanks dude

Great video! Cool to see someone making their own denoising algorithm instead of doing the shortcut of importing scikit :). Maybe try running IQA metrics like MSE/PSNR or SSIM to help quantify to the viewers how good your image enhancement is.

Cool. This is kind of like morphological dilation

that was extremely fascinating. you're great at visualizing the concepts that you wanna describe. however your voice sounded pretty dull compared to the sound effects. if i were you i'd consider slapping a fast compressor or an exciter on those vocals to bring up more speech intelligibility out of the highend

This gives me an idea: Modern generative image models like Stable Diffusion support "inpainting". That is, they can complete missing parts of a given image. This suggests the diffusion models could simply inpaint all the missing (black) pixels from the noisy image. This would be quite slow but the resulting quality should be very high.

As I remember noise appear because of a different reason. IRL when light bounces off of a mirror it have the same angle before and after. But every other rough object work a bit differently: light bounces with the same angle, but the angle is measured from a rough (really smally curved) surface, wich on approximation makes light bounce in random direction (randomness is relative to object's roughness). As of Ray Tracing/Path Tracing, rendering mirrors is a peace of cake as it requires only one ray shooting for one pixel. But the things get pretty hard when working with rough materials, we have to shoot many many many lights rays and every time randomise their bounce angle, then find an average color of all rays and get output. We are getting the noise exactly because IRL light comes from the source of light with pretty high resolution, but in RTX we shoot rays from the camera and can't calculate all of the light falling on object, so we have to simulate many bounces and approximate the color. (Correct me if I'm wrong)

Dynatron - Throttle Up. I wasn't expecting this song to play in... Well... Any video...

really amazing, i learned so much from this video, thank you

This looks a lot like pull push (OpenImageIO calls it push pull) a reasonably common algorithm from 1999. Used for denoising and filling holes in textures. You create a mipchain down to 1x1, then merge under until you get back to your original format (unpremulting the alpha along the way), the idea being you preserve detail where you have it and fill in the missing data with blurred neighbours.

Using a shader instead of a convolution would fix the problems with it not looking right. Also can't wait for the ULTRAKILL bot

Okay is scratch ACTUALLY FASTER THAN PYTHON or are you fucking with me

It's normal for dark areas (dark because of a lack of light, not because of material color) to be blurrier because they will have less ray intersections. This is the situation with all denoisers.

Try applying Minimum and then Maximum filter, thats faster than Median.

Can’t you just force alpha max post blur / use average of non-black neighbours?

Hand written techniques are great but this is one of the few time AI techniques are genuinely unparalleled. I highly recommend looking into even the most basic AI denoising techniques, that are absurdly effective

write it in scratch/js if thats so much faster lol. Even lua would be a giant leap for making it “reallt fast”. You can do an order of magnitue more work in the same time/less memory, especially in a compiled language. If you’re using indices for access/mutation of arrays then rust is a decent choice for this bc it looks similar to python (and can interface easily with your actual scripts).

v solid video small nit: render matplotlib w/o axis

love the video. but i feel like someone who has no experience with the 3D pipeline shouldnt be optimising it. you get wierd assumptions like noise in raytracing only comes in black

0:21 From experience this might not the case, since noise can still be seen with ray-marching without any reflections. My theory is that the noise is actually caused by the bit limit essentially giving objects a rough surface so some rays get stuck in the tiny sized crevices. I haven’t looked into blender’s source code specifically so I’d take this with a grain of salt.

ah yes, there are images that are made so that they "poison" AI image generators, now with this, we can noise, and then de-noise images to un-poison our AI image gens! Great coming 8AAFFF!!!

Nvidia's denoisers use deep learning trained on pairs of noisy images and original images, you ain't gonna outperform those. Your account of raytracing noise seems somewhat erroneous or incomplete (to my limited understanding). The noise usually comes from the finite random sampling used to scatter rays. It's not about failing to hit a light source. In fact if you fail to hit a light source, that's information you want to use rather then try to repair.

30 seconds into the video and i liked and subscribed at the same time, great video 🙌....

Very interesting!

8AAFFF: I fear no man, but that thing... *whatever demon is editing their videos* it scares me.

i mean probably cant keep up with nvidia's denoisers in blender, but this looks better than a lot of other stuff ive seen

אחלה סרטון אוהבים אותך וורקין

Nvidia might exaggerating there image they known for doing that.

My eyes see noise in dark areas IRL.

Is it of my phone or does the video have no sound??

now in a video

Nice sfx!

lol

"raytracing lore" lmao

1:42 tbh idk either editor

How not to make a denoiser :D

Amazing 'neighbours having a party at 11pm' vibes to the music 😅

I wish you would have gone into convolution at 3:27! Maybe make a separate video on those?

How do we deal with fireflies though?

why are you even using python?

give your editor a raise

i like the results, the slight blur makes it look dreamy

Very nice video. I might try this too

so cool

this looks so cool

i read the title as "coding is a really fast de-noising algorithm" and it kinda makes sense

I imagine it's already been implemented, as it would just make sense to my brain, but could you not take pixels that bounce off into the void and try to perform some check to see what percentage of light that hits that spot would bounce hit the last surface it hit, and bounce in the direction it came from when it hit that spot? Or is that an operation that's just too expensive to compute within a reasonable amount of time? Perhaps if you could bake some information though, it might be easier to perform, and could help with realtime raytracers at least by just being ok with a bit of extra error, at the benefit of possibly significant quality improvements at similar sample counts Also, most raytracing denoisers have access to far more data than just the color for example, they usually have access to a normal map of the entire camera view in tangent space (relative to the camera's perspective), so you know where the edges of objects should roughly be, which can help make object differentiation much cleaner there's also usually some depth information which helps with this as well

This is amazing, your 3rd version works so well, I'd never guess such a simple algorithm (conceptually) would be so good. Of course, nowadays image generation with neural networks is all the rage in tasks like this.

Question : What if we cast shadows and invert the colors? Basically light everywhere and inverse?

you invented CSI's "enhance" function