Denoising Data with FFT [Python] 

Dude this guy is crazy, I still cant believe these videos are free.Thank You for making it free means a lot.❤️

the way this video is setup, overlaying him, the code and a whiteboard, is really slick

I've spent many hours trying to apply FFT to my data and I've finally done it with your amazing explanation. For sure, the best Fourier Transform video.

Pure gold. Fourier analysis still feels a bit like voodoo to me as I'm just learning the basics, and your videos have been very helpful. The python examples are really handy. Thanks for taking the time to do these things in both MatLab and Python.

People like you make the world a better place. Free education helps everybody in the end. Thank you.

This has been very useful for me. I am a Mechanical Engineer and I am working in dynamic studies of steel structures. This method is very practical to apply to the acquisition of accelerometer data in dynamic tests. Thank you so much Steve!!!

Steve Brunton, I may never meet you in person but you helped me a lot with these videos. I wish you a good health and a prosper life.

Hi Steve, I watched many of your videos in Control the comments there are disable so I took this opportunity just to say thank you

Brilliant. Just brilliant. The quality of this lecture is off the charts.

Professor Brunton, I love all your videos.

this is mind-boggling-ly useful! (written after hours of scrolling in vain attempting to understand both the transform itself and the code that implements it .) Thank you!

This is amazing! I am finally getting rid of the feeling that the topic is complicated thanks for these videos! I mean it is, but I also am capable of getting it with your explanation.

super-duper, as always. Python examples are what we need!

This is so excellent. Absolutely the best, and most comprehensive video regarding real world FFT on youtube. Probably going to buy your book now. Thanks a lot!

Thanks very much! This really helps students who are struggling with denoising.

Very nice video professor Brunton. I think the idea behind is FOurier transform of white noise is a constant depending its amplitude. So for large signal-to-noise ratio , you will see a plateu + the real signal, then picking those peaks will work.

Very good explanation. GG for the backwards writing, REALLY nice.

one of the best channels I could've ever stumbled upon...immediately subscribed😅

One of the best explanation I have ever seen ❤️❤️

Your videos are crystal clear! I cant thank enough for sharing this high quality content. Loved the approach you took of writing parallel to the code!

Wow, I am super impressed, why didn't I find this channel early. Thanks so much for sharing this valuable information.

I told my self lets see what this guy is talking about... Realized I discovered a wizard. I will have to see all your videos. I always was afraid from fft. Thank you for simplifying it. Thanks again.

Take note: f is the noisy data 1. f_hat = FFT(f) - f_hat is a vector of complex Fourier coefficients (increasing frequency: low to high) with its magnitude. 2. PSD = |f_hat| - vector of real magnitude (power) for each frequency. 3. Filter: on PSD, keep the power > Threshold, and set 0 to the rest (because they are the noise frequency) and we have new f_hat. 4. f_denoised = iFFT(new f_hat)

Hello Steve, I just wanted to say thank you. It's been really helpful.

The magnitudes of the signals with 50 and 120 Hz depends on the random numbers that you generate at the beginning of the code. I had this problem that my magnitudes where different from those presented in this video.

Amazing Prof. Brunton. These videos are extremely helpful for DPS researchers

thank you for the education kind sir! math + code example is a great idea

Thanks for such great videos! Been following you for years and I’ve learned so much and am truly inspired by your work. Just ordered the book and am excited to apply!

I had Steven Brunton as a professor at University of Washington in Applied Mathematics, and he was excellent. So glad I just stumbled upon his channel. It looks really good. Will be tuning in! I am hoping you will do a video on linear programming and optimization; I have a problem at work I want to apply this to. :)

Thanks a lot again Prof! Just wanted to mention that, when calculating the power spectrum (PSD), the data type of the production result of complex number with its conjugate is actually: complex with 0j (Python3)

You forgot to multiply your np.fft.fft(...) output by dt. np.fft module assumes the sampling spacing is 1. So we have to fix for this if we have a different sampling spacing. Also, you don't have to give n as the second argument of your np.fft.fft because n is also the length of your signal, so it's effectless.

Effective! Your amplitude at per unit time teaching is above threshold.

Very helpful content. I have used MATLAB but not python. I want to use FFT and some basic signal processing techniques to learn how to use python, so these videos are very helpful

just in love with the way u teach concepts

For those wondering how to filter non stationary series, you can use overlapping windows to perform the same process per window then blend the signals in the overlapping regions

Beautifully explained ! Wonderful to learn from Steve !!

Many many thanks professors this was very clear.

Steve, you are amazing. Thank you a 1000 times.

Thanks much Prof Brunton! This is gold.

You are awesome, dude. Thanks a lot. I wish I had a professor like you at the college.

Your lecture gives insight to the content what is in there books.

Great application of the convolution theorem. Well explained.

These video series are worth more than the courses you spend thousands of dollars in a university

Thanks for showing us how useful is learning Math!! Amazing!!

Every body's a gangsta until a man with glasses enters the room and explains fft

Thank you a lot, Steve! Your lectures are absolutely amazing and extremely helpful!!

This video is adorably stylish!

Thank you so much sir ,i was really confused before this video

damn the video quality is amazing here. got a sub

thanks for great explanation! You guys are doing it on a different level :)

Thank you very much Prof. Brunton for the interesting video!

Thank you for the clean explanation

Yes, THIS IS WHAT I WAS LOOKING FOR THANK YOU RU-vid ALGORITHM

Thanks for contributing! Exactly what I was looking for.

thank you very much, you're a really good teacher!!

First I liked, then I watched. Amazing!

Thank you very much for this amazing video! I was struggling to understand this, but you made it look easy.

This work really helps, Sir! Thank you for your videos

Thanks for your videos Steve.

epic lighting. subscribed.

i liked and subscribed before even starting the video.. i knew it was going to be great!

Thank you teacher for the awesome explanation!

Love your presentation. Interesting though that you got your code to work without errors. Casting complex values to floats ist verboten! 😏 Here's how it should be for the graphs to display without error: plt.plot(freq[L], np.absolute(PSD[L]), color='c', lw=2, label='Noisy') However, this does not solve the problem at it's root, because the Boolean return for a complex like `PSD` referenced to and integer as in: indices = PSD > 100 does only return a Boolean array filled with `False`, unless np.absolute(PSD) gets implemented. The best place to do this without undue repetition throughout the code is where the problem starts. Here we see the amendment that forces absolutes and brings all the trouble caused by attempting to handle complex values like floats to an end: PSD = np.absolute(fhat * np.conj(fhat) / n) # Power Spectrum Density (power of FFT) However, the error finds itself repeated in the last set of plots with the `Inverse FFT for filtered time signal` and is best corrected like this: ffilt = np.absolute(np.fft.ifft(fhat)) # Inverse FFT for filtered time signal 😉 All in all, a very clear and understandable `toy example`. 💚

Agreed. A wealth of knowledge.

Wow great content. Python example just boosted my interest in signals processing by many folds. Thanks a lot Mr. Steve, I really wish to see your behind the scene setup by the way :p

Thank you very much Prof. Brunton for the interesting video! It is amazing this way of making videos. Which software and devices do you use for that? Is that a glass board? or something similar?

import numpy as np import matplotlib.pyplot as plt plt.rcParams['figure.figsize'] = [16, 12] plt.rcParams.update({'font.size': 18}) # Create a simple signal with two frequencies dt = 0.001 t = np.arange(0,1,dt) f = np.sin(2*np.pi*50*t) + np.sin(2*np.pi*120*t) # Sum of 2 frequencies f_clean = f f = f + 2.5*np.random.randn(len(t)) # Add some noise plt.plot(t,f,color='c',linewidth=1.5,label='Noisy') plt.plot(t,f_clean,color='k',linewidth=2,label='Clean') plt.xlim(t[0],t[-1]) plt.legend() ## Compute the Fast Fourier Transform (FFT) n = len(t) fhat = np.fft.fft(f,n) # Compute the FFT PSD = fhat * np.conj(fhat) /n # Power spectrum (power per f freq = (1/(dt*n)) * np.arange(n) # Create x-axis of frequencie L = np.arange(1,np.floor(n/2),dtype='int') # Only plot the first half of fig,axs = plt.subplots(2,1) plt.sca(axs[0]) plt.plot(t,f,color='c',linewidth=1.5,label='Noisy') plt.plot(t,f_clean,color='k',linewidth=2,label='Clean') plt.xlim(t[0],t[-1]) plt.legend() plt.sca(axs[1]) plt.plot(freq[L],PSD[L],color='c',linewidth=2,label='Noisy') plt.xlim(freq[L[0]],freq[L[-1]]) plt.legend() plt.show() ## Use the PSD to filter out noise indices = PSD > 100 # Find all freqs with large power PSDclean = PSD * indices # Zero out all others fhat = indices * fhat # Zero out small Fourier coeffs. in Y ffilt = np.fft.ifft(fhat) # Inverse FFT for filtered time signal ## Plots fig,axs = plt.subplots(3,1) plt.sca(axs[0]) plt.plot(t,f,color='c',linewidth=1.5,label='Noisy' ) plt.plot(t,f_clean,color='k',linewidth=2,label='Clean' ) plt.xlim(t[0],t[-1]) plt.legend() plt.sca(axs[1]) plt.plot(t,ffilt,color='k' ,linewidth=2,label='Filtered' ) plt.xlim(t[0],t[-1]) plt.legend() plt.sca(axs[2]) plt.plot(freq[L],PSD[L],color='c',linewidth=2,label='Noisy' ) plt.plot(freq[L],PSDclean[L],color='k' ,linewidth=1.5,label='Filtered' ) plt.xlim(freq[L[0]],freq[L[-1]]) plt.legend() plt.show()

what a great instructor! Thanks!

Would you consider doing a lecture on FFT in the context of NMR/Xray Crystallography/electron microscopy. Especially "Fourier Filtering"

This lecture is very clean and precise. But I need to do this in matlab and I am really getting confused on how to implement the index vector and multiplying it from the noisy signal

You just made my day. THANKS :)

鏡文字書けるのすごいですね!( It's amazing to be able to write mirror writing!)

I love all your videos.

A fantastic video with great informational content...and mindblowing production!!! How do you do all the overlays???

Very nice work! Thanks!

I gotta say, watching these and re-remembering uni courses has been quite a blast. Thanks for your lectures! Also, what exactly do you use to make these videos? I assume you write on glass and then flip the video? It always turns out surprisingly clean, with the screen projection as well...

Amazing lecturer!

Thanks Steve for your knowledge.

WOW, thank you for sharing this video!

Thanks for your amazing videos and book. I'm watching them all, and I hope they'll help me with a Kaggle competition on detection of gravitational waves from time series obtained by LIGO and Virgo.

So helpful. Thank you so much!!

Thank you Dr. Brunton.

@6:20 Presentation Master! 🙂

I don't get the frequency formula to create the x-axis. 1/dt*n * index meaning that it is an array of 1000 values with space 1.

Brilliant!!!

Thank you very much for your amazing videos, but what about averaging to remove the Gaussian noise? (if the recordoing is long enough to perform several FFTs of course) It feels like a better method to remove noise since it can show information hidden below the noise floor that would otherwise be removed with the truncation method.

This video is amazing.. Just to the point !! I'm speechless on how good the video is.. ❤

Thank you for making this video. I have one question with making a threshold, the threshold that you made, is subjective i thought. Is there any non-subjective threshold method? I mean, defined threshold to 100 isn't any valid reason.. (Sorry for my english)

Love your videos, I have a question tho. In the previous video Proffesor Brunton talked about how efficient the FFT was when our n was a power of 2. If I understand it correctly, here n = 1000? Why wouldnt we want n to be 1024?

short and precise

7:25 Start of the explanation on FFT filtering

absolute king!

Thanks .....great teaching ....

If you plot the PSD vs f until 1000 Hz you will see a symmetric pattern in the PSD plot with peaks at 120 Hz and 880 Hz, 50Hz and 950Hz. Can someone explain why?

thanks professor, for this detailed video. I

wow , ure just great !!!! thank you !!! never pressed that sub button this fast.

To go further, it would be interesting to have a method to determine the threshold for PSD that we keep. In your case, it’s obvious that there are only two frequencies but what kind of methodology will you recommand if there no clear contrast between noise and real Frequency ( would a simple Normal t’est be sufficient?)?

Do anyone knows why only plot the first half of the frequencies? It turns out it's mirrored, but i don't know why

Can I also use this to get the phases and amplitudes of the waves that compose my signal?

I have tried it with an audio sample of 60000 samples and 16000Hz. The fhat was in between +/- 400 and PSD 0 to 3. Then, I made indices=PSD>0.001. I finally saved the audio resulting from inverse fft. It was noiser than before. What is supposed to have happened in the reported experiment?