Using the popular seasonal-trend decomposition (STL) for robust anomaly detection in time series! Code used in this video : github.com/rit... Data used in this video : github.com/rit...
As a data scientist, these techniques I found very helpful, because in real data you need to deal these issues, i would request him to add more videos from machine laerning from his experince , because his way is explain to so simple, I binge watched complete playlist.
I've watched tons of math/data science videos before but always find something missing..finally settled on this channel! continue the great work Ritvik!
Nice video! In this example it might be best to do a multiplicative decomposition by taking a log first, which should get rid of the heteroscedasticity you see in the seasonal and residual components.
Absolute Gold content.. Thank you for making this video. Also, can you please make a video on how LOESS work? It will help me in understanding what is going on behind the scene. Thanks, Bhuvanesh
Man… you are amazing! Please never stop teaching this way… this is pure didactic… Could you explore more videos on how to treat the outliers after detection as you showed with the mean approach from the previous video? Or the mean calc would be just enough to remove the outlier?
Great video. One clarification - When creating the lower and upper limit thresholds, wouldn't you want to use the mean and standard deviation of the residuals UP TO (but not including) each point. Otherwise you are using future information, which wouldn't allow you to determine if an event is an outlier in real time.
Hello RitvikMath. Those thing seems so easy for you ..I envy you!! I have one big question for you. Is there a way to use Time series to pick the most "suitable" outcome in a set of finite possible outcomes we have. Not predict but find.
Thank you so much for the video! But I have a question on this. What if I want to make a daily anomaly detection check program? Should I calculate the standard deviation for the entire period every time? Or should I set a window size to calculate for the specific amount of time period?
Once you've identified these anomalies would you consider doing anything with them as they could affect your forecast? Would you consider removing the data points? Or maybe replacing them with something else?
Amazing work bro! I got a quick question. If you want to account for future data, and you want to set an anomaly detection threshold, how can you do it? You can't really use lower and upper from the code as lower gives u a negative value. How can we get the actual numbers from the data? Thx!
Briliant. I am new to data science and timeseries analysis in general, and i find this video very useful. My question is, how can I quantify or assess the 'strength' of my data's seasonality? more specifically, I want to write a code that automatically detects if my data is seasonal or not. an example of a dataset i want to work on is hourly temperature fluctuations (high temp during the day, low temp during the night). How can I automate a test that checks whether this data follows a seasonal trend or not, and if so at what frequency? Thanks!
Thanks for wonderful Video on STL . I just want to ask you, I am working on real world dataset.When I try to decompose time series into trend and seasonal components, I need to specify period value, but you didn't do that. what is that?
I often get confuse when interpreting the seasonal_decompose plot. How can we certainly know if the seasonality is daily, weekly, monthly or yearly? I would appreciate a lot your answer
is small residuals (remainder) a good thing? I sometimes found stl, x11, seats retrurn different scalse of seasonality and remainders. How to evaluate a decomposition model? Thanks
@@komalsaini5668 I used my own dataset so so I skip this part, but maybe you could calculate the frequency with a groupby().count() instead of infer_freq
How do we configure to detect the Anomalies in real time (as soon as possible).. Will there be a delay in identifying if say yesterday's data had an anomaly for a daily frequency dataset
Hey, ice_cream_interest = ice_cream_interest.asfreq(pd.infer_freq(ice_cream_interest.index)) this set the column `interest` to NaN Do not know what I'm doing wrong.
@johnysmith1375 , this will work: ice_cream_interest = pd.read_csv('./data/ice_cream_interest.csv') ice_cream_interest["month"] = pd.to_datetime(ice_cream_interest["month"]) # add this line ice_cream_interest.set_index('month', inplace=True) ice_cream_interest = ice_cream_interest.asfreq(pd.infer_freq(ice_cream_interest.index))
