Time Series Anomaly Detection with LSTM Autoencoders using Keras & TensorFlow 2 in Python 

if anyone has a problem with plot statements at the end, it helped when I used: scaler.inverse_transform(test[TIME_STEPS:].close.values.reshape(1,-1)).reshape(-1), and scaler.inverse_transform(anomalies.close.values.reshape(1,-1)).reshape(-1),

This is gold. I'm doing something similar for work. Glad I discovered this channel. Subscribed! Looking forward to more content!

Great tutorial. Just wanted to point out that the problem at 24:50 is in the way that the mean absolute error is calculated: np.abs(X_train_pred, X_train) is no the same as np.abs(X_train_pred - X_train)

Doing my bachelor's thesis on this. Very helpful gaining overview over the topic, thank you!

Appreciate all your help man, it's really made a difference in how quickly I've learned a lot of these concepts! You are the best.

Dear Venelin, You are training your model using labels (y_train) which are t+1 timestamps for each training sequence (X_train), but Autoencoder is implied to train model with labels that is the same as input training sequence, it means you should use model.fit(X_train, X_train), I guess. Could you kindly explain why you use this scheme of training and name it like Autoencoder ?

Great work Venelin. Very Clean step by step explanation. Keep it up

This is exactly what i wanted to learn.. would you be able to do the same for a multistep multivariate time-series and identify the anomaly and forecast? Thanks!

Great Tutorial, it's really noticeable that you know what you're doing. Keep it up

Hi, why do we use y_train to fit the model and not X_train? it is autoencoder right? so we train the model to be able to reproduce the input so X_train

Very helpful tutorial! I have to do something similar for university and this helps a lot! :)

I am wondering a little bit what we gain from detecting historical anomalies? It is like knowing that last weeks weather probably had an anomalie. What we need is a prediction (not of the future price, that is not enough) of the future performance. If we can't have this than we would at least like to have a prediction of future anomalies.

Uhh...this is non-stationary data, you need to remove the trend or you’ll get these bogus results. LSTMs assume stationarity. Convert it to daily percentage change for stock data.

Thanks for the tutorial. Question: Why do you create target label (y), when you are doing an unsupervised analysis?

Excellent video, thanks a lot! However I would like to see an extension of this example using multiple features. I tried to extend it using more features with no luck so far.

Really really helpful to help me with my time series problem related to climate change, thank you!

This tutorial helped me a lot! Thank you!!

Great walkthrough with the code!

This is very nicely explained. Thank you sir.

That was a really well done video, thank you!

How do we take multivariate features to perform anomaly detection? I.e. X1, X2, X3 as the input and we want to predict the future Y value.

Thank you so much for this, this is what I wanted to learn

Thank you for the tutorial. There is one piece I didn't understand. The shape of y_test is 380, so I was thinking that the model would make 380 predictions, but the shape of the predictions (y_test_pred) is 380*30. Is it making the 30 predictions per date? For example it uses the prior 30 days as the input sequence and its predicting the next 30 days? I was also thinking since the shape of y_train is a single closing price per day that the model would be trained to only predict one value per date, not 30. Can you clarify?

Hey Mr. Venelin, thank you for the video. If you allow me to ask you some questions, why do we have, while train the model, pass the X and the Y? Is the model reconstructing the original sequence and trying to predict the next value based on the 30 values provided? (I am asking because I was expecting that we would bass the same sequence, something similar as we perform using a vanilla autoencoder). It seems that we input a sequence, tries to predict the next value for the given sequence while we reconstruct the initial sequence. When we calculate the error, the error is based on the reconstruction process am I right? Thank you in advance!

Fantastic tutorial!

Thanks for this! Could you elaborate on your choice for the loss function? Would you make things different if you had >1 features (multivariate time series)? Cheers

very good tutorial . Can you do another one about anomaly detection in images ? it will be very interessting

This is cool !

I have an ad every 3 minuts... Except that, excellent video. However the anomaly detected don't seem to be abnormal for me. It' more abnormal when the change is big and sudden

Very informative, thank you

Firstly thanks. My question is that when input is 30*1 means 30 then how can be output 64 while in autoencoder we compress data then decode for example 30 to 15 to 10 then decode

Excellent content on your channel. Thanks

standard scaling doesn't work well here since the S&P 500 is increasing (test samples are strictly larger) But I guess batch normalization makes up for it when only looking at 30 days Maybe it'd make sense to scale with a logarithmic function of the date?

You are Great! That helped well so thx 🙏

Very interesting, thank you !

Note: the shared colab notebook also gives exact same error too. Hence I double checked my lines with yours. Any help would be super beneficial

First of all, thank you for the video tutorial. I am curious if can do a video on how to setup a email systems once the value reaches the anomaly threshold. The idea is to set an alarm before some thing bad happens. You can also point me where I can get an idea to set up an alarm system. Thank you

Hi Great tutorial. Trying to utilize your code with some of my data. Only change was number of time steps. Getting shape incompatibility errors when I try to use y_train in the fit method. Also using Keras 2.2.4 and tensorflow 1.13.1 ... Any advice?

Thank you so much for the nice video.

Very cool 👍 was it helpful for you to predict the stock price in real life?

Consider behaviour of user logins and find the anomalies

Amazing video, Could you also do the same anomaly detection on oil production profiles?

I tried to implement on multi variable dataset but it doesn't work , the last layer gives incompatible shape error

In order to predict for next 8 hours, what needs to be changed in the code? Time_Steps should be 8?. How the model identifies it is in hours or Days? Thanks

Why did we use time distributed dense layer any specific reason?

Wouldn't it perform any better if used the return_states in encoder and used it and initial_state in the decoder? It seems that your approach passes just the last output of the LSTM, wich doesn't carry much information...

Dear Sir, Greetings!! Thank you very much for guiding us throughout the tutorial. Kindly guide the error to solve the below issues. I am getting an error here: THRESHOLD = 1.9 test_score_df = pd.DataFrame(index=test[TIME_STEPS:].index) test_score_df['loss'] = test_mae_loss test_score_df['threshold'] = THRESHOLD test_score_df['anomaly'] = test_score_df.loss > test_score_df.threshold test_score_df['close'] = test[TIME_STEPS:].close Errors are: --------------------------------------------------------------------------- ValueError Traceback (most recent call last) in () 1 THRESHOLD = 1.9 2 test_score_df = pd.DataFrame(index=test[TIME_STEPS:].index) ----> 3 test_score_df['loss'] = test_mae_loss 4 test_score_df['threshold'] = THRESHOLD 5 test_score_df['anomaly'] = test_score_df.loss > test_score_df.threshold 3 frames /usr/local/lib/python3.6/dist-packages/pandas/core/frame.py in __setitem__(self, key, value) 3042 else: 3043 # set column -> 3044 self._set_item(key, value) 3045 3046 def _setitem_slice(self, key: slice, value): /usr/local/lib/python3.6/dist-packages/pandas/core/frame.py in _set_item(self, key, value) 3118 """ 3119 self._ensure_valid_index(value) -> 3120 value = self._sanitize_column(key, value) 3121 NDFrame._set_item(self, key, value) 3122 /usr/local/lib/python3.6/dist-packages/pandas/core/frame.py in _sanitize_column(self, key, value, broadcast) 3766 3767 # turn me into an ndarray -> 3768 value = sanitize_index(value, self.index) 3769 if not isinstance(value, (np.ndarray, Index)): 3770 if isinstance(value, list) and len(value) > 0: /usr/local/lib/python3.6/dist-packages/pandas/core/internals/construction.py in sanitize_index(data, index) 746 if len(data) != len(index): 747 raise ValueError( --> 748 "Length of values " 749 f"({len(data)}) " 750 "does not match the length of index " ValueError: Length of values (7752) does not match the length of index (380).

why do you need to create a "y" dataset if for autoencoders "x" should be mapped to "x"?

At around 24:05 time, it should be np.abs(x - y) and not np.abs(x, y) right?

is there a way to do what you did on data consisting of multiple attributes rather than just one?

How to convert it as script where we give data and output is going to be anomaly ..mean end to end functions based script rather than jupyter notebooks

😍😍😍😍😍😍😍

how can we actually feed the real-time test data to this model to get real-time predictions? is there any tutorial or link for this?

in autoencoder you should do .fit(x,x), not .fit(x,y)

Please can you change the font of your system to make the text legible

I get the ValueError: Expected 2D array, got 1D array instead at scaler.inverse_transform(test[TIME_STEPS:].close).How could I fix this?

Does that work for multiple time series of varied time length also?

Hi there, my val_loss is constant from the first epoch, so it is overfitting. Any tips?

@ Venelin Valkov Thanks for the video and great explaination. I am working with multiple fearture (Multiple attributes) for Anomaly detection. Could you provide any sample code or any reference for that. Would be great. Thanks :)

Thanks Venelin for the tutorial ! 👍 Prefered way to use Tensorflow 2.x on Colab is via the %tensorflow_version magic: colab.research.google.com/notebooks/tensorflow_version.ipynb

Hi, thanks for such a detailed and compact explanation. I need some help in .fit() method. I am getting an error time_distributed_ to have 3 dimensions, but got array with shape (7752, 1). I checked github and error logs. The shape of X_train is still 7752, 30, 1. Is there any solution?

Can we do for multiple variable

If it would be profitable, would you still invest time to create and publish a video?

can i execute this by using of image datasets?what will be the code ??

Any help for logintime based anomaly

You are copying stuff from another screen and STILL, you messed it up :) 🤣

And then TradingView stomped on him. Go there, learn Pine Script, Make money. Then go on his Patreon and give him money. OK GO GO GO

Hello noob here. Why gpu?

There is a course on Coursera with identical content of this video. So someone is plagiarizing. The course on Coursera is a paid one: www.coursera.org/projects/anomaly-detection-time-series-keras

Sorry, this is fundamentally wrong...Autoencoder should try to reconstruct 30 time steps from the input 30 time step values...Here you are trying to mix forecast of 31st value with autoencoder...It should be trained with target values same as input values.

You are close