End to End Multivariate Time Series Modeling using LSTM 

Yep, best tutorial I found so far on inputs to LSTM.

@@pravdomirdobrev4850 code link plz

Thank you. Will try to add ARCH and GARCH videos as well

Hi.. One reason can be maybe time series is random and it is more dependent on most recent values than a longer window. Do you see any seasonality or trend in data?

@@AIEngineeringLife I am using your dataset and setting length to 1 produces better results than 720.

@@suemetzger4579 win=1 is just comparing the data the period before, and reflects some auto regression present.

Vikas.. Change keras.preprocessing.sequence.TimeseriesGenerator to tensorflow..preprocessing.sequence.TimeseriesGenerator In latest TF api (2.3 and above) they changed or use TF 2.2 to run the code as it is

Looks like my old comment was deleted. Actions can be depending on business process, in retail it can be product inventory or manpower allocation, In energy sector it can be amount of energy to sell in open market, In loss forecasting it can be to be better prepared to meet regulatory capital allocation requirement. In this video I show how to predict for next 4 periods but it can be increased to 10 as well

Vishal.. I have covered it as literature in this video and few others as well. There are 2 ways one can do multivariate time series. One is using VAR model which creates separate equation and gives out model coefficients for each variable. In this case you might be able to predict all variables in future forecast. Challenge is each variable will have its error and overall can distort your target prediction as well. Second way is when forecasting you need to forecast separately the future variables of multi variate variables. Some variables can be easy to get data like temperature we can get from external data and others and for few we create separate model and then feed to LSTM or other models like facebook prophet. This is the way out in many cases and reason multivariate is hard

@@AIEngineeringLife Thanks for the information sir...now I am clear about the solution we have for multivariate. I was wondering we would have something different to tackle this problem...Thanks alot 🙏🙏.

WHAT A MODEL, WHICH DOES NOT FORECASTS .... SHEER WASTE OF TIME

Yes that is one way or you can also predict multiple observation in loop. Check in my data prep video, I have shown how to do multistep prediction

Paolo.. I will try to create a notebook in a week and share or video. I did not find much online examples as well which is surprising. But quickly you can add TimeDistributed layer in end and add units to your target size

Hi Paolo, were you able to implement milti-output prediction? I have the same problem rn I can´t find how to solve it

@@isameneses12 did you able to do the multi-output prediction? Thanks in advance.

@@data-scientist-sweden yes, I was. You have to search for multi-input/multi-output step prediction. I don't have the code with me rn but the most important thing is that you have to use as many neurons as steps to predict ahead in the last dense layer

@@isameneses12 Thanks it is a great input. If i understand it correctly if you have 3 target to predict for 4 time steps each, you need to have (4*3=12) neurons in the output dense layer. However, @AIEngineering did not recommend multi target output. Did you use TimeDistributed function at the end?

Thank you.. You can go for either CPUs as for CPU workload both works good. Intel as some packages that provides DL acceleration on CPU. I am not sure if that will wok on ryzen. Typically more cores is better as packages like xgboost and some of scikit ML can be run on multi cores. For GPU based workload while GeForce is enough for some of it for others you can use colab or others

Not sure unless I get to see the code. You can check my code here and compare it - github.com/srivatsan88/End-to-End-Time-Series/blob/master/Time_Series_Functions_for_Sequencing.ipynb

Importing TimeseriesGenerator with the following command solved the problem for me: from tensorflow.python.keras.preprocessing.sequence import TimeseriesGenerator

Try this from tensorflow.keras.preprocessing.sequence import TimeseriesGenerator

LSTM does not care about time element unless the order is fine. So missing data is fine but when you predict in future it might not have knowledge of missing information if missing is consistent as per business process

I have interpolated missing data and when I run the runn the Lstm model I have loss of NaN and mean absolute error also Nan. Is there any function to call the interpolated dataset?

You can forecast up to the prediction length using window length. When you train model you can define both length. beyond that as well you can iterate using current prediction and forecast next. Yes we need to forecast other features separately. But in case if you need to predict all then you can check my VAR model for time series video

@@AIEngineeringLife thank you for the explanation,

I have a seperate video explaining timeseriesgeneeator and how it can be done. This must give you the answer - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-i4vGKgbtf1U.html

Assalamualikum how to get the GitHub code?

Nan values from train generator you have to drop it as based on context_length you might have few. Coming to LSTM size I tried out various architectures. You can check my Keras Tuner video where I do hyper parameter search to find right units that fit the data better. Same can be done for LSTM as well

Ahmed.. You can set the variable you want to predict as your target. In this case I have set variables at 0 index but if visibility has to be predicted then you can use index 3 instead that is last variable. Note, you might have to do analysis and see on model parameters related to visibility in model

@@AIEngineeringLife Thanks for your respone. so i will only have to change target = data_scaled[:,0] --- > target = data_scaled[:,3] and pd.DataFrame(x_test[:,1:][win_length:] ---> pd.DataFrame(x_test[:,3:][win_length:] and df_final["predicted_value"]= rev_trans[:,0] --- > df_final["predicted_value"]= rev_trans[:,3]

Rahul.. For Multiple time series you have to create multiple models for each of it. You can also check my DeepAR video which allows you to create one global model as well but not recommended if every time series will have different pattern - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-B_a0K2Uj85Q.html In my time series playlist I have 3 to 4 videos on multiple time series using different tools, please see if it helps

You can write your own function and feed it as metrics. Said that 1.10 is end of life, atleast upgrade to 1.15 or better to get to 2.x

@@AIEngineeringLife Hi, yes. I was able to fix the code by replacing it with this: model.compile(loss='mean_squared_error', optimizer = 'adam', metrics = ['mean_absolute_error']) However, since I am trying to train my own dataset which has 13 features, I am getting the below error: ValueError: `validation_data` should be a tuple `(val_x, val_y, val_sample_weight)` or `(val_x, val_y)`. Found: I checked solutions online but I was not able to fix it. Any alternate approaches?

Madhura, It must be in git repo in coming week. You can check it here where my other time series videos are there - github.com/srivatsan88/End-to-End-Time-Series

Nope autoencoder is coming.. This is just LSTM model

Thank your for replying! I am waiting for your video of an autoencoder!

Adeel.. looks like you did not import tensorflow as tf. Tensorflow needs to be installed on this

​@@AIEngineeringLife Thank you very much for your prompt response. I've tensorflow installed, however i updated and getting different error now. May be I am missing some model in anaconda. NameError Traceback (most recent call last) in () 2 patience=2, 3 mode='min') ----> 4 model.compile(loss=tf.losses.MeanSquaredError(), 5 optimizer=tf.optimizers.Adam(), 6 metrics=[tf.metrics.MeanAbsoluteError()]) NameError: name 'model' is not defined

@@adele_90 .. I think you missed one step.. here is my notebook for reference - github.com/srivatsan88/End-to-End-Time-Series/blob/master/Multivariate_Time_Series_Modeling_using_LSTM.ipynb

@@AIEngineeringLife Thank you so much :) I was using Anaconda before and then switched to Colab. Now it's working perfectly fine. I've just started learning deep learning/ML that's why struggle in some areas. Thank you again for posting great work.

Yes.. Thats right only thing is I think it must be data_scaled[:3,] instead of ,3 . Need to try it out. Indexing are always confusing :) ,3 might pick all column but we want only last one

@@AIEngineeringLife data_scaled[:3,] has value error:Data and targets have to be of the same length........but ,3 no error!! can u explain???

This is a time series problem and not regular regression. In time series there are lot of cases where we use past lags of data, seasonality and trend to model future occurrences. Now coming to 1:1 if you see the data preparation step the model sequences past lags to predict future. To give an example forecasting holiday sales we add previous holiday lags and see trend as well as additional variables like economic indicator and all In typical regression problem we do not sequence the data and that might cause 1:1 if target is added. You can check my time series sequencing video to get more details on sequencing data for time series

@@AIEngineeringLife Thank you for your fast reply. May I ask you to provide a link to your "time series sequencing video", please? Thank you very much in advance for your help.

Here it is - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-i4vGKgbtf1U.html

@@AIEngineeringLife Thank you. Merry Christmas and Happy New Year!

My lstm input shape is (None, 720, 128) . First 720 is my window length before I get the first prediction and after that with sampling rate 1 it is a sliding window of length 1. So when I feed my test data to model, I will get my first prediction at only win_length. During training you are right that my last few records might not have target labels based on my win_length and sampling_rate. You can validate the first label by just selecting first 721 records and comparing the first label of test_generator x_test[0:721] test_generator[0] In above case first label output of test_generator must be last record of x_test above Hope I am making sense. LSTM data prep is confusing for me all the time :)

@@AIEngineeringLife Thanks for your answer. Could you also give an advice regarding the right way how to tune parallelizing training model process on CPUs? 1) in google colab there is total time for fitting model 10 min There is NVIDIA Tesla T4 GPU. 2) in my virtual machine there is total time for fitting model 2hours + There is Intel® Xeon® Gold 6230 (20 cores) * 16 I tried different approaches to tune setting ig fit_generator: history = model.model.fit_generator(train_gen, epochs=50, validation_data=test_gen, shuffle=False, callbacks=[early_stopping]) a) changing use_multiprocessing and number of workers b) set config and creating session before fitting # jobs = 12 # number of cores # config = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=jobs, # inter_op_parallelism_threads=jobs, # allow_soft_placement=True, # device_count={'CPU': jobs}) # session = tf.compat.v1.Session(config=config) # tf.compat.v1.keras.backend.set_session(session) but I cannot get acceleration in productivity in my virtual machine, fitting run just on 1 CPU from 16. It would be perfect to get some advice or link where I can get answers on this question. Have a good day and thanks in advance!

@@mikhailb8026 CPU parallelism to my knowledge will be beneficial for regular ML models not in this case where order of data is important and we are sequencing everything as single row. I doubt gradient updates will give clear picture and we cannot do data parallelism due to nature of data. In this case GPU will be ideal but higher cores can help matrix multiplication process Above threads might work for computer vision model or regular LSTM based NLP models. I have separate videos of distribution strategy for computer vision but for time series you can parallelize multiple time series cases, not cases of single multivariate time series

It is the same.. Instead of test data use model.predict_generator on train data. Once you have training prediction you can use same steps to plot as we did for test data

@@AIEngineeringLife Thanks for quick reply 🙏. I’m going to try it immediately.

@@AIEngineeringLife It worked. Thanks a lot!! :)

First way of identifying the context length of window is domain knowledge of data. Second using EDA like auto-correlation plot and seasonality. Say if your data has weekly seasonality so window length has to factor it in so your length might be 7 or 14 or some higher duration

@@AIEngineeringLife tq sir, i see..but my data are sporadic and non-consistent, sometimes in a month, 2 or 3 days are recorded..how about this case?

@@aqilahrose What is the reason for missing data?. Are these zeros in other day or is business process not capturing data for those days?

@@AIEngineeringLife I believe it is not capturing the data for those days. it goes from 8 Jan,9 Jan and suddenly 20 Jan. I play with sensors data provided by the respective websites. but the size of the data is quite big from 1986-2018..

@@aqilahrose If you have huge gap in data and also not consistent pattern it is better to aggregate data at monthly level and do time series. because daily series with only few days captured will not help much and more so if every data has different pattern

If your data is quaterly and you want to look back 2 quarters length will be 2. so it must match your data granularity. 0 index is first batch 1st is 2nd batch and so on

@@AIEngineeringLife ok, thank you. This is really helpful

@@AIEngineeringLife and what would be the win_length for quarter data?

label encode, or hot encode it

Hi, Are you asking layers or unit size here?. You can check my NLP playlist where I have covered it for classification task - ru-vid.com/group/PL3N9eeOlCrP6zMkHMxFJV4yXIsET5aWlc

​@@AIEngineeringLife hi, asking for layers... i have a classification problem all x- variables are continuous values and target is either good or bad or we can say 1 or 0. I want to apply LSTM so what losses, activation function should i chose and how?

@@monikadogra9153 Since you have 1 and 0 categories (only 2) you can use binary_crossentropy as loss but for activation you can try with either relu or tanh and see which performs better. Last layer will be sigmoid though

@@AIEngineeringLife thank you

@AIEngineering sir i have one doubt why we are including the column which we want to predict in input variable as well in general case we don't do that? we don't include target variable in X variables bur here we are including?

Peniel.. This is time series model and past lags of target is used to predict future lags. Like past sales is used to forecast future sales, reason we have target to it. Even though we use target it does not learn mapping due to the way we prepare data for it. YOu can check this video on how we prepare data for time series - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-i4vGKgbtf1U.html

if you are running CNN then basically you are running a 1D CNN. CNN will ok for univariate but for multivariate I am not sure. For predicting beyond test data you need to know future values of independent variable. You can check VAR model video of mine if you want to predict all variables

Thomas.. If I understand you want to predict multiple variable in same model. I would not advice that as when we talk multi variate each variable output has a certainty and typically we run separate models to forecast it and feed it to main model. Multi step is possible though where we can create target to forecast multiple steps on single variable. I have covered it in my data preparation video to some extent

woaaaah, thx for your response. didn't think you'd actually answer me. I'll surely take a look in the video you've told me about :'3 Thx 4 the attention

Tarun... each company is a time series by itself so in this case you need multiple models. You can check out my video on multiple timeseries

How do I do this, please help

Its so simple just replace old code with new one@@PoncyPenguin

Should it be 'predictions[0]' instead of just 'predictions' in the concatenation step?

It probably has something to do with my LSTM model output since I've changed it from what you have shown. I'll try to figure it out. Thanks!

The shape of my raw data is (185,4). For the time series generator, I have window_length = 5, batch_size=16 and num_features=4. I am using 70% of the data for training and the remaining for testing. The shape of my predictions array is (51,5,1). Why is that, I checked your predictions array shape and it is (3227,1). Why does mine have 3 dimensions? Does that have something to do with the model output? That is what is causing the error.

I figured it out, it has to do with return_sequences. When I set it to True, I am getting a 3D output array and when I change it to False, it becomes 2D. Sorry for being a bother. I hope someone is helped due to these comments of mine.

@@vedsoni3600 Sorry just seeing it. Thanks for commenting your results as it might help someone with similar issue

Based on forecasting length we can create a future dataframe and append it to existing one

@AIEngineering can you give an example of how to add a future dataframe? *And can this data system use in CNN model?*

Yes you can. If you just take previous time length from train you will get next future value

Yes that's right. We are doing single step forecasting here

@@AIEngineeringLife What if we want to forecast 12 hours ahead? Can we make some tweak to this code that will enable us to do that? Or is it covered in some other video? Thanks again for your super fast replies 🙏🙂

@@harisshaukat4780 . Now one prediction is one time step but u can iterate and forecast for longer duration Another option is to model as multi step prediction. I have covered it in my data preparation for sequencing video. We can use that data to train a model

0 show the first element of the printed request

Kiran you can just sequence it as it is done in this video or pass numpy array. Model and scaler is required during prediction that you can save

Nibin.. It must be 2.1. Created this video a year back. Unfortunately tensorflow apis have been an headache with constant changes. Concept must be similar. Let me try to make a video with latest version next month

@@AIEngineeringLife thank you sir, then if use tensorflow 2.1 instead of current version will i get the correct results

No its wrong

@@omerfaruque6228 How to forecast for next 10 days in a step in LSTM?

You mean during prediction time?. You are passing batches and want to see if data is in pattern or out of pattern with trained data like in anomaly detection cases?. Trying to understand the question here

​@@AIEngineeringLife ok, sir i have batch(say 100 batches) wise data each batch run for 0-48hr and finally we classify each as good batch or bad batch, based on three parameters called as quality parameters have some threshold values. if batch is >= those thresholds named as good otherwise bad. now my objective is to predict before the completion of the batch whether it is going to be good or bad( by checking the value of quality parameters ) ? so by giving some history value say 30 time instances want to predict end point value (not t+1,t+2,) every time end point.So that we can check at any hr (at early hr)current batch is deviating towards good or bad.

Yes so this is regular time series forecasting and it is possible provided your bad batch elements have a journey that can be well modeled in good batch.. To give an example of AIOps say u want to predict API failure and before failures we typically see out of memory or packet drop error then good batch will capture pre failures and we can predict there is a possible failure I hope I am making sense

Sougata.. it s difficult in real world data unless data is just a multiplier of previous value. Best models can do is get closer to actual and give a confidence range

@@AIEngineeringLife I tried your code but my actual and predicted values are exactly same. Can you please help me

@@sougataroy6532 you mean you ran the notebook from my git and it gives similar value?. I reran it and I do not see it that way. Is it possible for you to share your output

@@AIEngineeringLife no no dear...I tried your code on my dataset ... But my actual and predicted values are coming same

@@sougataroy6532Few reason it might happen is the data can be easily model and might be a simple exponential function. Other reason is maybe the validation or comparison is done with actual y rather ypred, just by mistake. Too hard to say for me without seeing the data and notebook

did u find any solution?

You can pass anything for LSTM instead of time stamp. Basically you have to just make sure sequence of data flowing in to LSTM is arranged in right order

did u find any solutions?

@@prathamsoni69 did you?

@@isameneses12 test_predictions = [] first_eval_batch = scaled_test[-win_length:] current_batch = first_eval_batch.reshape((1,win_length,num_features)) for i in range(win_length): current_pred = model.predict(current_batch)[0] test_predictions.append(current_pred) new_input = np.append(current_batch[:,1:,:],[[current_pred]] , axis=1) current_batch = new_input test_predictions will have scaled values so u can use inverse_transform Note: thesepredictions wont have a date but u can add dates using for loop.

Did you pick the notebook from my repo and run it?

@@AIEngineeringLife hi, I have solved the issue. Thank you.

@@tanmoybhattacharya2761 How did you solve it, please help

I have used my test data as future forecast here. We can build a future dataframe similarly and get temperature data and others from website like weather dot com and then run the data frame for test data. In this case appliances which is also a input column has to be used in regressive way

I was going to ask the same question. Is it not possible to forecast (predict a future point) without any knowing of input features in multivariate time series? Because in univariate time series we can predict the future by adding the predictions as inputs.

@@erensarnic4058 Multivariate time series is hard in that sense as dependent variables have to be forecasted or sourced first. With multivariate to my knowledge there is no other way out. Univariate deoends on itself and so it is easy but lot of real world problems has external dependency as well and univariate in these cases alone might not suffice

So the same thing goes with classic time series models such as ARIMA, SARIMAX etc. If it is multivariate time series it is hard to go beyond test data.

@@erensarnic4058 It is hard but not impossible. External data like weather, economic indicator can be sourced relatively easy from external vendors but internal data has to be simulated along with time series modeling

Here it is - github.com/srivatsan88/End-to-End-Time-Series/blob/master/Multivariate_Time_Series_Modeling_using_LSTM.ipynb

@@AIEngineeringLife Thank you soo much :) This was really useful. I have question i.e while preparing test data from other csv file we usually dont have y_test, How can we use TimeSeriesGenerator function in this case?

@@sachinbainur2636 if you see TImeseriesgenerator it does not store any parameters, so the same wat you use in train you can just copy it in test as well

@@AIEngineeringLife suppose I have 30 rows of data in and I dnt have a column ' App' which is also an input and a column to be predicted. All the values of column must be predicted, how can i pass input to model? It will show me dimension error

@@sachinbainur2636 if you have to predict all future values of variables then start with VAR model and see. Like here - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-TpQtD7ONfxQ.html LSTM multivariate prediction is difficult but I have plans to make a video of it

Both Stateful and Stateless are valid configuration depending on need of data. I do not see a general rule that all TS has to be stateful. Usage of Stateful or stateless in Time Series depends on how long in your sequence you want dependencies to be factored in. If I have very long sequence length it makes sense. If you see sequence length here I have boxed it already. Back testing was not scope of this video. I have covered Back Testing with DeepAR which is based on LSTM network in a seperate video

@@AIEngineeringLife I recommend to you to read Jason Brownlee blog and understand what is time series (LSTM must be stateful when it is a time series, review the LSTM cell)

@@WahranRai I will check it out. Meanwhile you can check below from stackoverflow as well.. I do use stateful in some cases where my sequences are very long. Say i need a long context for forecasting like based on yearly seasonality or econometric model stackoverflow.com/questions/39681046/keras-stateful-vs-stateless-lstms stackoverflow.com/questions/52027278/do-i-need-a-stateful-or-stateless-lstm In this case I feel stateful will not give me much of an advantage