Time Series Analysis using Python| ARIMA & SARIMAX Model Implementation | Stationarity Handling 

Glad it helped!

Glad you liked it

Glad it was helpful!

Glad it was helpful

Glad it helped!

thank you very much

@@learnerea. Please, can you make a video on how to use the transformed data especially one gotten using log, sqrt and shift. I have been trying to figure that out. The area that got me confused is how to transform the data back to the original format. Thank you

Thank you, I will

Glad I could help!

Glad it was helpful!!

Noted

Thank you very much for watching it. Yes, that was primarily because it was a beginner level and hence we did not want to spend a lot of time in reverting it back. Certainly we will make another one where we conclude and utilize the stationarity.

You are most welcome, and I'm glad that it was helpful.. keep watching

Glad you liked it!

Yes, soon

Thanks a lot 😊

Glad it was helpful

exactly,,, i have this question as well? Because we were taught to fit the model on transformed data? Please reply it would be very helpful.

Glad it was helpful

Using arima model i got 43 as my mean squared error

Super

Assuming you are using the same data as in video, please share your code at learnerea.edu@gmail.com so that we could have a view.. and guide you more specifically.. include the data as well if it's different from the video

For that airP['12diff'] we have to take, as it is seasonal difference

Yes, the situation you're describing is not uncommon in time series analysis, and it's often a delicate balance to strike between achieving stationarity and preserving important characteristics like autocorrelation. When you difference or transform a time series to achieve stationarity, you are essentially altering the original data to make it more amenable to modeling. However, as you've observed, too aggressive a transformation can result in the loss of autocorrelation, which is crucial for capturing temporal dependencies in the data. Here are a few considerations and potential approaches to handle this situation: Selective Transformation: Instead of applying a uniform transformation to the entire time series, consider selectively applying transformations to specific components. For example, you might difference the data only where it's necessary or apply different transformations to different seasonal components. Partial Transformation: Rather than making the entire time series stationary, consider transforming only certain parts of it. For instance, you might apply differencing or another transformation to the trend component while leaving the seasonal component untouched. Different Models for Different Components: If your time series exhibits both trend and seasonality, you might consider using models that can handle each component separately. Seasonal decomposition of time series (STL) is one such approach where the time series is decomposed into trend, seasonal, and residual components, and each can be modeled independently. Advanced Models: Explore advanced models that can handle non-stationary data more effectively. Long Short-Term Memory (LSTM) networks and other recurrent neural networks (RNNs) are known for their ability to capture temporal dependencies in data. Ensemble Approaches: Combine predictions from models trained on the original data and models trained on the transformed data. Ensemble methods can sometimes capture the strengths of different models. Grid Search and Cross-Validation: Systematically experiment with different combinations of transformations and models. Use grid search and cross-validation to evaluate the performance of various configurations and find the optimal solution. It's worth noting that the ideal approach can vary depending on the specific characteristics of your time series data. Experimentation and a deep understanding of the data's behavior are key. If possible, consider consulting with domain experts or seeking feedback from colleagues who have experience with similar time series patterns. Remember that achieving stationarity is a means to an end (better model performance), and the goal is to strike a balance that preserves the essential characteristics of the data while making it amenable to modeling.

Thanks for your detailed reply. How do you conduct a partial transformation? for example, do I difference only a section of the source data that I’m training the model on? How would I even then reverse transform predictions?

Glad it was helpful. the below one is on linear regression - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-IigoyVON0eM.html here is a problem we solved using the regression and other best fit models - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-2YAheiIHNzI.html I recommend you to have a look at the whole datascience playlist - ru-vid.com/group/PL4GjoPPG4VqOmyh7hQ730evtLaz04LwSf

@@learnerea Thank you so much. Love you guys!

There are several potential approaches you can take if you find autocorrelation in the residuals of your ARIMA model. Here are a few options you could consider: Adding additional AR or MA terms to the model: If the autocorrelation is due to a pattern that has not been captured by the current model, adding additional terms may help to capture this pattern and improve model performance. Differencing the data: If the autocorrelation is due to a trend in the data, differencing the data may help to remove this trend and improve model performance. Using a different model: If the ARIMA model is not suitable for the data, you may need to consider using a different model altogether. For example, a seasonal ARIMA (SARIMA) model may be more appropriate for data with seasonal patterns. Modeling the residuals: If none of the above approaches work, you can try modeling the residuals as a separate time series. This can help to capture any remaining patterns in the data that are not accounted for by the primary model.

@@learnerea 🥰🥰🥰🥰🥰❤❤🧡💛 thanks a lot 🙏🙏

I have the same query as well. I can understand the section on checking on stationarity, but I don't see how that's getting incorporated into the subsequent training and model fitting. If the original dataset can be used for training rather than the transformed dataset, what's the use of determining if the data is stationary or not? Did I miss something ? Otherwise, excellent video, clearly explained. Would be interested to see videos on Time Series Analysis using other models such as XGBoost, Prophet. Thank you sir.

>> You may like to revisit the code, you have created >> You can put the code here as well, we will analyze the diff. and can help

Being the Data Scientist, you gotta explore all the posibbilities.. as explained in the video as well... the decision was taken basis on analysis where it was observed that it won't perform better comparatively and it has also been suggested, that we will try making another video where we utilize the stationary data to see the how it performs.. As a learner, your question make sense.. keep asking the questions for clarity

i was wondering the same

diff computes the difference of a set of values, essentially subtracting each value from the subsequent value in an array or list, if can provide the timestamp here, will be able to give you the specific guidence

Stationarity can be on year basis as well.. When you're dealing with time series data that only has a yearly frequency, the approach to making the data stationary is similar to what you'd do with more frequent data, but with some specifics to consider. Visualizing the Data: Start by plotting the data. This will give you an idea of the overall trend, seasonality, and variance. Since the data is yearly, you might not observe any distinct seasonality. python code - import matplotlib.pyplot as plt plt.plot(year, passenger) plt.xlabel('Year') plt.ylabel('Passenger') plt.title('Yearly Passenger Count') plt.show() Differencing: A common approach to making time series data stationary is by differencing the data. Differencing helps to remove trends in the data. You subtract the previous year's observation from the current year's observation. python code- passenger_diff = passenger.diff().dropna() After differencing, plot the data again to see if it appears more stationary. Checking for Stationarity: The Augmented Dickey-Fuller test is commonly used to check the stationarity of a time series. python code - from statsmodels.tsa.stattools import adfuller result = adfuller(passenger_diff) print('ADF Statistic:', result[0]) print('p-value:', result[1]) A low p-value (typically ≤ 0.05) indicates that the time series is stationary. Transformations: If differencing isn't enough, consider other transformations like: Log transformation: To stabilize variance. python code import numpy as np passenger_log = np.log(passenger) Rolling means: To smooth out short-term fluctuations and highlight longer-term trends. python code - rolling_mean = passenger.rolling(window=5).mean() # 5-year window as an example passenger_detrended = passenger - rolling_mean passenger_detrended.dropna(inplace=True) Decomposition: Even though the data is yearly, if you suspect any seasonality or a strong trend, you can use decomposition. The Seasonal Decomposition of Time Series (STL) from the statsmodels library can be useful. python code - from statsmodels.tsa.seasonal import STL stl = STL(passenger, seasonal=13) result = stl.fit() detrended = result.trend deseasonalized = result.seasonal You can then work with the residuals from the decomposition process, which should ideally be stationary.

Welcome

I think, not yet..

It will be great if you can share the time stamp where you spot this point

the dataset is part of seaborn library.. you can just run the code - import seaborn as sns df = sns.load_dataset('flights') you can also download the notebook github link provided in the description

thank you very much for watching

That's an excellent problem statement to choose, little bit of more detail which you might have provided is - >> what sort of model you want to develop >> what is the main purpose/scope of the model etc. lets assume that you want to build a credit risk model and the data which you are taking under consideration, includes the covid period as well. (Before I start, make sure that the data is in relatively balanced quantity & period). Below are the approach which you can undertake - Data Collection: Gather historical credit risk data, including loan performance, defaults, delinquencies, and relevant economic indicators. Include data specific to the COVID-19 period, such as unemployment rates, government stimulus programs, and financial relief measures. Data Preprocessing: Clean and preprocess the data by addressing missing values, outliers, and data inconsistencies. Create relevant features, such as lagged values of credit risk indicators and economic variables, to capture time dependencies. Exploratory Data Analysis (EDA): Perform EDA to understand the data's characteristics and relationships. Explore trends, seasonality, and patterns, paying specific attention to changes during the COVID-19 period. Define the Target Variable: Define the credit risk metric you want to predict, such as default probability or loan delinquency. Feature Selection: Identify relevant features that may influence credit risk. This includes economic indicators, loan characteristics, borrower information, and external factors. Time Series Decomposition: Decompose the time series data to understand underlying trends, seasonality, and residuals, considering the effects of COVID-19. Create a Historical Train-Test Split: Split the data into training and testing sets, ensuring that the testing set includes the COVID-19 period. Model Selection: Choose a suitable forecasting model. In this case, time series models like ARIMA, SARIMA, or Prophet may be appropriate. Consider using machine learning models like Gradient Boosting, Random Forest, or LSTM if you have sufficient data. Model Training: Train the selected model on the historical data, excluding the testing period. Model Validation: Evaluate the model's performance using the testing data, specifically during the COVID-19 period. Use appropriate evaluation metrics, such as Mean Absolute Error (MAE), Mean Squared Error (MSE), or classification metrics for binary outcomes. Model Interpretation: Interpret the model's predictions to understand which factors contribute to credit risk during the COVID-19 period. Feature Importance: Analyze feature importance to identify key drivers of credit risk during the pandemic. Model Refinement: Fine-tune the model and hyperparameters if the initial model's performance is suboptimal. Scenario Analysis: Conduct scenario analysis to assess credit risk under different economic conditions related to COVID-19, such as varying levels of unemployment or government interventions. Model Deployment: Deploy the trained model for ongoing credit risk assessment and predictions. Monitoring and Feedback Loop: Continuously monitor the model's performance and retrain it as new data becomes available. Regulatory Compliance: Ensure that your credit risk model complies with regulatory requirements and standards relevant to your industry. Documentation: Document the entire modeling process, including data sources, preprocessing steps, model selection, and evaluation metrics. Keep in mind that the unique challenges posed by the COVID-19 pandemic may require you to adapt your model and data sources to reflect changing economic conditions and government policies. Regularly update and refine your credit risk prediction model to account for these dynamics.

you can mention the function name which has been used in the video from statsmodel but you do not find them in the model now.. we will try to find and help you with closest alternative function if that doesn't exist

​@@learnerea​​@learnerea can you make a new video on implementation of arima.. On share market dataset or weather dataset

Here you go - github.com/LEARNEREA/Data_Science/blob/main/Scripts/time_series_air_passengers.py

Hi Meronika, you can find that using - file name - time_series_air_passengers.py url - github.com/LEARNEREA/Data_Science/tree/main/Scripts