Would you be uploading your Advanced Machine Learning for Analytics course as well? I think I found the teacher I want to learn from! Thank you for your service!
The deep learning course is already there, look for the playlist “Deep learning concepts simply explained”. I am planning to start the “Deep forecasting course: advanced time series forecasting” soon.
Hi Pedram, one question. Why do you scale the whole dataframe in this notebook? In the other notebooks you only scale the X's; in particular, you define X_train_sc = sc.fit_transform(X_train) and X_test_sc = sc.transform(X_test). However, in this notebook you do df_sc= scaler.fit_transform(df), where df contains the y and the Xs. So in this case, you scale both (y and X) and potentially introduce some bias by fit_transforming all: the train and the test split (since you take them from df). Why doing that?
Good point! Yes, we typically use the training set for fitting the scaler and then apply it to the test set. However, for this dataset, as I mentioned in the video, I was a bit lazy and scaled the entire data since it's a small cross-sectional dataset with no outliers. You can certainly do it properly (fit_transform(X_train), transform(X_test)) and let me know if you find any differences. Regarding scaling 𝑦, it doesn't impact model performance much but can help with numerical stability and faster convergence during optimization. The model's predictive power remains the same. Interpreting RMSE would be different, though, because the unit is scaled.
