Dealing with MISSING Data! Data Imputation in R (Mean, Median, MICE!) 

I'm glad you are liking the content :) That is a good point you've made. I'll probably validate future imputations with future models that I'd like to go through and compare and contrast down the road.

@@SpencerPaoHere Hi thx for the video, may i know where can i get the data set and code you used?

@@larrygoodnews As requested, here is a link to the code and data github.com/SpencerPao/DataImputation

Hi! Thank you for your support! :)

The More NAs in the dataset = the less effective the method is unfortunately. If you have lots of NAs, perhaps in your dataset eliminate the observations that might not make the most sense, thereby decreasing the total amount of observations. However, at the end of the day, go ahead and try it out! Compare your imputed observation accuracy with whichever target variable in question.

The typical train/val/test will go a long way when swapping the imputed datasets. You can get a general sense on how well a model performs on all of the datasets.

That depends. Are you expecting negative values? If not, then you may have to do some data cleaning in your original dataset. There are some unorthodox ways of imputation where you can perhaps set a boundary for imputations. But the method behind it is sort of a mystery to me. I attached a link which may be more of use to you. stats.stackexchange.com/questions/116587/multiple-imputation-introduces-negative-values-dataset-still-valid

Are you referring to 14:00 in the video? Line 80 is doing just that. (new dataset with the estimated values of missing data)

No rule of thumb in particular!

I'm flattered :3 I don't tutor at the moment; however, if you have any "quick" questions, I do regularly answer comments on all my videos for free.

Hmm. The link: github.com/SpencerPao/Data_Science/tree/main/Data%20Imputations works with me.

1) You could just extract that 1 specific imputed column from the imputed dataset and replace it to the raw dataset. 2) depending on your use case, you can use whichever column you so desire to be imputed (typically when you want a filled column) 3) Pooling columns? As in merge 2 different columns together? I mean you could.....

Thanks! Yes. You should be able to run the MANOVA on the imputed datasets. However you will have to run the algorithm on each set individually. Then aggregate the model results thereafter.

@@SpencerPaoHere I was successfully able to run a Manova for each of my imputed datasets (m= 10). As an example, here’s the code I used to run the Manova for the 10th imputed dataset: model.10 = manova(Anxiety ~ Treatment + DepStatus + Treatment_X_DepStatus + BLanx, data = finished_imputed_data10). I’m still however having trouble aggregating the results, and I’m not sure if the code I’m using is correct. Here’s the code I’m using for this: pooled_model = with(imputed_data, manova(Anxiety ~ Treatment + DepStatus + Treatment_X_DepStatus + BLanx)). Does this seem correct? Here’s the error message I’m receiving: Error in (function (cond) : error in evaluating the argument 'object' in selecting a method for function 'summary': Problem with `summarise()` column `qbar`. ℹ `qbar mean(.data$estimate)`.✖ Column `estimate` not found in `.data`. ℹ The error occurred in group 1: term = Treatment. In addition: Warning message: In get.dfcom(object, dfcom) : Infinite sample size assumed.

​@@felixangulo4677 Hi! Yes. Try saving the model weights! (a model for each imputed dataset) And, once you have the model weights, you can ideally aggregate or do some form of model aggregation. You can also choose which model to go with based on best model performance. I did a video on just this: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-6pw9IDFxWFM.html

No worries! I am still active on this channel. It might just be a matter of comment volume, but I'll probably get around to it :) 1) Yes. I used a single dataframe that was imputed via the MICE method. 2) You can also copy and paste my code from github ! github.com/SpencerPao/Data_Science/tree/main/Data%20Imputations Other than that, I am unsure what the question is.

You can still use Random forest for a dataset that has numerical and categorical features! It would typically come down to comparing your model output to see which one performs the best and decide from there.

least squares? That's built in the Rstudio framework. Checkout out lm() EDIT: Least squares imputation: check out pcaMethods()

@@SpencerPaoHere thank you for the reply. Do we have same for python package?

@@mohammedabdulkhaliq2644 Pandas has an interpolate function per column. pandas.pydata.org/docs/reference/api/pandas.DataFrame.interpolate.html

Hi! I'm glad you liked it! :) The pool function averages the estimates of the complete data model (and outputs a variety of statistics related to those features) The complete function fills in the missing values with the imputed values. AND yes. This function imputes ALL of the features based ONLY on 1 imputed dataset that you have chosen. See complete(data, m) where m represents which imputed featured you would want to plug into your original dataset. You can use the finished dataset with other programs. All you need to do is write the file to a CSV and load the file into a different program.

@@SpencerPaoHere thanks for the clear answer. Is it possible to carry out an EFA (exploratory factor analysis) utilising the pooled function? If yes, is it possible for you to illuminate us with a tutorial on how to carry it out? :). Thanks in advance

​@@PoetenfranLevanten Hmm. I have done a video on Factor Analysis. You can check that out! The only thing different you would do is to apply the pooling function on whatever data you might have. THEN, go ahead and utilize the FA function as noted.

@@SpencerPaoHere Thanks, how would the code look like if I apply the pooling function? Tried to do that on a dataset I have but received an error message in R.

@@PoetenfranLevanten Hi! you'd want to apply the pool function on a model object. i.e pool(fit) It won't work if you plug in a dataset.

When you have more than 20% of the rows missing for a column it might be best to drop them since the imputations are largely based off of the rest of the rows. So you may be getting garbage imputations (it would be intersting to see on a simulated smaller chunk of data if costs were a factor) The basis on which imputed dataset to choose is somewhat arbitrary. I'd just do batch inferencing to see which gets better predictive results. Though the resulting differences can be more or less neglible.

​@@SpencerPaoHere Well said. But what if the column is essential for the analysis?

@@atthoriqpp haha yeah. You're going to need more data. Or, you can do a 'haily mary' and see how the imputations fair. (A lot of testing will be needed to ensure you are confident about the results.)

@@SpencerPaoHere Thanks for the answer, Spencer!

The mdmb seems like a brand new package. (10/13/2022) -- I am unfamiliar with it. Though from taking a peak at the documentation, it seems that the mdmb package uses MLM and or Bayesian estimation. This is definitely more "narrow" than MICE, which there are many more models to choose from. However, you may get better results? Not sure. You'd have to try on your data.

@@SpencerPaoHere Thank you for your explanation. May I know, how many percentage of missing data is good for MICE? or other imputation methods?

@@kalemguy I think if ~20% of your rows have a missing value, then you can probably impute. However, if you are missing 80% of your data, it might be advisable to just drop the feature altogether.

@@SpencerPaoHere Thank you very much for your insight...

My code is located here: github.com/SpencerPao/Data_Science/tree/main/Data%20Imputations

I believe you don’t even need to cast your objects to be a certain type. Try using the tidyr and dplyr package. You can define the features as a specific data type where you don’t have to cast over when reading the data.

OK-will do! Thank you! I just have two follow-up questions if possible: 1. I got an error about logged events. My reading thus far tells me that these essentially get at perfect prediction and that R will not impute those values when this occurs. What I am confused about, though, is that (a) the number of logged events reported is greater than the number of missing values and (b) looking at my imputed data set shows no NA values. Do you have any read on these two pieces? 2. As opposed to pooling estimates and whatnot across all the imputed data sets, is it permissible from a statistical standpoint to select one of the imputed data sets to use for all analyses as opposed to the pooling procedure that you did here for the regression)?

@@markelov 1) There can be a multitude of factors that cause the issues of logged events. Try printing out the loggedEvents of your MICE object. (Variable$loggedEvents) -- That might give a hint. Also, sometimes in a dataframe, NA values can be "null" values as well. So, you may need to run a few different checks to see if the null values actually do exist in the data. 2) You could just use one imputed dataset but it might not be a representative of all data. Pooling gives you a wider band and a concentration of outcomes in an area.

Thank you!

Yep! In layman terms, the with() is inserting each of the imputed dataframes to the glm() model. As a result, at 15:23 or so, you will have 5 glm models where each model builds off of each individual dataframe. You can then do an average of the weights (or provide standard errors with the model weights) and evaluate predictions etc..

@@SpencerPaoHere Thanks for the answer! :) One last question: after using the function with() you just used the function plot() to have some information and analyse it. But using the with() function I was not supposed to be abble to extract the "new" values to substitute my NAs too? (The "combined" values)

@@TheFabricioosousaa Can you provide a timestamp? (or a line of code); But you can think of it as having different models for similar but unique datasets.

@@SpencerPaoHere For example, if I have this: #INSTALL AND LOAD MICE install.packages("mice") library(mice) #IMPORT DATASET: library(readxl) data_md

@@TheFabricioosousaa Yep! And, that should occur after your data_imp variable -- imputation should occur. (Try printing it out to see if that is what you are looking for form an imputation POV) Then, you will use the imputed data set(s) for your training/testing process.

Hmm. Yeah. It must be related to your input data. Try to one hot encode your categorical variable and see what happens.

@@SpencerPaoHere I tried it and I got the same problem unfortunately. I tried to create a new dataset of these categorical variables and I imputed them and it works. But when I impute them with the main dataset I get this problem :(

@@mahmoudmoustafamohammed5896 What's the stacktrace? There might be an issue with your other variables perhaps. It's strange that one set of features don't work but the others do? And combined?

1) The 'm' term refers to the number of times you want to impute your dataset. I was using the term "cycles" colloquially. So, if m = 5, you are expecting 5 different datasets with different imputed values. (more or less) 2) maxit : (in the mice package) just refers to the number of iterations taken to impute missing values. This is related to whichever objective function you utilize and thus uses the maxit as the upper ceiling for its iterations. 3) The "seed" is a deterministic number generator. It doesn't matter what value you use for seed(numeric) as long as the numeric is consistent among all your experiments. Hope that helps!

That helps a lot, I understand the video a lot more now! Thank you! I know you learned a lot at post-secondary, but I'd love a video explaining your background, and tips/tools to self-learn to other people (if you have any)! We all want to be experts like you one day.

@@kaili3477 haha thanks I appreciate that. Maybe when this channel gets bigger, I can do a video autobiography of some sorts.

Perhaps this might help you? www.researchgate.net/post/I-am-looking-for-a-Matlab-code-for-Multiple-imputation-method-for-missing-data-analysis-can-anybody-help-me

@@SpencerPaoHere I appreciate your help.

In my later videos, I handled the background noise. Sorry for that!

Strange. I've ran install.packages("mice") and the library was able to be installed -- then run library(mice) to load the package in your environment. Perhaps updating your Rstudio might do the trick?