No video :(

One-Hot, Label, Target and K-Fold Target Encoding, Clearly Explained!!!

Подписаться 1,2 млн

Просмотров 48 тыс.

50% 1

In theory, discrete variables, or features, are easy to use with machine learning algorithms. However, in practice, it's not always so easy and we often have to transform discrete values, like favorite colors, into numbers. There are lots of ways to do this, and this video walks you through 3 of the most popular methods.
English
This video has been dubbed using an artificial voice via aloud.area120.... to increase accessibility. You can change the audio track language in the Settings menu.
Spanish
Este video ha sido doblado al español con voz artificial con aloud.area120.... para aumentar la accesibilidad. Puede cambiar el idioma de la pista de audio en el menú Configuración.
Portuguese
Este vídeo foi dublado para o português usando uma voz artificial via aloud.area120.... para melhorar sua acessibilidade. Você pode alterar o idioma do áudio no menu Configurações.
If you'd like to support StatQuest, please consider...
Patreon: / statquest
...or...
RU-vid Membership: / @statquest
...buying my book, a study guide, a t-shirt or hoodie, or a song from the StatQuest store...
statquest.org/...
...or just donating to StatQuest!
www.paypal.me/...
Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:
/ joshuastarmer
0:00 Awesome song and introduction
1:24 One-Hot Encoding
3:25 Label Encoding
4:39 Target Encoding
6:27 Target Encoding with a Weighted Mean, or Bayesian Target Encoding
9:56 K-Fold Target Encoding
#StatQuest #DubbedWithAloud

Опубликовано:

14 авг 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 162

@statquest Год назад

To learn more about Lightning: lightning.ai/ Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/

@matthewmechtly5026 Год назад

It's mind-boggling how much better Josh is at explaining complicated topics than anyone else.

@statquest Год назад

BAM!

@bridgetelly8554 Год назад

Thank you so much for your videos, this is by far the best educational Machine Learning channel I’ve ever come across

@statquest Год назад

Wow, thanks!

@sungminson3658 4 месяца назад

This is literally the best explanation about statistics & traditional ML model. I am so lucky to see your video with my journey of data science started.

@statquest 4 месяца назад

Thank you!

@Li-dv8lr Год назад

Much better explanation than what I had at class!

@statquest Год назад

BAM! :)

@michaelsipes1570 Год назад

What class?

@myfoodfeast Год назад

Hi Josh. Just came across your channel. Your method of explaining is so concise, clear and appealing. Definitely I would learn a lot from this channel.

@statquest Год назад

Awesome, thank you!

@LuizHenrique-qr3lt Год назад

Hi Josh I wanted to thank you for your content, I'm finishing your stats playlist it's very good. Statsquatch has become my friend. Big hug straight from Brazil!

@statquest Год назад

Muito obrigado! :)

@moazhendy9394 Год назад

I really loved your explanation and your sense of humor. I really did!

@statquest Год назад

Thanks!

@marcom5873 4 месяца назад

Love the dry humour in your videos 🤣. Great content too!

@statquest 4 месяца назад

Thank you!

@amirrezaabedini9795 6 месяцев назад

thank you your explanations are always simple and clear.

@statquest 6 месяцев назад

You are welcome!

@SnipeSniperNEW Год назад

but what happens in inferring? say you trained a great model and now you are predicting the new data, do you use the mean of the old data or the mean of the new data? if you use target encoding, well in the new data you don't have a target? so what now?!?

@jamesgunn1560 Год назад

Typically u took the avg of the training data for each category for inference

@statquest Год назад

BAM! :)

@shahnurislam5100 Год назад

For target encoding. Usually you store a dictionary of the encodings for the categorical data. If it’s an unseen value from the training set. Best practice is to use the mean

@MilenkoCurcin Год назад

Totally great explanation, congratulations

@statquest Год назад

Many thanks!

@razapoetra9355 4 месяца назад

Never learning ML this fun before 😂

@statquest 4 месяца назад

bam!

@monkeystoot Год назад

How do you use k-fold target encoding for a test data set, since blue now has several distinct numeric values as a predictor in the training set?

@statquest Год назад

Great questions - with new data, you can use all of the original training data to find the best value.

@hasandaaboul5322 6 месяцев назад

Hey Josh, great job. Thnak you a lot!

@statquest 6 месяцев назад

Thank you!

@onkarpatil4262 Год назад

Awesome as always ☺️👏

@statquest Год назад

Thank you! :)

@dvergnordicalfar4052 Год назад

Hi Josh, thanks for explanation. But I want to know how to transform the unseen data using k-fold target encoding? Is it oke to use mean value of the transformed category? Thanks before

@jizhiliu1878 Год назад

I have the same question

@statquest Год назад

Great questions - with new data, you can use all of the original training data to find the best value.

@lakshmanbharath 11 месяцев назад

Hi Josh, a heartfull Thank you for sharing these encoding techniques I have one doubt; it may look stupid, but I just want to clarify it with you. On 13:40, the encoding of green colour with target value 1 is 0.42, and below that, green colour with target value 1 is 0.67. So when encoding transforms the new data, will system change the green colour to 0.42 or 0.67?

@statquest 11 месяцев назад

When encoding new data, we'll use all of the training data to convert the option "green" into numbers. Thus, I believe we'll convert it to we'll convert it to ((3 * (2/3)) + (2 * (3/7))) / (3 + 2) = 0.57

@kaus0463 Месяц назад

great!

@statquest Месяц назад

@lbognini Год назад

Great video. 👍🏽 I find it less confusing, however, to say categorical or qualitative data instead of discrete data. Numeric data can be discrete (integers)

@statquest Год назад

Noted

@simoneparvizi775 Год назад

I NEED YOUR HELP At 4:20 our brother Josh said that some ML models might have problems with label encoding and as always he explains it in a beautiful way. I'm writing my Thesis and I need at least one paper that confirms this (although it makes totally perfect sense...). It's been 2 days of and still can't find anything...anybody can help me?... Thank you people, stay smart and keep learning

@statquest Год назад

I think part of the problem is that the issue with label encoding is pretty obvious so no one thinks to publish about it, and reviewers might think it is too obvious to be published. Sort of like publishing a paper that says 2 + 2 = 4. However, I was able to find this: arxiv.org/pdf/2201.11358.pdf

@simoneparvizi775 Год назад

@@statquest Josh, thank you do much. Really, it was already enough the video you made. Can't imagine how much i appreciate it, wish you the best

@user-fe7fd3yc4s 4 месяца назад

Thank you for explanation. Is it possible to buy these slides in this video? I didnt find them in Illustrated Guide To Machine Learning. How to buy all your slides?

@statquest 4 месяца назад

Thanks! Unfortunately my slides are not available. :(

@ameerkamal9305 Год назад

Hi josh .... thanks for your extreme efforts.... I hope to see a statquest about lightgbm and catboost.....

@statquest Год назад

This is actually a lead in to CatBoost (as well as word embeddings).

@kaicheng7271 Год назад

it's time you introduce Quadruple Bam!!!!

@statquest Год назад

Not yet - I'm waiting for 1 million subscribers.

@khaikit1232 Год назад

Hi Thank you for the video but I just have some questions and hope I get clarifications from you: 1) Could you explain why you were concerned about data leakage when doing the k-fold target encoding? The data leakage I know is when some test set information is incorporated into the model training. Is the dataset shown in the video supposed to be the full dataset or is it just the training/testing data for a machine learning model? 2) Target encoding feels very different from one-hot encoding and feels like we are replacing the original column with a derived column. Hence I was wondering, what is the intuition and reason behind target encoding? In what situation would I use it over one-hot encoding?

@statquest Год назад

1) There are different ways to corrupt training, and one of the ways is to use the target values in the training data to impute values that are also used in the training data. 2) One-hot-encoding is great when there are only a few options. But if you have a lot of options, target encoding can be better.

@speedtent 5 месяцев назад

thanks for a great video! i am trying to apply k-fold target encoding on my train and test data. i target encoded my train data using k-fold target encoding just like the video, but how should i encode my test data ? If the feature is BLUE, should i get the mean of BLUE (target encoded) in the train data and use it for test data? OR should i just use the whole train data to get new target encoding values for the test data?

@statquest 5 месяцев назад

Use the full training dataset to encode the test data.

@lautarocisterna3339 Год назад

Hi Josh! Great video. Are you planning to add to these videos how to apply them in Python? Thanks!

@statquest Год назад

That's a good idea. I'll keep that in mind. I show how to one-hot encoding in my video on XGBoost ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-GrJP9FLV3FE.html but I haven't shown how to use Target Encoding.

@Futureyouth-be1bo Месяц назад

i have two differnt datsets those have columns 750 after one hot encoding and aligning manually now my problem is i'm facing long load time in the process so how can i reduce those columns si

@statquest Месяц назад

If you don't store them that way, they should load faster. If the datasets are not super huge, you should be able to one-hot-encode in RAM and then run it through your analysis.

@tarun8122 11 месяцев назад

here, at 7.54 I see that while calculating weighted average for 'Red', you mentioned 'overall mean' as 3/7 however Isn't it incorrect?, shouldn't it be '1/7'? please correct in case I am claculating it wrong?

@statquest 11 месяцев назад

Overall mean is the average of the "Loves Troll 2" column for all colors. The "option mean" is the average of the "Loves Troll 2" column for a specific color (like "red")

@bkleinman123 Год назад

Hey Josh, Love the videos. I'm left with one question: is there anything we can do when we are doing multiclass classification and need to transform our predicted variable so that the algorithm isn't working with string data?

@statquest Год назад

See: stats.stackexchange.com/questions/452022/can-target-encoding-be-performed-on-a-multi-label-classification-problem

@nitinsiwach1989 9 месяцев назад

Hi Josh. You rock. Can you help me understand how the k-fold target encoding results are used on the test data that appears months after the model is trained and is in deployment?

@statquest 9 месяцев назад

Once trained, you use all of the training data to encode future data.

@jfndfiunskj5299 Год назад

For one-hot encoding, shouldn't there be one less position to avoid the dummy variable trap?

@statquest Год назад

That only applies for models that depend on design matrices (like linear models or logistic regression).

@eoinwhelan4783 Год назад

Great video! One question - in using kfold target encoding it seems like you're being counter productive in producing encodings that is useful to a model. For example a node of an XGBoost tree will look at 'colour' and split based on the value of the encodings, treating it as a continous variable. But what if by kfold encoding maybe 'Blue' has values between [0.3, 0.5] and 'Red' has values between [0.2, 0.5 ] - now you've disrupted the models ability to effectively split on this node! Appreciate any thoughts on this you might have

@statquest Год назад

First of all, I'm pretty sure XGBoost efficiently uses one-hot encoding for all discrete features, no matter how many options there are (by implementing sparse matrices). However, assuming you still have to translate the discrete features to continuous values, if there is overlap between Red and Blue, than that might actually be fine. The important thing isn't to split Red from Blue, but to make good predictions, and that might still happen, even with overlap.

@cyberwarlus2758 Год назад

Isn't the formula of weighted mean is (num of option pos targets + m*overall mean)/(n+m), since n*Option Mean = n * (sum of option targets)/n = sum of option targets = num of option pos targets?

@statquest Год назад

Yep, you can write it either way. I like the way I wrote it because it makes the weighting explicit and easy to see. The other thing is that 'n' does not have to be the number of rows, and thus, it doesn't have to cancel out the denominator of the mean. So that's something to think about as well.

@Monkey_uho Год назад

Hey! I have a question regarding K-Fold target encoding. While I grasp the concept and the reason behind obtaining multiple values for the same category, I am uncertain about how to encode new data. Specifically, if I train a machine learning model using K-Fold target encoding and subsequently wish to test the model with new data, for which I do not have the target value, how should I go about encoding this data? Should I take for instance the mean of all values corresponding to "red" ?

@statquest Год назад

Yes, exactly. You use the full training dataset to encode new data when you are using the model to make new predictions.

@Monkey_uho Год назад

@@statquest Thank you very much! I have another question though, what if I want to encode the target itself ? In my problem, I have a classification problem where the target can take ~250 different values. I guess Label encoding or One hot are not suitable for this, and I don't really know if I can use target encoding to encode the target itself... Is there another way to encode data in this situation.

@statquest Год назад

@@Monkey_uho Depending on your method, you might need to encode the target. For example, if you were using a neural network, you would just have one output per target value. Or, if you are using a tree based method (like xgboost), you can probably just assign a number to to each target value - this is because you don't use the target value itself to determine how to build the tree.

@michaelbohnen636 11 месяцев назад

Hi Josh. How can you apply these methods in regression models? One Hot Encoding seems self-explainatory, but the weighted mean or the catboost methods seem, like they need additional steps...

@statquest 11 месяцев назад

For the CatBoost method, they quantize the "label value". catboost.ai/en/docs/concepts/algorithm-main-stages_cat-to-numberic

@natannvw1 2 месяца назад

Is there any tool like sk-learn that holds the Kfold or any of these advanced encodings?

@statquest 2 месяца назад

Good question! I don't know.

@adiskaop Год назад

Hi Josh! Thank you very much for sharing information regarding statistics. Do you have a video regarding linear model and linear mixed model? I have been struggling with these two. Have a nice day!

@statquest Год назад

Here's my series on linear models: ru-vid.com/group/PLblh5JKOoLUIzaEkCLIUxQFjPIlapw8nU

@adiskaop Год назад

@@statquest thank you, but the linear mixed models is not in there, would love to watch your explanation about it

@statquest Год назад

@@adiskaop I'll keep that in mind. However, I'll tell you the basic idea of linear mixed models boils down to a model where you don't have enough data to estimate all of the parameters in a standard linear models. So, instead of estimating parameters, you just make assumptions.

@tompease95 Год назад

This is the first Statquest I've ever watched which I've actually found quite confusing - two things it leaves unanswered: a) It's quite a leap to go from encoding categorical features in a way that each category always has the same numerical representation (e.g. blue is always 1), to a way that doesn't, and there is no explanation of why that works. I'm thinking - "How does the model know that 0.22 and 0.5 are referring to the same thing - isn't that an issue?". b) With k-fold TE, how do you encode the test data?

@statquest Год назад

a) The model doesn't know that 0.22 and 0.5 are referring to the same thing. Is this ideal? Probably not - leaving the features as discrete options to begin with would probably be better. But if we don't have that choice, we have to make some sort of compromise. b) We just use the full training dataset.

@tompease95 Год назад

@@statquest Ah okay, that makes sense re. a), thanks so much for the response. For b), makes sense that you would use the whole dataset for training, but how would you encode the categorical variables if you wanted to use the model with new data/in production?

@statquest Год назад

@@tompease95 To be clear - when you have new data and your model is in production, you use the full training dataset. In other words, given the 7 rows of data in the example in the video, you would use all 7 rows to encode new data when using this in a production setting. You can precompute these values from the full training dataset so the encoding can be very fast.

@heteromodal Год назад

hey Josh and thank you as always for an amazing tutorial! :) when we perform k-fold target encoding - it applies on all the data, and then you split to train-test sets? and then - we you have new data that you feed to the trained model and want to get predictions about - what would be the values you assign to the categorical features? the ones you'd get by just performing target encoding on all the initial data you had (train + test)? thank you!

@statquest Год назад

Typically, we would split the data into training and testing bits. Then apply k-fold target encoding on the training data and build the model. Then, when testing, we use all of the training data to encode each row of testing data and run it through the model.

@heteromodal Год назад

@@statquest thanks!

@lutzsommer4708 Год назад

Great work like always! What to do, when target encoding results in the same number for two labels?

@statquest Год назад

That's fine - because the goal isn't to find unique values, the goal is to find values that help make the best predictions.

@SkN097 Год назад

If my column has 3 categories, should I convert it into 2 columns? (because with the information from two columns, I can determine the value of the third category). I understand that this conversion is commonly done in linear models to handle multicollinearity caused by the additional columns, but I'm unsure about its application in models like decision trees. Great videos!

@statquest Год назад

The way design matrices are made for linear models is different than one-hot-encoding because they, specifically, need to deal with multicollinearity. However, most other ML methods, including decision trees and neural networks, don't have that problem, so for them, if you have 3 options, you have 3 columns.

@shridharvaidya-fv6bz Год назад

Hello Josh, I have a question how can we provide value for testing data when we don't know the outcome?

@statquest Год назад

When we have new data that we want to classify, we simply use all of the training data to calculate the encoding values.

@benstallone6784 Год назад

So how do we assign values to a categorical column that underwent K-fold Target Encoding for training when it is time to make actual predictions?

@statquest Год назад

If you have new data to make a prediction for, you use the entire training dataset.

@125Markus Год назад

Hey, how can i use K-Fold Target Encoding when I have more than two targets? (not only 0 or 1) In my dataset are 1.604 different classes.

@statquest Год назад

See: towardsdatascience.com/target-encoding-for-multi-class-classification-c9a7bcb1a53

@samore11 11 месяцев назад

The methodology is very clear here. But one thing that bothers me is how a categorical feature can now have different numeric values - like Blue being .22 and .50. Let's say that all people who liked Blue also like Troll 2, and nobody else liked it. It seems that relationship would be easily recognized in a model if Blue was kept as is, or was given the same numeric value, but now it could be masked with the encoding techniques giving Blue different values. Am I missing something?

@statquest 11 месяцев назад

Yes, when you have crazy situations where there is a super tight correlation between a category and the label, this can cause problems. Thus, the folks that created CatBoost came up with a work around. To learn more about that strategy, see: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-KXOTSkPL2X4.html

@chenzeping9603 Год назад

Hey! What if my label is also categorical? How do I do target encoding then?

@statquest Год назад

If you just have 2 options, it's the same as here. Otherwise you use counts for each option as described here: catboost.ai/en/docs/concepts/algorithm-main-stages_cat-to-numberic

@nov1195 2 месяца назад

When I get stuck in comprehending any ML concept, I turn to StatQuest.

@statquest 2 месяца назад

bam! :)

@nov1195 2 месяца назад

@@statquest Double bam!

@DanishAlam-lp2cq Год назад

Hi Josh.. Another great video.. Just curious, if we want to encode postcodes or something similar like user-id's, customer-id's where the target variable is not available which technique should we use?

@statquest Год назад

All of the methods I know of require some sort of target.

@DanishAlam-lp2cq Год назад

@@statquest Thanks Josh.. If possible could you please cover some videos on Contextual Anomaly detection .

@statquest Год назад

@@DanishAlam-lp2cq I'll keep that in mind.

@blackpilledbuddha4944 Год назад

is there a sklearn package to do the K-fold target encoding ?

@statquest Год назад

Not that I know of, but there are lots of good examples on the web. Here's one: www.kaggle.com/code/anuragbantu/target-encoding-beginner-s-guide

@blackpilledbuddha4944 Год назад

@@statquest thanks, will have a look

@Makumazaan Год назад

Well, it is explained for categorization, what about regression?

@statquest Год назад

Everything is the same for both methods (classification and regression). Just replaces the 1's and 0's in the target variable with continuous values.

@justinwhite2725 Год назад

13:33 okay...but how do you use this trained model? If you wanted to make a prediction using k-target which of the numbers for that color would you use? (Ps - for colors I'd use their HSV values, or for simple colors like this I'd just use their hue which is a simple spectrum)

@statquest Год назад

You can use the entire training dataset for new data.

@BlueRS123 Год назад

@@statquest Could you please elaborate on how would you do this? I'm probably missing something obvious, but thanks in advance! Is it that you apply the K-Fold target encoding to the new data based on the training data?

@statquest Год назад

@@BlueRS123 When we have new data, instead of splitting the training data into k-folds, we use all of it to calculate the option mean and the overall mean.

@lucho8731 8 месяцев назад

Hello, I had watched some video but don't know the meaning of "Bam". Can anyone explain for me pls

@statquest 8 месяцев назад

Sure! If you want to learn about "Bam!", check out this StatQuest: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-i4iUvjsGCMc.html It's clearly explained!!! :)

@gandalfrj9364 Год назад

Question . How did you arrive at the conclusion that the weight of the overall mean should be 2 ?

@statquest Год назад

Answer: See 7:23

@khushalkumar31 7 месяцев назад

It's not clear why you choose m = 2. Can you explain again or give me some resource to read?@@statquest

@statquest 7 месяцев назад

@@khushalkumar31 Ultimately, this, and all hyperparameters in any model, should be determined with cross validation. In other words, try a bunch of values (maybe 1, 10, 100, 1000) and see which one works the best. For details on how cross validation works, see: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-fSytzGwwBVw.html

@kom_senapati Год назад

BAM!!

@statquest Год назад

@simoneparvizi775 Год назад

If anybody can help me understand the ONLY thing that's not clear to me: - In K-Fold target encoding, based on what do you choose the value of the weight? - Moreover, when I try it, I have results slightly above 1 (like 1.04,1.07, etc). It shouldn't be right? But I can't get why tho

@statquest Год назад

I talk about how to think about what would be a good value for 'm' at 7:11 . However, if you have a large dataset, I've seen a lot of people just set it to 10. Also, you the output should always be between 0 and 1.

@willw4096 11 месяцев назад

3:03 5:33 6:17 6:27 6:42

@jakubstrawa8629 3 месяца назад

So essentially everything greater than 1-fold target encoding is the same as leave one out target encoding?

@statquest 3 месяца назад

What time point, minutes and seconds, are you asking about?

@jakubstrawa8629 3 месяца назад

@@statquest Hi, first of all, thanks for replying! At mark 10:30 youre showing the method for 2 fold target encoding where you divide dataset into 2 subsets. Then at 13:55 you're dividing it into 7 subsets as an example of 7-fold target encoding but at the same time Leave one out target encoding. Isn't it the same? The terminology got me confused. Whats the difference between the terminology ie. 7-fold target encoding and leave-one-out target encoding.

@statquest 3 месяца назад

@@jakubstrawa8629 For k-fold encoding, if k=the number of rows, then it is the exact same as leave-one-out encoding and we can use either term interchangeably.

@jakubstrawa8629 3 месяца назад

@@statquest Ohhh okay, i understand now the difference. So if we have a dataset let's say 10 rows and we divide it into 3 parts (3/3/4) then its only 3-fold encoding but not the leave-one-out?

@statquest 3 месяца назад

@@jakubstrawa8629 yep.

@berke7255 Год назад

in the case of LOO, do we sum the weighted means of the remaining subsets while encoding subset?

@statquest Год назад

For LOO, we use all but one row to calculate the encoding value.

@berke7255 Год назад

@@statquest so how do we decide which subset to choose and calculate its value for encoding another subset

@statquest Год назад

@@berke7255 For LOO, you don't decide. When you are encoding a row, you use all other rows do calculate the value.

@berke7255 Год назад

@@statquest do we sum all other row values or just take the average?

@statquest Год назад

@@berke7255 You just plug the data into the equation for the Weighted Mean, as specified at 11:05. To encode a row, we treat that one row of interest as subset "A", and all of the other rows as subset "B".

@brandonsager223 Год назад

I will stick with the 43k columns

@statquest Год назад

@akshatsrivastava3480 6 месяцев назад

i come here for beep boop boop beep beep

@statquest 6 месяцев назад

bam!

@novicadakovic6188 Год назад

Why did we choose m=2?

@statquest Год назад

This question is answered at 7:23

@justinwhite2725 Год назад

5:06 that would be the average. The mean would be 0. Edit excuse me, I was confusing mean with median.carry on.

@statquest Год назад

bam! :)

@youngzproduction7498 Год назад

Multiple BAMS

@statquest Год назад

@habibiibnnabeel 3 месяца назад

pin for absolutely no reason? who's watching when neural networks took over the world(2152) 👇

@ehxanhaq2883 9 месяцев назад

Like this comment for "Shameless self liking" 😀

@statquest 9 месяцев назад

bam! :)

@thevitorialima Месяц назад

What if you are using a categorical column data to create new features through feature engineering (with groupby in python for instance)? Does it mean that you should do your feature engineering first and then apply K-fold Traget Encoding after doing this?

@statquest Месяц назад

That makes sense to me.