Feature Engineering-How to Perform One Hot Encoding for Multi Categorical Variables 

Same here also

same here.....i always found your videos very usefull

pro i have problem that is iam using two different datasets one from kaggle and one from local but the problem is when making hot encoding when ever i try doing this flightdata = pd.get_dummies(flightdata, columns=['OriginCityName', 'DestCityName']) df = pd.get_dummies(df, columns=['OriginCityName', 'DestCityName']) # Ensure both datasets have the same dummy variables flightdata, df = flightdata.align(df, join='inner', axis=1) but the public datasets have many more categorical than the local how can i solve it ?

Analytics and analysis of data in all the fields ,for automation!

I also have this doubt

We can see from value_counts(). Where the count number starts dropping too much, we can take till that category

you can use same technique 10 most frequent category for source and destination

same here'

This method is for feature engineering of categorical feature, which means that the data columns that do not have numbers in them, they have strings which means something for the application that captured or uses this data. When this kind of Data is used for model building you have to think the big picture that the models are trained using mathematics in the background so the common sense is that you cannot use string to do calculation so during feature engineering there are techniques like these that convert the categorical feature to numbers using encoding technique. However, using one hot encoder technique brings the added burden of increasing the number of columns in the dataframe which in terms of efficiency of training decreases the overall performance of the trained model. One hot encoder is great to use when you have few categories in a column but if you have hundreds of them then the method of repurposing one hot encoder only for top 10 categories was proven to be beneficial and reduced the burden of added features or dimension. I hope you get the point now. I would always want you to think not just the detail but the bigger picture to understand the reasoning behind them and then things will be easier to understand. All the best

@@rabintimalsina9263 check out my response below. Hopefully it helps you

@@navneetmohit1 Yes sir I understood.... Thank you for the patience and the explanation 😊. Actually I ws following the path told by krish sir n at that tym I wasn't knowing machine learning models and techniques of data Pre processing that's y I ws confused......

@@navneetmohit1 what if we have 7 to 8 categorical features after one hot encoding it may results to curse of dimensionality how can we perform at that moment ?

use naive bayes

As said this will not always work..this works only when u have a imbalanced categories in ur features. Still I will be uploading more videos to handle different scenarios

agree with you. Moreover, this kind of encoding can overfit

Open cmd as admin and type 'pip install jupyterlab'

What should be the conditions?

You can find out top 10 for each column and pass this in the function.

lead Data Scientist, Panasonic , bangalore as per Github

Thats why you should first merge two datasets if there are like in video train and test and then perform encoding.

@@akashravindra.. but you do not know how many values will be getting in real life in production. Let’s say you create 4 different columns A,B,C and others as fourth column based on 4 categories. Now let’s say you get totally new value E on that column where you created dummy when making predictions on production data how will we handle this ? I get it it works if you know all values on the columns and create dummy columns.

@@abhinasneupane2392 You should create column transformer using pipeline. And use it. And One Hot encoder from Sklearn is used for that.

i think, label encoding is used for only target variables.

Encoding is always done on entire dataset because your test data cannot have categories and you expect the model to predict the output based on that.

take %age ratio by applying 80 20 rule if 80% of data lying in top 10 to 20 then you can apply this otherwise convert into Target guided mean value

Frequency encoding or target encoding maybe

I think mean encoding is better in a way because it gives different values for different categories and later you can standardize or normalize them. But in this all the categories are treated as 1 which signifies huge loss of data.

Binary its good to use but honestly you can simply replace the one of the binary value with zero instead of one hot encoding and then dropping the original column. You can use str.replace() function and replace one binary value.

so their is no need to this u can simple do get dummies, after all there are only 4 categories. You only have to apply this method until you have many categories

@@JaySingh-qj3hv instead of seperately doing for x1,x2,x3..... cant we do that like this, for i in data.columns: top_10=[x for x in data.i.value_counts().sort_values(ascending=False).head(10).index] for label in top_10: data[label]=np.where(data['i']==label,1,0) data[['i']+top_10] when i try to do this i am getting that i is nt an attribute of data, how can i resolve this ,can somebody help

@@snehithoddula7905 Here i itself is each and every column present in dataset.So write data[i] instead of data['i']

www.kaggle.com/c/mercedes-benz-greener-manufacturing/data you can download from here

Suppose if you need to perform clustering how will you move ahead then without performing this step for fix category

features extracted automatically by cnn why do you need to extract features separately..all u need to specify the kernels that it

@@georgedong3789 i want to extract LBP feature from image store them in a pickle file and thn want to use that pickle file( that contains LBP feature of image) to cnn....so that model accuracy can b improved...but unable to do....plz make a video on this....plzzzzz Sir