Correcting Skewed Data with Scipy and Numpy

Подписаться 2,9 тыс.

Просмотров 8 тыс.

50% 1

Skewed data can adversely affect your analysis and machine learning models. In this video, I demonstrate five methods for cleaning skewed data using the NumPy and SciPy modules. The methods include taking the square root, cube root, fourth root, log, and Yeo-Johnson transform. I also showcase the effectiveness of each method by summarizing the skewness of the data after each transformation with a bar plot.

Опубликовано:

4 окт 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 37

@marcom5873 2 месяца назад

First time I have seen your videos. This is genuinely a very good video. Very well explained and clear. I am subscribing. The music wasn’t off putting either!

@CJP3 2 месяца назад

Thank you so much!!! I really appreciate it. If there’s anything you’d like to see just let me know!

@marcom5873 2 месяца назад

@@CJP3Sent you an invite on LinkedIn!

@Lendemeier 6 месяцев назад

Bro this is data science ASMR 🤤

@CJP3 6 месяцев назад

Hahaha I didn’t mean for it to be but glad you enjoyed it (I hope) 😂

@officialscience101 Год назад

the on-screen text is a great addition, Dr. P!

@CJP3 Год назад

🙏🏽, I’ll incorporate more in upcoming videos! Thanks for the feedback!

@metinunlu_ 8 месяцев назад

Thank you for the video, subscribed! RU-vid needs more quality content like this.

@MikitaRashetnikau 4 месяца назад

Amazing video I like it's structure: motivation, overview with examples, practical advices Thanks!

@CJP3 4 месяца назад

Thanks for the feedback! I’ll do more of this style!

@mushinart Год назад

Outstanding explanation, professor

@CJP3 Год назад

Thank you so much!

@noobmasteroo69 10 дней назад

You're doing great, please avoid bg music while explaining. Thank you.

@CJP3 9 дней назад

Thanks for the feedback! Glad you enjoyed the video

@nicolaslpf Год назад

Amazing video! I was creating a function for measuring the same you forgot to name log1p Wich is log of (x+1) really useful for right skewed data with values less than 1

@pabloagogo1 3 месяца назад

This is interesting. If one corrects the original skewed data, via doing these kinds of transformations, in the context of linear regression or multiple linear regression, will that not change the interpretation of the original data. Curious to know.

@CJP3 3 месяца назад

Perhaps, but that change may be for the better. I’d say it’s worth considering these transformation if you know you have skewed data. Many models especially linear models assume normally distributed variables. I usually build models with and without significant preprocessing and feature scaling/engineering.

@dannybee9068 Год назад

Thank you! That was helpful! So we basically can make the root of any power? Is there a drawbag for exploiting it , like keep increasing the n value for feature to the power of 1/n?

@CJP3 Год назад

Hi Danny! Context definitely matters. For analytical chemistry 1/n scaling is usually ok. a few downsides are that it makes the models less sensitive to potential outliers. Also its not suitable for certain distributions. Lastly, because 1/n scaling is non-linear, it can make data interpretation more difficult.

@thoniasenna2330 6 месяцев назад

SUBSCRIBED! What should one do before? Or, what's the correct order? - treating outliers, impute missing values, correct symmetry? Thanks Dr. P!

@CJP3 6 месяцев назад

You’re not going to like the answer 😂… it depends a lot on the application. It’s first best to be aware they exist and then evaluate their impact on your outcome. For example if you’re trying to determine outlier samples - then outlier msmts wouldn’t be so bad.. maybe. Or missing values could be useful depending on the application so instead of imputing maybe you engineer a new feature.

@CJP3 6 месяцев назад

Don’t unsubscribe after my answer! 😂 🤣