Тёмный

Data augmentation to address overfitting | Deep Learning Tutorial 26 (Tensorflow, Keras & Python) 

codebasics
Подписаться 1,1 млн
Просмотров 135 тыс.
50% 1

When we don't have enough training samples to cover diverse cases in image classification, often CNN might overfit. To address this we use a technique called data augmentation in deep learning. Data augmentation is used to generate new training samples from current training set using various transformations such as scaling, rotation, contrast change etc. In this video, we will classify flower images and see how our cnn model overfits. After that we will use data augmentation to generate new training samples and see how model performance improves.
#dataaugmentation #dataaugmentationdeeplearning #addressoverfitting #cnn #deeplearningtutorial
Do you want to learn technology from me? Check codebasics.io/ for my affordable video courses.
Code: github.com/codebasics/deep-le...
Deep learning playlist: • Deep Learning With Ten...
Machine learning playlist : ru-vid.com?list...
Discord: / discord
Website: codebasics.io/
Facebook: / codebasicshub
Twitter: / codebasicshub
Linkedin: / codebasics
DISCLAIMER: All opinions expressed in this video are of my own and not that of my employers'.

Опубликовано:

 

31 окт 2020

Поделиться:

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист
Посмотреть позже
Комментарии : 140   
@codebasics
@codebasics 2 года назад
Check out our premium machine learning course with 2 Industry projects: codebasics.io/courses/machine-learning-for-data-science-beginners-to-advanced
@andrewzolensky9807
@andrewzolensky9807 2 года назад
You are hands down the best teacher I've found on youtube for deep learning and coding. I've spent hours and hours trying to figure this stuff out, and you just make it so simple and elegant. Thank you good sir.
@ayoolafakoya9841
@ayoolafakoya9841 3 года назад
You are a gifted teacher. I easily understand any topic you teach. Thanks
@luvieee321
@luvieee321 7 месяцев назад
Hi bro are you an ml engineer
@sandiproy330
@sandiproy330 Год назад
Your teaching makes things so simple. Thank you, sir.
@leamon9024
@leamon9024 3 года назад
Thanks sir, looking forward to more in-depth content in the upcoming videos.
@pabloreynoso2402
@pabloreynoso2402 2 года назад
Awesome content man!! I have liked all your tutorials and I started to follow you on youtube!!👌🏽
@acrux2061
@acrux2061 Год назад
nice video, I've struggling in data augmentation for a long time T_T. Now I found how to use cnn with data augmentation, I hope I can create my own deep learning model using this method. thanks bro
@vikashdas1852
@vikashdas1852 3 года назад
Video by video this tutorial is getting more awesome, excellent teaching and very calm explanation by our Guruji
@codebasics
@codebasics 3 года назад
Glad it was helpful!
@codinghighlightswithsadra7343
great job, I've found your channel recently it's helped me a lot, thank you so much🍀
@redigeremew1478
@redigeremew1478 2 года назад
YOU ARE BEST MY ONLINE TEACHER
@yccdavis
@yccdavis 2 года назад
It was just so helpful!!! Thank you so much!!!
@medinikb
@medinikb 2 года назад
Nice explanation!!!!
@arnolddumba7949
@arnolddumba7949 2 года назад
sir, you are the best ! thank you sir.
@pritishpattnaik4674
@pritishpattnaik4674 2 года назад
Great video Sir
@rb4754
@rb4754 10 месяцев назад
Very well explained sir...🤗
@jongcheulkim7284
@jongcheulkim7284 2 года назад
Thank you so much.
@asifhasan7444
@asifhasan7444 2 года назад
great fan of your teaching style
@codebasics
@codebasics 2 года назад
Glad to hear that
@pallavipriyadarshini5065
@pallavipriyadarshini5065 3 года назад
sir..i am a big fan of urs..ur codes are very simple and easily understandable..sir please do some videos on image segmentation in medical imaging.
@womenfitness_helper
@womenfitness_helper 2 года назад
Ok
@izharkhankhattak
@izharkhankhattak 3 года назад
A very nice lecture as usual.
@codebasics
@codebasics 3 года назад
👍😊
@aigen_erated
@aigen_erated 2 года назад
Thanks a lot. ❤️
@MarcusVinicius-cu4rd
@MarcusVinicius-cu4rd 3 года назад
Can someone explain me this: After the data aug, the model.evaluate function returns 070~0.75 accuracy, but when using classification_report(y_test, y_pred_classes), i get 0.51 as accuracy. Wasn't it supposed to be the same as model.evaluate()?
@sumit121285
@sumit121285 3 года назад
again useful....thanks dear...thankyou so much.....
@codebasics
@codebasics 3 года назад
Glad you liked it
@snehasneha9290
@snehasneha9290 3 года назад
Amazing content
@codebasics
@codebasics 3 года назад
👍😊
@daniaafrina9056
@daniaafrina9056 3 года назад
Thank u teacher
@siddharthct4799
@siddharthct4799 3 года назад
Thank you for this video
@codebasics
@codebasics 3 года назад
Glad it was helpful!
@sanooosai
@sanooosai 6 месяцев назад
thank you sir
@eyerusalemtsehaye5358
@eyerusalemtsehaye5358 3 года назад
I am getting an error while scaling the train and test set. Unable to allocate 35.7 GiB for an array with shape (3258, 700, 700, 3) and data type float64 How can I solve it?
@primeprime5396
@primeprime5396 Год назад
Thank you sir. I have one doubt, what % of the image dataset should be augmented to generate a good model?
@jrajpal5
@jrajpal5 Год назад
Hii sir, i have applied same configuration of model as mentioned but i am still getting .2442 accuracy. Have i made some mistake or i have missed something here
@NguyenNhan-yg4cb
@NguyenNhan-yg4cb 3 года назад
you should save this video for us, so I can learn it later lol :)) Bye thanks so much
@codebasics
@codebasics 3 года назад
yes it is saved. you can always refer it later
@wussboi
@wussboi 3 года назад
Hi sir, for the stack of Conv2D layers...are there any general rule-of-thumb for selecting the number of filters? Whether its recommended to go in an increasing order (like you did), decreasing order or constant?
@lindadelalifiasam5878
@lindadelalifiasam5878 2 года назад
i also get confuse sometimes
@user-re8wc1yl5s
@user-re8wc1yl5s Год назад
I was trying to use loss='SparseCategoricalCrossentropy' in the cnn model and i got the same accuracy after each epochs. Can someone explain me why it's happening??
@hardikvegad3508
@hardikvegad3508 3 года назад
Sir can you please explain RNN and LSTM just like you explained ANN....Please Sir :)
@shikhamishra7327
@shikhamishra7327 2 года назад
Hello Sir, the data_dir outputs me as Poxis Path instead of Windows Path. Any suggestion for this issue.
@sazeebulbashar5686
@sazeebulbashar5686 Год назад
incredible Video. Hey how can I get the Slides you used in this video?
@kevinvigi9791
@kevinvigi9791 2 года назад
Sir, why was a layers.input (input layer) given. Why did we start with the conv2D layer. If there is no input layer, how will the model know the size of input data?
@hanabi3622
@hanabi3622 2 года назад
where does those augmented data got saved?
@chaudharee
@chaudharee 2 года назад
what about using stratify = y for these multiclass label flower label? Can we do that? You didn't do that here
@RanjitSingh-rq1qx
@RanjitSingh-rq1qx Год назад
how can we get this url for another datasets
@summerxia7474
@summerxia7474 2 года назад
Hi, in 15:58, how do you open the window for the `train_test_split`? Thanks~
@devadathgs402
@devadathgs402 8 месяцев назад
u found it ?
@usamaashfaq4188
@usamaashfaq4188 3 года назад
Is it possible to get a certification for this course?
@siddharthsingh2369
@siddharthsingh2369 2 года назад
The part were image is getting zoomed in and out is not working for me and while plotting it is not showing and difference. Is anyone facing the same issue ? Sometimes it is working sometime it doesn't.
@markjasonong5668
@markjasonong5668 3 года назад
Good Day Sir, thank you for giving us awesome tutorials. I would like to ask why you chose 180x180 dimension to resize the images? Thank you in advance 😊
@RafiKhalil
@RafiKhalil 3 года назад
If you see the images in the dataset all of the images have different dimensions, to be able to train the model well we need to have the same dimension for all the images.
@kishore961
@kishore961 3 года назад
No reason to choose 180x180 dimension. you can choose any sensible dimension after exploring the sizes of different images in the dataset.
@ajaysaroha2539
@ajaysaroha2539 3 года назад
Sir,how syllabus will remain & in future when it will be done from your side?
@siddarthbali12
@siddarthbali12 3 года назад
Hi, you haven't specified input_shape while defining first convolution layer..., is it not necessary to define the input shape???
@sameer5671
@sameer5671 3 года назад
CNN detects it automatically.
@sanskar_choudhary
@sanskar_choudhary 2 года назад
What if I use plt.imread instead of cv2.imread ?
@umarfarooqmodelschoolshaha5909
@umarfarooqmodelschoolshaha5909 2 года назад
Hi sir thanks for such a good lecture but i have i have a problem when i run data augmentation code the code show error that NameError: name 'img_height' is not defined so i can do for that
@siddharthsingh2369
@siddharthsingh2369 2 года назад
u can provide 180 to both img_height and img_width, it will work
@aldairdejesushoyostorres4940
Hello there! Awesome tutorials so far. I’m having an error here 29:11 , it tells me the following: NotImplemented error, Cannot convert a symbolic Tensor to a numpy array. Can anyone help me, please. I get the previous error when I add the data_augmentation to the model and then compile. I will appreciate your help!
@antoniobelard7270
@antoniobelard7270 2 года назад
Congratulations on this and all the other videos on the series, they are very informative and instructive. I was trying to run the notebook on github but I'm getting an error when trying to create data_augmentation. Variables img_height and img_width are not defined and using (180, 180) in their place and running the cell just gives me the following error : "NotImplementedError: Cannot convert a symbolic Tensor (random_rotation_5/rotation_matrix/strided_slice:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported" I´m just running the notebook, no changes to it. Every other coding from previous videos work just fine, Can you please check if there's a problem with this one? Thank you so much!
@antoniobelard7270
@antoniobelard7270 2 года назад
Well, I got it running! I downgraded numpy to version 1.18.5 take a look at this article. exerror.com/notimplementederror-cannot-convert-a-symbolic-tensor-lstm_2-strided_slice0-to-a-numpy-array/ Thanks anyway, I´m sure you would have helped :) Amazing videos you make, Keep them coming:) Best regards
@charumathibsccs1811
@charumathibsccs1811 3 года назад
Hi sir, Can you explain the Python 3 in a single video . because the basic is so taught for me
@raghavendrac4710
@raghavendrac4710 2 месяца назад
use layers.RandomFlip(), layers.RandomRotation(), instead of using random.experimental.preprocessing . This fixes the issue of module not found
@hssp1534
@hssp1534 2 года назад
So all of this effort of improving accuracy is taken care by pre trained models right. We just need to use pre trained models in our model and with just a small number of layers and epochs we can achieve maximum accuracy, am i correct?
@Hamza-vk6sc
@Hamza-vk6sc 7 месяцев назад
What about the augmentation of tabular data ?
@manu1983manoj
@manu1983manoj 3 года назад
Hey Dhaval, can you tell me which version of TF and keras should I use? I am facing issues with importing preprocessing module.
@codebasics
@codebasics 3 года назад
You need to have tensorflow 2.3.0 for this to work. pip install tensorflow==2.3.0
@manu1983manoj
@manu1983manoj 3 года назад
@@codebasics Thanks :)
@MohamedAshraf-zs6nv
@MohamedAshraf-zs6nv 3 года назад
when I get a high score in training much more than testing we call it overfitting or what??
@codebasics
@codebasics 3 года назад
Yes it is called over fitting
@anirbandas4985
@anirbandas4985 2 года назад
Hi, during the first process of trainning the model, i used loss='sparse_categorical_crossentropy' as loss function and the accuracy was terrible (~20-30%) but when i used loss=tf.keras.losses.SparseCategoricalCrossentropy (as shown in video), the accuracy was so much better (~99%) can someone explain what is the difference between these two losses? (irrespective of having the same name)
@nadiiaduiunova
@nadiiaduiunova Год назад
it was explained shortly here ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-7HPwo4wnJeA.html or in previous videos of this course
@ousmanealamakaba3135
@ousmanealamakaba3135 2 года назад
Thank you so much sir,please i want you to show us how to deploy this project in django?
@varungupta7539
@varungupta7539 2 года назад
in this model how can I test an single image from my pc and predict the result??
@hssp1534
@hssp1534 2 года назад
you have to put the image through the pre processing step for image and then use it to test against y_pred
@zainulabideen_1
@zainulabideen_1 10 месяцев назад
😍😍😍😍😍😍😍😍
@manu-prakash-choudhary
@manu-prakash-choudhary 2 года назад
if you will be available in stock market then i will put all of my money on you cause I know you are going to rise exponentially on youtube in upcoming years 🙏
@codebasics
@codebasics 2 года назад
Ha ha thanks. Looks like I need issue an IPO in myself 🤓
@user-ek1qb2qo2u
@user-ek1qb2qo2u 3 года назад
Hello, i got a question. Which level of mathematics(do i have to know what do limits, integrals, derivatives and etc mean?) i have to know before watching this playlist?
@codebasics
@codebasics 3 года назад
you dont need any math knowledge at all. Most of the time i try to make clear math concepts and if I dont refer to mathisfun.com to clear the concepts yourself.
@chymoney1
@chymoney1 3 года назад
@@codebasics my teacher from highschool created that website
@husniahmed4955
@husniahmed4955 2 месяца назад
hi in this one error coming for me when I start to do data augmentation. That is " AttributeError: module 'keras._tf_keras.keras.layers' has no attribute 'experimental' " . please sir help me to solve this issue. There is no any issue with the libararies. All the libraries are upto date.
@gopalvarun03
@gopalvarun03 Месяц назад
The experimental and preprocessing modules are deprecated. You can use the following code: data_augmentation = keras.Sequential( [ layers.RandomFlip("horizontal"), layers.RandomRotation(0.1), layers.RandomZoom(0.1), ] ) To achieve the same
@saravanapandian8860
@saravanapandian8860 Год назад
I have faced some errors.....I don't fit x and y.....any one help to me???
@ariouathanane
@ariouathanane 3 года назад
i'm supprised that in the first layer, there is no the input shape? please could you clarify ? thank you
@usamaashfaq4188
@usamaashfaq4188 3 года назад
CNN does not require input shape to be specified as it figures it out itself i think
@ariouathanane
@ariouathanane 3 года назад
@@usamaashfaq4188 are you sure. How does cnn know the image dimensions etc.. i m not sure. But thank you for your replay
@nikhildesai2460
@nikhildesai2460 3 года назад
Hello to All, I am facing issue on data_augmentation part saying that img_height is not defined. Can Someone tell me where I am wrong? Thanks
@yogeshbharadwaj6200
@yogeshbharadwaj6200 3 года назад
I initialized like img_height = 180, img_width = 180, it worked, pls try. Tks.
@kishore961
@kishore961 3 года назад
Hey Nikhil, just replace the img_height and img_weight with 180. This should work
@rajasaad4913
@rajasaad4913 2 месяца назад
i have done argumentation but my accuracy have been decrease from 55% to 22% can anyone help
@imanefahim8557
@imanefahim8557 3 года назад
When I do data augmentation I get this error"Input 0 of layer lstm_155 is incompatible with the layer: expected ndim=3, found ndim=4. Full shape received: (None, 30, 126, 3)" how to fix it please I tried everything
@longvietng04
@longvietng04 3 года назад
I also encounter this error in the previous lesson
@imanefahim8557
@imanefahim8557 3 года назад
@@longvietng04 have u been able to solve it. It guess I fix it but don t know how to visualize the augmented data to see if it s actually working
@foysalahmmed9013
@foysalahmmed9013 3 года назад
Sir, I'm stuck in "import cv2". I have been tried many ways and many times. But in the end, it's not working. Can you please help me with that? Advanced thanks for your help.
@kishore961
@kishore961 3 года назад
What error are you seeing? have you tried installing using pip install opencv-python? If nothing else works, create a new notebook on google colab - colab.research.google.com and execute the code. Here, all the libraries come preinstalled so you can use imports right away.
@foysalahmmed9013
@foysalahmmed9013 3 года назад
@@kishore961 Thank you for your suggestion. I was using windows 10 pro N. From various sites I acknowledged that if you are using windows 10 pro N then it's obvious that you will face this problem. Now, I am using windows 10, and my problem is solved.
@MdAbdullah1509
@MdAbdullah1509 2 года назад
29:07 block 63, img_height, img_width is not defined. What would be the dimensions?
@siddharthsingh2369
@siddharthsingh2369 2 года назад
u can give 180 to both of them it will work
@nainapandey9732
@nainapandey9732 2 года назад
@@siddharthsingh2369 Hi I was facing same issue 180 did work like a charm but I wanted to know why 180 only cuz I tried other values giving shape error in model training. could you please let me know how you came up with 180?
@AdrienLegendre
@AdrienLegendre 11 месяцев назад
Seems it would be better to encode image using measures that are scale and translation invariant.
@kazifahimlateef8699
@kazifahimlateef8699 Год назад
Sir, in data_augmentation section, while running this cell, it says "img_height" is not defined!! How to get rid from this problem? Thnks for the great tutorial
@aryac845
@aryac845 Год назад
Same here Did u get any solution for this?
@nadiiaduiunova
@nadiiaduiunova Год назад
you can define them manually, like this: img_height = X_train_scaled[0].shape[1] img_width = X_train_scaled[0].shape[2]
@user-kk1li5mk7q
@user-kk1li5mk7q 7 месяцев назад
Shouldnt the img_width also be 180? @@nadiiaduiunova
@priyasunariwal2290
@priyasunariwal2290 4 месяца назад
hello I am facing some error tried all the solutions on stackoverflow and github even changed code to import tensorflow as tf from tensorflow.keras import layers data_augmentation = tf.keras.Sequential( [ tf.keras.layers.experimental.preprocessing.RandomFlip("horizontal", input_shape=(img_height, img_width, 3)), tf.keras.layers.experimental.preprocessing.RandomRotation(0.1), tf.keras.layers.experimental.preprocessing.RandomZoom(0.1), ] ) but erros is same, AttributeError: module 'keras._tf_keras.keras.layers' has no attribute 'experimental' Help i am stuck on this error
@asadpanhwar634
@asadpanhwar634 3 месяца назад
Ussss. U got any solution?
@asadpanhwar634
@asadpanhwar634 3 месяца назад
Tried many different things. Downgraded my tensorflow but still I couldnt solve, used other methods but they are very different
@rakeshgowda6187
@rakeshgowda6187 Год назад
please tell me what is the image height and width
@nadiiaduiunova
@nadiiaduiunova Год назад
img_height = X_train_scaled[0].shape[1] img_width = X_train_scaled[0].shape[2] or just img_height = 180 img_width = 180
@rakeshgowda6187
@rakeshgowda6187 Год назад
@@nadiiaduiunova thank you
@quosswimblik4489
@quosswimblik4489 3 года назад
You could use an adversarial network to fit the data better. have a Clown train a Identifier by competition. So the clown tries to fall the identifier with the kind of issues the identifier would face in the real world relative to it's task until the identifier is guessing with a good enough data fit in the competition. You can use both clown and identifier Adversarial networks and GANs to better fit a Generator also. Say you want to make a generator that generates real world c++ code. First you train the identifier with a clown then you train your code generator with the identifier.
@zaindurani8168
@zaindurani8168 3 года назад
sir can you cover NLP in this playlist
@codebasics
@codebasics 3 года назад
Yes that is coming up
@dontdemotivate6575
@dontdemotivate6575 3 года назад
Your DeepLearning play list is not opening. Some thing is very very wrong please check your play list.
@codebasics
@codebasics 3 года назад
The playlist is working perfectly good: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-Mubj_fqiAv8.html
@Mahesh9_
@Mahesh9_ 3 года назад
hello sir , nice video.... Anyways are you gonna make video on RNN ??
@codebasics
@codebasics 3 года назад
Yes I will
@work-dw2hl
@work-dw2hl 3 года назад
in data augmentation sir u consider 3 transformations it means for single images it generates 3 images or it applies 3 transformations on 1 image plz clarify my doubt thank you
@RafiKhalil
@RafiKhalil 3 года назад
Data augmentation applies different transformations to the same image.
@rishatdilmurat8913
@rishatdilmurat8913 5 месяцев назад
Why don't you use normalization instead of dividing it to 255?
@kiranmenon3044
@kiranmenon3044 3 года назад
Sir do you have plans to update Julia Playlists. Seems there has not been a single video since 2016. I understand there could be a lot to chew, but in your future plans I request you to consider it. Till then I would be learning from your other Playlists. Thanks
@codebasics
@codebasics 3 года назад
Ok I will keep this in mind
@kiranmenon3044
@kiranmenon3044 3 года назад
@@codebasics Thank you
@randomdude79404
@randomdude79404 3 года назад
Yes I agree please add more to Julia playlist it would be helpful
@abdullahmuhammadmoosa7170
@abdullahmuhammadmoosa7170 2 года назад
Thanks sir! Excellent tutorial
@ugorjirichard9282
@ugorjirichard9282 2 года назад
run all codes again
@umer_c0des330
@umer_c0des330 10 месяцев назад
I am noticing that no one is talking about the loss before augmentation and loss after augmentation and how to reduce it
@kunalrastogi6530
@kunalrastogi6530 Год назад
Anyone :- WARNING:tensorflow:Using a while_loop for converting RngReadAndSkip cause there is no registered converter for this op.
@abhijeetkaithwas6604
@abhijeetkaithwas6604 4 месяца назад
I having an error on the following code : --------------------------------------------------------------------------- error Traceback (most recent call last) Cell In[24], line 6 4 for image in images: 5 img = cv2.imread(str(image)) ----> 6 resized_img = cv2.resize(img,(180,180)) 7 X.append(resized_img) 8 y.append(flowers_labels_dict[flower_name]) error: OpenCV(4.9.0) D:\a\opencv-python\opencv-python\opencv\modules\imgproc\src esize.cpp:4152: error: (-215:Assertion failed) !ssize.empty() in function 'cv::resize' Please help me to resolve it.
Далее
127 - Data augmentation using keras
20:10
Просмотров 71 тыс.
How I'd Learn AI (If I Had to Start Over)
15:04
Просмотров 763 тыс.
25 Nooby Pandas Coding Mistakes You Should NEVER make.
11:30
PyTorch in 100 Seconds
2:43
Просмотров 883 тыс.