How to build custom Datasets for Images in Pytorch 

Your series of pytorch tutorials has been legit amazing. Keep up the good work!!

This video helped me understand how to make any kind of custom dataset! thank you

Finally! Was looking how to do this for a while now.. was wondering how to load text files from a directory, but now I can adopt the code and do it!

Have spent lots of time lerning DL courses and then get stuck into this very first step when goes to practice, dont know why it's hard to find relevant tutorials. I want a dataset that label is also picture, but the ImageFolder method is confusing and seems dont meet my need(Or should I say I dont know how). Thank you so much for solving my problem. Now I can finally get start.

Maybe the only exalmpe of how to ACTUALLY use the frameworks. It all starts with importing our own data to the model and how to do it. PyTorch and TensorFlow are quite easy to be used just by following the instructions anyway. Thank you so much!

GOD THANK YOU SO MUCH I've been looking for this for SO LONG you're a survior

I recommend using: train_size = int(X * len(dataset)) test_size = len(dataset) - train_size train_set, test_set = torch.utils.data.random_split(dataset, [train_size, test_size]) where X is a percentage as float e.g. 0.6 for 60% of the dataset to be used for training. Since I'm quite new to ML, PyTorch etc. it took me some time to figure out how to split the set (I used the small dataset of 10 images). I was getting an error regarding the "Sum of input lengths does not equal the length of the input dataset!". I couldn't find any use of in_channel, num_classes, batch_size etc in your code. Thanks for the tutorial! It really helped me a lot!

Thank you my friend, from China

This is the most clear to understand ever! Many thanks👍

Great video! One little doubt: This custom dataset is the training dataset or the test dataset?

Thanks so much for your guide. Finally I found how to make my own dataset

bro , you saved my ass again

how can I load if my dataset is having both images(x and y)?

Very clear and precise! Great tutorial.

Thank you enormously for the video, you are the best!!

simply awesome. Very well explained

Thank you for this video! It really helps!

can anyone tell me what do i need to do - if i don't have a CSV file but the images are classified as folder. - I have two types of dataset for a single code and i need to use both one is an image dataset and the otherone is short clips

Thank you !!!! Perfect Explanation!

Thank you so much! This was enormously helpful.

I have built an simple model in pytorch using training dataset and validation dataset but stuck how to check the test dataset and submit on kaggle. Please make a video on this topic.

This is such a good video. Very helpful. I have one question tho. Would this work with different types of datasets I want to create? Like not just for images.

what about hyperparameters? are they defined differently for Cats and Dogs dataset or were used the same hyperparameters to fit the pretrained googlenet?

Hi Aladdin, great video, if we don't want any corresponding labels for our image data and we just want to load the text captions associated with the image then what do we do ?

Thank you so much for wonderful video. I have a query . input image size 224x224x3 of the GoogleNet . Whats the image size of your custom dataset. is it 224x224x3?

Can you please do a video on custom dataset for object detection using the VOCDetection class, i can't find any tutorial on this anywhere.

Hi, How can i convert data into csv like you ? Thanks

Thank you for the video. What do we do when the image format is .tif?

great tutorial; learned a lot.

This video is so clear thanks a lot ! I just don't know why when do print(len(dataset)) it tells me that there is my number of data -1 for example if I create 400 data it print 399 :( and then it fail in my test_data :(

So I am creating a GAN for mario level creation. I decode each image that is 12x9 (Each image is one level in the game) to an array of integers! So I currently have a bunch of 2d int Arrays of size 12x9 with all the info about the level. How can I have access to this text file and reshape it so that I can train my GAN?

One question. When I apply the model it gives me this kind of error: FileNotFoundError: No such file: 'C:\path\img.JPG;Healthy', where "Healthy" is the y label and the rest is the image path. What could be going wrong? Thank you. It seems like it searches for both the filename and the ylabel in the image folder.

Hey Aladdin, wow, that was a great workaround to load the data I've got 2 doubt and 2 Question: Doubt: 1. Well imagine I dunno the size of the dataset (variable size) but I want it to be 80% train and 20% test, so while performing random_split, can we put [0.8,0.2] instead of [20k,5k]? Also if we randomly select some value, won't the model overfit? We need to then substitute it in a confusion matrix to check which split is the best. [That'll take shitload of time] 2. We never used/called __len__, why did we declare it anyway? Or is it being called under some sub-process? Questions: 1. Is there any general way like: train_loder=torch.utils.data.Dataset(path, train=True)...etc, like we do for other example datasets? Or let's say we use for loop to iterate over the path_directory and append the label as directoy_name and images under that directory becomes our feature list then use something like img_to_array to convert x_train to array 2. The above tutorial was performed using a csv as base...but 1st of all how did you convert your whole imageset to csv? (By looping over each image and appending it to a csv?) Anyways, most of the dataset don't have a csv format available online, so what do you suggest, do we convert the whole dataset to csv then proceed? For example, I collected huge data using web-scraping of different objects [chair, table, stands and stoves, etc] what do you suggest I do next! "A lot of effort in solving any machine learning problem goes in to preparing the data." -some-guy said this and I believe it now that data selection and preprocessing is 1 of the hardest part.

Thank u so much for this video. why isn't it possible to use multi txt files expect one csv file? can i get some advice?

Usually, we don't shuffle data for test dataset, Please correct me if I'm wrong

I need to learn the basic of pytorch or tensorflow are you know the way to learning these ?? thank you

Thanks a lot

Best ONE!

@Aladdin persson how can I load if both of my train and targets are images? please answer

Hi. How can i convert my folder of images to csv file same as yours?

GOAT

Am I really the only one getting this error: ``TypeError: object.__new__() takes exactly one argument (the type to instantiate)`` ??

👏👏👏👏👏

I think it is illegal to use pytorch's logo... You will get into trouble man. Don't do it.

Thankyou so much, I was stuck on this custom dataset problem for a while now. Really appreciate it.

This doesn't allow proper indexing of the dataset. Like (yx, ys, = loader[:1000]

How do I index through the dataset and check the images in the training set or test set I tried using train_set[0] but it does not work. Not sure if there is any other way to do it

Hey thank you for the video! But since the network is pretrained, does it has 2 output classes (Dog and cat) or 1000 (the ImageNet classes)? I would like to use the resnet50 to classify my own dataset (only texture images) and also only have 2 classes. If i got it right, i can pretrain = false and train the network on my images and classify those 2 classes in the end, right?

Loved the simplicity and clarity !

please do a tutorial of using tarfile image datasets and do it in google collab. am having trouble with both

Thank you so much! It's so helpful for my course project🥰

How we can apply train_test_split instead of pytorch random split?....Is this same for train_test_solit?

Can i write these codes in command prompt or do i need to download pycharm or pyscript? plz tell i am new in this field

Thank you so much for the video, dear. Can you please make a video on k-fold cross validation by comparing its two methods with each other: i) record-wise validation ii) subject wise validation.

this dataset can i use for text to image generator?

Great video! Thak you so much!

print('Thanks')

Appreciate

Hi, I have a question to ask, I want to retrieve image and label from loaded dataset to split dataset for k fold cross validation, then how it's possible. I have been trying for it but failed, here is the code: dataset = Custom_Dataset(csv_file = 'file.csv', root_dir='./data', transform = my_transform) train_set, test_set = torch.utils.data.random_split(dataset, [2400, 664]) x_train, y_train = train_set =this gives output => ValueError: too many values to unpack (expected 2) also this is info I got printing about train_set. print(type(train_set)) this gives output => print(train_set.shape) tihs gives => AttributeError: 'Subset' object has no attribute 'shape' I think I am confused, please help me out in this, thank you for amazing videos.

I adapted the code to my dataset and the accuracy on the training set is well calculated, however when it tries to check the accuracy on the testing set it goes out of the range of images (my training set has 900 images and the testing 100, and it tries to read images >100 on the testing set when checking it's accuracy). Does anyone know why this is happening? :/

Thanks vey much for the video and the channel. One thing please, Could you please show how you make the csv file that contain the image names and the target values. Is there any quick smart way to do it.??

Thank you for the video. I just have 1 doubt. How to change the y_label line in the code if i have words as my classes. Please help me with this. Thanks in advance.

Pleaseeee help me to solve this issue , What should we do if we have different test.csv and test image folder. Please make a end to end image classification project with different train and test csv file and with different images folder and how can we submit it in csv format , it really help me.

Could you pls share a link where we could download the data? The links on description don't have full data.

Hi, If I have images in .png format labels in .txt file then how can I read that file to train dataset

whats the easiest way to load all the names of your images to an excel spreadsheet? do you have to do it by hand?

What if instead of images, it was text that is in the first column of the csv file? How can this be adapted?

Hi, I have own dataset. I want to convert to csv. How can i convert data into csv without image path directory?

great video! I'm am having an issues where "at least one stride in the given array is negative". not sure where I went wrong, but I think its something to do with how the images are transformed into tensors?

Why is it called "resized"? Did you resize the pictures?

Thank you, it's a very helpful tutorial. But what if I have my images stored in such way: two subfolders: in 1st subfolder images of 1st class, in the 2nd subfolder images of 2nd class? Do I have to create a csv file and re-arrange my data or I can somehow manage to learn pytorch to read data in the way that I have?

this is amazing tutorial. thanks for making this 👍👍👍

What a great video! Thanks! it's helped me a lot! :)

Very helpful! Thank you so much!

Thanks Aladdin for your video series. I'm learning a lot. I looked at the GoogLeNet architecture. It takes as input 224x224 images. Did you have to resize your images to 224x224 to make them work on the GoogLeNet? Also, I noticed that that architecture outputs 1000 different classes. How did you check accuracy if you only have 0 for cat and 1 for dog? Those are different classes in the ImageNet class list. I resized the images to 250x250 and ran your code and got 0.00% accuracy on the GoogLeNet architecture. Got 96.5% / 80.8% accuracy on my own custom CNN architecture.

The Video helped a lot . Thanks :D

Thank you for tutorial! This was very helpful!

Awesome video!

This was perfect, thank you so so much :)

Wow, what a helpful tutorial, but, I have a little question: Can I make the dataset with more than 0 and 1 output? For example if I want to add another animals (fox) to this dataset can I have another number (2) to return?

Awsome

Thank you very much for great video. But Please try to keep the videos separate . Do not mix the previous tutorial's code with new code. It becomes quite confusing.

So thanks. I've been looking for this !

Hello bro im getting error while trying to load the dataset

I am getting a runtime error of stack expects each tensor to be equal size, but got [3, 123, 199] at entry 0 and [3, 404, 499] at entry 1 Do we have to do some resizing inside transforms to get past through this? Or has anybody found a better way?

Thank you so much for the tutorial, I just have one question. How can I create a csv file with file names and class labels depending on which folder the image belongs to?

great tutorial, i find it's easy to understand! Thanks!

Can you add the excel sheet in the github repository as in the full one for training with the image paths in it the kaggle dataset does not include that. Great video!

thanks sir its very helpful to the beginners

How do you create the csv file

I have a problem: FileNotFoundError: No such file: 'path\m3.jpg;1' m3.jpg is my dataset and 1 is the y_label thing. But why are those to made to one and then searched? Please help!

Can you please please tell me that how to create that csv file when you have 10k data ? I am stuck in a project where I have image dataset and have to use multi class classification, for that I require csv file (annotated table) for mapping images to their respective labels.

Sir, where can i find more examples to practice dataset create?I was stuck on this problem for a long time.And i want to learn more about it.Thanks you!

Thank you so much for the video. I'm working on Language identification. I have a root directory that has 14 different folders of languages. Every folder has 500 .wav files with the respective language spoken. Eg: /root ----/English ----------1.wav ----------2.wav . . . ----------500.wav -----/Spanish ----------1.wav ----------2.wav . . . ----------500.wav . . and so on for 14 languages I'm performing multiclass classification so I'll be having 14 labels here I'll be glad if you can share the best method to load this data

This really helps. Thanks. 多谢

You're amazing

How can we create custom dataset loader for an image caption generating problem?

friends were asking about how to make a .csv file to load the dataset. Here I have made a quick video about it: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-siTHs7Cc7c4.html PS: voice is low but one can understand by the following video.

Hello! Great video! I have one question. How can i get the predictions of an unlabeled test set? (All test images, of custom dataset, are in the same folder and there are no labels. I need to get predictions for kaggle competition). Thank you in advance! (Btw I wrote a script to organise training data in subfolders in order to use the ImageFolder function from pytorch, and during training I got better results, compared to loading the image from custom dataset class)