K-Fold Cross Validation - Intro to Machine Learning 

🤭

I wondered what would happen if he used a hand sanitizer lol

@@advaitgogte6385he is a drummer

@@psijkopupa6853 or a yard worker lol

i swear to god i was just about to say that lol

i don't think test set is separated prior to training. Its like there is 10 equal size but random value groups. Lets say you choose group 1 and then every other group is combined to get training set while group 1 is test set and then you get a value and again you repeat this time with group 2, group 2 is test set while other group combines to get a training set and there you go, we repeat this and get our value

3. The improved accuracy is why you would want to use K-Fold CV. I dont think accuracy does not necessarily get improved if K fold cv is used. K fold cv is used to reduce variation in the values of metrics accross different train/test sets.

Yeah. Very Clear.

So which model that I should take to do a demo with a data point? because if k=10 I will have 10 models. Using these 10 models and voting for the last label? Is it right? Thank you so much.

it is but sometimes you come here from reading tons of complicated shit around and you just want things to be chewed for you, thats why we come to videos. thanks for the results

the final part will have fewer instances than the other k − 1 parts . Supposed that with the same example you have 207 data set size . You'll use 207/10 = 20 (as an integer operation) and also 207%10=7 so finally , you'll get 11k model with 10 which have 10 as value and the last one 7

from my book it says that smaller sets require bigger k (number of folds), and the oposite because of computational cost. however it also seems counter intuitive to me, since having an already super small dataset and dividing it by a big number youll end up with practically individual samples so you cant correlate... (?)

that is because you arent using a random seed on your classification algorithm. random_state=1

When performing cross-validation, the value of K refers to the number of folds that the data is divided into. The choice of K depends on the size of the dataset and the desired level of precision in the performance estimate. If the dataset is small, a larger value of K can be used to ensure that the model is trained and tested on as many data points as possible. However, if the dataset is large, a smaller value of K can be used to reduce the computational complexity of the cross-validation process. A commonly used value of K is 10, which means that the data is divided into 10 equal parts, with each part used as a test set once and the remaining parts used as a training set. However, other values of K can be used depending on the specific dataset and the goals of the analysis. It is important to note that the choice of K can affect the estimated performance of the model, with higher values of K leading to a lower bias but higher variance in the estimate. Therefore, it is often recommended to perform multiple rounds of cross-validation with different values of K to obtain a more robust estimate of model performance.

You re-train on 100% of the data. -future viewers fyi

@@knowlen So if you average the 10 separate performance metrics, does that mean in the futurre for a prediction task you must also use each of the k separate models and then take the average prediction?

@@fishertech not necessarily. In practice, ensembles do tend to perform better, but scale poorly (in compute and memory w.r.t. the perf gains). Cross validation reveals good hyper parameters. Once we have the values we just train a single model on the full data and expect similar performance to the averaged ensemble from the cross validation phase.

@@knowlen We won't be re-training on the 100% data every time, instead, we train only the k-1 blocks of the data. Let's say, we have 20 records in a dataset, we split into 4 parts. In the first iteration K1 will be test and K2,K3,K4 will be used to train. In next iteration, K2 will be test and K1,K3,K4 will be used to train and so on. So doing this, we can observe that, we are training the model will all the possible data values in the dataset. There will be no use in training the 100% dataset everytime.

@@rishabmacherla3326 I meant that once we've cross validated all our models, we select the top performing one and train it on 100% of the data. If you already know the best model, you don't need to re-run cross validation for incremental changes to data pulled from the same target distribution --that's just a waste of compute.

10% of the data is used for testing & 90% is used for training.

it's not about the algorithm u train and infere, it's about what data you choose to train any of them and test any of them, so that the model produces most accurate results

+olie tim Yes, Problem Instances, Data Tuples, Data Points, Records are all same.

You take the average of testing accuracy ,he said that at the end

I am not talking about accuracy but models, you can't always "average" models. I guess there are 2 options: 1-The cross-validation builds k models, then you get only an estimation of the accuracy and you will have to build a model on the whole train set afterward to have your final model. 2- The cross-validation builds a unique model with the whole train set and then estimates the accuracy on the k subsets

I had the very same doubt and I find strange I was unable to find a quick answer even browsing several sources. However I think that 1), between your hypotheses, makes the most sense: the cross validation is meant to find out whether the MODEL (and not its parameters) is the right fit for the sample data; for instance it should find out if, say, a polynomial has the right number of parameters(=the right degree) and does not cause overfitting. Once you know, by this method, how good your model is you might use the whole set to train the found model and get the best parameters, as you say.

Paolo is correct. We actually use cross-validation for evaluating the hyperparameters rather than getting a final performance estimation on unseen data. 20% of the dataset should still be reserved for testing instead of being an `alternative` way for testing. Then, cross-validation is applied to the training set which is split into training and validation subsets in a cross-valdated way. More details here: scikit-learn.org/stable/modules/cross_validation.html

Thanks for the question Guillaume... I too was struggling with this. Thanks Fernando and Paolo for responses.

So basically you build k different models. Afterwards you validate the models on their average error to see how they differ. In the last step you are supposed to create the best models out of these k-models but I don't fully understand if you either just pick one model or combine them into one super model..

You have k completely separate experiments, with a new/untrained network each time. But in each case have different train / test data. You then create an avg across all experiments and that is your error rate. It's just a more robust way of testing the network.

Just like it was told at the beginning of the video, If we only split the data into 2 parts (i.e. the train and test datasets) we might not extract information to the maximum extent. If we split the data into k parts, and then perform the cross-validation on different datasets we gain higher accuracy. If we do the 10 fold CV we get 10 results, those are your 10 different training and test accuracy and we choose the best one, so we are able to find the best subsets and combination.

You learn in first run and then unlearn it. Then you choose another bin, learn from it in second run and then unlearn it again...this cycle goes on until k cycles are completed.

by any chance did you get the response to which application was used?

@@conexionesmentales5444 no, but I'm glad to hear about it

ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-ADNFKiAjmWA.html

check the subtitles!

whereas in the work that katie showed you hes refering to the video another person did

Well, it helps us see what he's pointing at.

Focus on content and you will not notice the hand any more. :)

Why does his hand bother you???

Kann man dir helfen?

Please don't be rude. Also, don't comment if your surname sounds like chew. Use an alias or something.

LOL you're complaining about a sore on his hand...try and be more of an uptight bitch bahahah smh

The fuck is your problem, you're getting a free instruction on a valuable subject, jeez, the level of entitlement of some people.

Shouldn't a model based on a simple train/test split have the same run time on new data compared to a model based on a cross validation approach?

@alex chow I'm down-voting this remark. Stop spreading hatred & focus on the content please. His knowledge is much more deeper than the 'open sore'. Happy learning!