Anti-Learning (So Bad, it's Good) - Computerphile 

Doesn't actually explain anti-learning. Explains how they thought of anti-learning, sure, but the closest we got to an actual explanation was "and then I flip it and it works"

Reminds me of a joke from business school a professor once said. "If you find a financial advisor who consistently gets it wrong, hire him! Do exactly the opposite of what he says and you will get rich. The dangerous ones are the ones who are 50% right and 50% wrong"

Where's the part where he explains the method? It's like having a book made of an introduction only :/

You forgot to talk about anti-learning...

This is the plot to every Star Trek NG episode. "Mr. Crusher, reverse the shield polarity." "It's working!"

As many others have pointed out in the comments, this video is quite uninformative as to what "Anti-Learning" is actually supposed to be. So I've looked at a couple of the papers of Aickelin and Roadknight and even though those are not particularly clear on what they're doing either, it seems like things are being misrepresented here. So first of all, what they call "Anti-Learning" is not the reversal of the classifier's predictions, but the phenomenon that a classifier achieves less than chance performance on the test set (or sets, as they are cross-validating). They claim that instead of being a sign of overfitting to the training data, this situation arises because the structure of the population is such that many non-similar cases are summarized under a single label, such that the population ought to be misrepresented in the sample similar to a situation of XOR, where one of the four combinations (00, 01, 10, 11) is missing from the sample, which necessarily leads to an incorrect classification of new data points that show said combination. They substantiate their distinction of anti-learning from overfitting by showing how, for a learnable data set, a neural network's performance on the training AND test set increase with an increase in flexibility of the model (more hidden nodes), but only up to the point where the test performance starts decreasing again (as the model starts overfitting), while for a data set that results in anti-learning, the test performance stays below 50% throughout, despite an increase in training set performance. (And for a random data set the performance on the test set stays around 50%). (I don't really find this convincing. The classifier picks up on distinctions suggested by the sample that don't hold up in the general population - that's overfitting to me and this is just a particular case, where the sample is systematically unrepresentative.) The classifiers they tried were not only linear classifiers, they used Bayesian Networks, Logistic Regression, Naive Bayes, Classification Tree Approaches, MLP and SVM (though I don't understand why this list doesn't include KNN), all of which performed poorly trained on sets of 35, 45 or 55 features. As stage I and IV could be classified reliably, their analyses focus on distinguishing stage II and III - so it's a binary classification problem. This is relevant, because their trick of flipping the classification wouldn't work otherwise. Reversing the predictions only makes sense when the reverse is actually specified (0 instead of 1 and vice versa), which it wouldn't be were there four classes. Lastly, I don't see where the 78% accuracy he reports on come from, though. From their 2015 paper all I see is the accuracy they get when they use an ensemble of different classifiers (half of those are trained to perform well, while the other half is trained to perform terrible and gets reversed) and - and this is what he really should have mentioned if that's what he is talking about - they only get this higher accuracy in a subset of the cases, namely those where the respective ensemble agrees. So they get the highest accuracy (~90%) for those cases where all six classifiers give the same label, but that is also a very small subset of the sample (29 data points).

I expected an explanation of anti-learning, and all I got was “Just put the hay in the apple and eat the candle”.

The argument at the ending was so Bad, it was Good.

Ugh. Video should be called "Computer science professor rediscovers nonlinear statistics and gets disproportionately excited about it."

... that was such an odd video, even as someone who has a fairly good grasp on XOR and logic and such, I'm left pretty confused by the whole thing. He took the wrong answer and he just reversed it? what?

So I work in this field... here's the gist of anti-learning using a football analogy. That's 'American football' to our UK friends. Imagine that you have a friend, let's call him Alan, who grew up in New York and was a huge Bills fan. Your friend never played football in school, doesn't know anything about football strategy, and growing up he only watched games that the Bills were in. Your friend (slowly) notices that the Bills have a great running back, and that when they play against a team with a weak run defense, the Bills usually win. When they play against a team with a good run defense, they usually lose. Using this simple rule, he is able to correctly predict the outcome of Bills games 80% of the time. This is the modeling equivalent of getting a high accuracy score on the 'training' data set. Your friend Alan then moves to Massachusetts and starts watching Patriots games. He applies the same theory about running backs and run defenses of opposing teams to try and predict the outcome of Patriots games. Because the Patriots rely heavily on the passing game, your friend's predictions are certifiably terrible, and he is only able to predict the outcomes of Patriots games correctly 30% of the time. This is the modeling equivalent of getting a low accuracy score on the 'test' (out of sample) data set. Because of his incomplete picture of football dynamics, you notice that your friend has become very good at incorrectly predicting the winner of any football game in the NFL, because today the NFL is predominantly a passing league. In fact, for any given football game, your friend is wrong 80% of the time. Still, your friend Alan clings to his theory about running backs and run defense. Now, it's the Superbowl, and you call up your friend and ask him his opinion of who is going to win. He gives you his opinion, and you bet on the other team. That is the gist of anti-learning. In machine learning, anti-learning just means that the model is drawing incorrect conclusions from the data, for whatever reason (noisy and incomplete data, small sample size, over-fitting, etc.). By using anti-learning, you are trying to maximize the rate at which the model draws these wrong conclusions, so as to get the worst model possible. Then you simply invert the outcome, and voila!--you have your prediction.

My understanding is that in this unique case is is actually quite easy to make a function that plots the wrong answers. When you check data in this case, you are checking to see if it does not fit in the function.

Basically the machine was so bad at guessing that they had to start asking 'what don't you think it is?'

I wish he'd explained more what the solution actually entailed. And plus, I happen to understand what he means by separating categories by a line, but I don't think most viewers are really going to understand. They don't even really know how he was approaching the data. Strangely subpar content for this channel which is generally pretty high quality.

But he didnt explain it

Am I the oney one who shudders at the waste of paper? He's only drawing at every other sheet. And his examples are simple enough that you could fit many of them onto a single one.

I think I got the answer / what the new approach is but not what he was doing before. This stuff with the medical data was a bit too without context for me to understand what he anted to say there.

So kinda like how the choice between doors on numberphile where you reject the first door you pick and pick a different one.

the amount of paper this guy wastes...

This video is enough to get me very intrigued about how this solution works. I'm hoping another video is in the works that takes a deep dive into how anti-learning works and how it applies to understanding this data.

I was trying to understand how the XOR example applies to what he's saying, and I don't really get it. If you can draw a line that separates the goods and the bads, then you've solved it: anything above the line is good, below is bad. But you can't do that for XOR. So, he drew a line through the middle. This solution is "wrong" in that it's not a good solution, but not in that it's always wrong. If it was always wrong, you could do what he said. It seems like what he's done, based on that visual, is found a "solution" that is right 50% of the time and wrong 50% of the time. So, reversing it is the same and not reversing it. That's how I understood his example to apply to what he was saying. Is this not correct?

We need more info on this, maybe an example with more variables, or an explanation on why the machine gets the wrong answer with xor. Please do more Computerphile!

I might just be being dumb, but I don't really understand what he meant by reversing the wrong answer. If the wrong answer is a horizontal line, what's the reverse of that?

I would be curious how it "anti-learning" works precisely, and how well it generalizes when compared to adding quadratic interaction terms or using a simple three layer neural network.

If we use that graph as an analogous for the Xor problem, can we not make the graph 3D? As in the values that are the same are transformed into a 1 value on the z-axis, and values that have a null-result transform into a -1 z-axis result, and you can simply draw a "line" (Or plane I suppose) between the two values? I'm not really sure what I am talking about but I am just thinking from a practical standpoint.

If you swap one of the inputs on the XOR problem, it becomes easy to draw a line through them

This is simple. Data doesnt always fit in a particular model (categorization), so by process of elimination, you find out what model/relationship remains, which best fits your data. Anti-learning is esentially knowing you'll get an answer wrong, but using that answer as part of an overall strategy towards finding the solution.

To seperate those 2 you could use the geometrical distance to (0|0). (1|1) is ~1.41 units away from (0|0), so you could say everyone whose distance is between 0.8 and 1.2 away from (0|0) is healthy... This should work no matter how many dimensions you have

is 500 a good set for machine learning? i thought they needed a bit larger but i could be wrong

well i didnt understand anything, but i still watch it

Have you tried PCA and PLS? (though it is still tricky once you've made something a mixed integer problem. better with the raw data.)

I'm just wondering. Can a computer do something like "similar problem recognition"?

As far as I know, a neural net with one hidden layer can solve the xor problem, and you can build a deeper (or wider?) net to generalize it to many dimensions. Did they try neural nets?

How do they "reverse" it? The REVERSAL of XOR is XOR.

If you included a third dimension of time (for example, from the medical data mentioned in the video) you might be able to program a parabolic arc to include the functions you are looking to compute?

Would pushing the parameter space into a higher dimension have helped? Either rbf-svm or a sufficiently large neural net? Or does the existence of a 200-dimension XOR problem seriously mess things up?

wouldn't be easier to renumber the sets? whenever the best values are in the middle... number them like 1,3,4,2? And when the good values are in the extremes, number them 3,1,2,4? What am i missing in the problem?

"reverse it"? What? The inverse of the function is also unsuitable, this doesn't really explain anything.

have you tried to square the explanatory variables?

Isn't that what kernel functions would be used for in support vector machines, to separate the data in a higher dimensional space? When do you actually have linearly separable data in real world examples? Or am I missing something here..?

There's seven and a half minutes of video explaining a problem, leading into a 10-second explanation overview of how it was solved. It is still completely unclear to me what this video is trying to explain. Was there more to this interview that was edited out for some reason?

Ok, here's my understanding: Say xY and Xy mean healthy patients. (Capitalization represents value) Machine learning will tend to group similar x and similar y values together, so we'll get groups like these: (xy, xY), (Xy, XY) This grouping doesn't mean anything, but if we "reverse" it, the grouping becomes: (xy, XY), (Xy, xY) which is meaningful. Another way I understood is, if you got a function with binary output that has a 30% success rate, reversing the output will produce a 70% success rate.

XOR-looking data is one reason why I used neural nets instead of decision trees for my research problem. Decision trees and XOR data do not mix.

I guess what he meant to say by "reversing" is that they used machine learning, but for the wrong answers since they are easier to find. This way, with enough of them, you will eventually find the right answers.

100 - 45 = 78. Thanks professor Uwe Aickelin

best accent!!! he should make a audiobook !! :D

How would one apply anti-learning to a data set with more than two labels? In other words, in cases where the prediction is simply not "true" or "false." Lets say a data set has four labels (1,2,3,4), and you supply test data which yields the following prediction in ranked order: "2, 1, 3, 4" (2 being the closest match, 4 being the furthest match) the learning approach would yield "2" as the respective prediction. When applying the reverse method to this same prediction ranking (2, 1, 3, 4), would you conclude that "4" would be the respective anti-learning counterpart since it's the prediction deemed to be furthest away from the test data or is this claim flawed? From the binary anti-learning example we can assume that proximity doesn't necessarily mean similarity. Surely "1", "3", or "4" are closer to the actual test label than "2" given that the training data set represents XOR characteristics.

ah, we had some similar problem in our remote sensing class. basically, a computer can't categorize data such as we do. for example in pictures a computer can use different algorithms to sample pixels and put them into categories to say that x area resembles an apple. to do that, the computer looks at the RGB-values of pixels, number of pixels, their positions etc. but for complex problems you can't use such simple methods for making said categories. but it's easy to let a computer find out what doesn't work as a sampling method.

Nice, I'm happy I've found this channel. One thing that did surprise me was there is still dotmatrix printing paper in the world. ;)

I like the video, well done. Looking on some comments below: it is not easy to explain some concepts, which are naturally captured by a few algebraic equations, in words. The idea of anti-learning is clearly stated in Uwe's presentation. However, it interferes with attempts by listeners, perhaps subconscious, to make a simple analytical model of it. Even simplest algebraic equation in words is a mess, until you have a proper model in your mind. Personally, I would like to see addition yet another, 2-dimensional version of xor which I used in my explanations of anti-learning. Maybe it is time for another you-tube video. In more general terms, I see here a need for formal maths which we cannot escape. At this moment I appreciate legacy of Rene Descartes, who introduced co-ordinates to geometry, so we now can perform very complicated derivations on space and time, universe, etc. Before that, the geometry was really hard to deal with and reduced in its scope.

Haven't Support Vector Machines solved the issue of linearly separating such a data set?

Aren't back propagation neural networks pretty robust for an XOR/XNOR type classification problem?

So what did he actually do there?

You learned how to apply derivatives to xor?

(?) a transistor is stop or go, and a neuron can do that or sort into categories up to a full reflection. So if you set up a circuit to sort data like a set of partial mirrors, would that do this job? ...Inspiration from organic process that cycles absorb/exclude over and over.

This is awesome!

I've read of this being done in other experiments and to test predictions. Example: We do not know the right answer, or how to get there. But we do know it's not "random" data. So we generate a set of random data and random predictions. Now we try the scientists or the AI/learning code predictions against our random data. We know if anyone matches the random data, they are wrong. So those who do not, are possibly right. We still need to test the surviving theories and predictions against real data and real answers later, but it helps find a place to begin when we are almost completely clueless. :P

Please tell us more

I could see how SVM might fail in the example he gave because it tries to linearly separate the data, but wouldn't a simple decision tree be able to learn data grouped in that way? Of course I realize the 2D example is illustrative and that his actual dataset is higher dimensional.

Sir James Goldsmith would use this technique back in the 70's. He'd phone up a few stock brokers and ask them their opinion. if they all said, "sell, sell! get out!" He'd buy and if they all shouted, "buy, buy! It's going through the roof!" he'd sell.

What?

Yeah this video needs a second part

so what is it?

Couldn't regression analysis solve this problem? You would need to create additional composite variables of some form but it doesn't seem like a 2 year problem.

why didnt they use an artificial neural network? isn't solving this x-or problem without the the problem of the "curse of dimensionality" a classic task for a NN?

Very interesting indeed.

Interesting, sort of reminds me of finding the value of what the denominatiors cannot be in algebra.

Interesting, I'd like to see an application of this though, how can one just reverse a false answer to get the right answer instead of another false one?

not too sure what a "learning function" is but after about one fourth int the video my little inner programmer started screaming "fuzzy logic" so loud, i coudn't quite understand the rest of the problem.

This is exactly the stuff neural nets were made for to solve. I think even a support vector machine would do it. Or even simpler a decision tree.

He could used the blank side of the paper to do that simple graphic...or div by 4 a single side that works too...

"If it's wrong, just reverse it!. There can only be two possible options!"

If you add Rolled and Folded versions of the input date as input. Wouldn't standard machine learning deal with the XOR problem.

I can see its use in binary classification problems. Hard to see how it could u useful in a non-binary classification problem. Any thoughts ?

My understanding is that when you know you are wrong, the opposite is more likely to be true.

amazing!

Wouldn't it be more efficient to use pattern recognition on data sets? Spit the cases into A-B groups "healthy" and "not healthy", human categorize them, then have the machine look for commonalities. Then spit each group again, re-run the algorithms, split again, re-run, etc. That way you'd end up with a machine that can not only do the initial A-B grouping, but also potentially tease out new patterns that the humans didn't notice before. Sounds like they were trying to have the machine do all the work unaided.

Maybe I'm oversimplifying, but why not just NOT one of the inputs?

Great! So we should take abs(x-y) as input also.

The computer is trying to sort into a binary group: healthy or not. The little bar graphs had a y axis that represented "good health" so to speak, which is actually what they need to computer to learn to determine. Given that there is a desire for a "happy medium" or a "happy extreme" on some values, applying a binary decision will be correct on binary problems, and slightly more often than not, WRONG ON QUESTIONS WHERE 1 OUT OF THREE POSSIBILITIES IS CORRECT. Basically, since the computer is drawing a binary line by educated guess through a tertiary problem, it will slightly more often be wrong. So if you simply reverse it the RESULT (not the process), the odds of getting the right answer go up, and even benefit from an educated guess. So, +Computerphile , correct me if I'm wrong, but this is a bit like the Monty Hall Problem.

Pat, The NES Punk 20 years later :D

I've got to go with the commenters who feel that something critical was left out. Look at the XOR problem in a little more detail to see that (e.g. he's recreating the old proof that a single layer neural network is limited in what it can learn, i.e. given that it can only generate linear boundaries it can't learn XOR). Label the edges of a square A,B,C,D. To get XOR right, the right grouping would be 2 groups containing the opposite edges, i.e. (A,C), (B,D). But there are 2 other ways to group the edges corresponding to separating the edges with a linear boundary. So the linear boundary algorithm will learn one of the "wrong" groupings (i.e. either (A,B),(C,D) or (A,D),(B,C)). Given that, it wouldn't appear that it's obvious that the proper alternative is (A,C),(B,D) rather than the other of the "wrong" linear boundary groupings. So it's not clear at all from the video what "just reversing it" means. Perhaps there's some way to exhaust the wrong choices and pick an alternative grouping but that would seem to lead to a combinatorial explosion for reasonable amounts of data so that doesn't seem likely to be the answer.

He's so happy about all this it's funny.

that was definitely a lot of paper

Well, I've failed to anti-learn.

Wait, why wouldn't you just use something like k-nearest neighbor for this data? I can almost guarantee that k-nearest neighbor will do better than 45% on this kind of data. And the way that the med student seemed to describe it, I would think a decision tree would work quite well also. Are we just limited to SVMs for some reason?

He's so excited :D

It would be awesome if the video had subtitles

This video seems to spend too much time on the initial cardinality bit (sometimes high X is good, sometimes low X is good, sometimes medium/extreme X is good) which doesn't seem to connect to the eventual solution, and then just gets started on the explanation of why certain classes of problem are hard for machine learning (because clustering doesn't work for XOR) and then.... stops. It really needs more time devoted to explanation of what "it didn't work so we flipped it and now it works" means.

I did not understand anything. Was there a point?

Explanation: "I've learned a simple way of choosing an answer to a complicated problem involving 2 choices. I know that I'm usually wrong more often then right. So, I'll choose the opposite answer to one I would normally pick. Now I'm right more often than not."

So basically, it's like looking for your wallet. You know where to look to not find it, so you look in the places where it could be instead. When you've excluded all places where it isn't, the only remaining place must be where it is.

Im not a data scientist.. it sounded like it was harder for them to weave a line between all the 200 dimensions of datapoints when they were looking to group patients into 2 groups. I think the solution they went with was to instead just concentrate on trying to how produce the worst result.. and then just flip that. Its basically like saying, we want to flip a biased coin but produce heads. If we know the coin is biased to produce tails 80% of the time, then we all need to do is toss the coin, then flip it once = 80% chance of heads. The thing I dont understand is why is it easier to classify the data wrongly, i.e why is it easier to wrongly conclude a sick patient is healthy, or vice versa, when looking at those 200 data points (and get that wrong answer 80% of the time) than finding the right answer 80% of the time without the "flipping". Perhaps thats where I need to learn maths ;)

I'd like to see a HowToBasic version of this video.

lmao when he briefly explained it I laughed like you laugh at a bad joke, its so bad its good!

I didn’t get the resolution. "Reversed it"? You had an algorithm that was so bad that it was worse than random, so you decided to do the opposite of what it was telling you? If the algorithm was 45% right, you could only get to 55% with this method... not 78%... Wait a minute, are you using the same methodology for computing your stats? Anyways, after hearing this, I wouldn’t feel much secure if I was one of your patients ^^ .

Hmm, isn't "anti-learning" just another way of saying "learning from your mistakes"? I did this. It was completely wrong. So, in future, I'll aim to do the opposite of all my previous mistakes and, lo and behold, I'm achieving greater success than when I was getting it all wrong. All my former errors have "anti-learnt" me what I should actually be doing instead!

"It's so bad, it's bad, let's not do that"

Great story about how research works in real life. Thanks I appreciate it. I'm reading Rojas' "Neural Networks: A Systemic Introduction", and it discusses the XOR problem in the context on NN:n, McCullogh-Pitts units and Perceptrons, and conceptualizes with Minsky's research contributions et cetera. It's a good book, check it out people!

wait, you're a professor in data science/machine learning, and you look at this data for 2 years and never even think about checking whether it looks like it isn't linearly separable? wth?