Verifying AI 'Black Boxes' - Computerphile 

This video and the linked papers are about model interpretability/explanability and not (formal) model verification, may I suggest changing the title to "Interpreting/Explaining AI Black Boxes"?

I love this woman. I need her to explain all computer science to everyone.

The way the question was posed made it super interesting, and the explanation was enlightening! But I also loved the Dr's accent, she could explain anything and I'd listen.

Can it recognize a red panda with a cowboy hat on?

Video is great explaination for explaining black box models. I am just wondering is there any explainable methodology for non-image data/models because quite a lot of black box models I assume would be sequential data? because images you can extract features which explain what the black box is doing, but with other data types what can we do?

god, I love this channel.

Bro how a channel is 9 yrs old and even uploading consistently

It all depends on the training dataset. If you have trained your AI to classify Cowboy Hat as one class and children as another class, what happens when you pass a child wearing a cowboy hat through the NN? (let's assume the child and hat are about equal in size in the image and let's assume the training dataset contains equal numbers of images of children and cowboy hats. ) Such an image would clearly be a border case for both classes. A NN that labels the image of a child wearing a cowboy hat as a cowboy hat would be correct. If it labels the image as a child, that too would be correct.

As a visual systems neuroscientist, I'm afraid of the number of wheels that are being reinvented by folks who don't study their 'in vivo' or 'in silico' counterparts.

👍

As always another nice computerphile video; can you guys please make another video about open GPT?

People calling them self-driving cars when they can't is the problem. They kill themselves and others.

This video is great, easy to understand and inspiring. With only one thing I want to say. Maybe it is not a panda?

This was a fascinating topic but the accent was hard to follow. I could not understand most of what the presenter said and the disheveled appearance was distracting. Love the channel but this video was subpar.

This was great! That idea about throwing away parts of the input is so clever, love it

I also trust machines more than humans, I just don't trust the human that told the machine what to do :P

Excellent stuff here. But I'm disappointed you didn't put the cowboy hat on the panda.

👎 This won't work. It doesn't verify the AI at all. Just think of the examples/attacks with some almost invisible noise or a couple of pixels changed that to a human are barely noticeable but completely changed the networks result.

There's an important aspect missing from this video: if you just consider arbitrary ways of obscuring part of the image, you can easily end up with a particular pattern of what's obscured and what's not that influences the classification. In particular, the standard convolutional neural networks are all subject to _adversarial attacks_ where it may be sufficient to change only a handful of pixels and get a completely different classification. So if one really just tries to find the _minimal exposed area_ of an image that still gets the original classification, one invariably ends up with a bogus result that does nothing to explain the actual original classification. There are various ways to circumvent this issue, but it's unfortunately still more of an art than a clear science. The recursive approach in Chockler et al. 2021 definitely is a nice construction and seems to work well, but I'd like to see some better mathematical reasons for doing it this way.

What an absolutely brilliant video - clear, sharp and understandable. Truly great.

I@Computerphile Thanks for yet another interesting and enjoyable video. And if possible I would love to see more from Mrs. Cochler about their research, as I'm very curious if/how they "test the systems" for not only recognise "pixel parts" but also more abstract attributes and "features" and "feature sets"... To make crude example what would be the "minimal identifying features" of a "drawing of a red panda", how crucial is colouring. How does it differ depending on "style of rendering"... and on and on... Perception is a complex and "tricky" thing and seeing how we try to "imbue" systems with it is is fascinating. Best regards.

Amazing. I loved the enthusiasm and energy for the subject, and the accent is lovely to listen to. If only my university lecturers were this passionate when when teaching us, it would make things flow much easier.

Extremely grateful to have taken the Formal Verification course with Hana during my final year of my undergrad. Great video and fantastic to see her here on this channel 😊

Aw, she seems lovely. The sort of person that hugs children with a big smile. Great explanation too.

Great example of outside-in testing. Perhaps a network could be trained to supplement the output category with an explanation.

It's so ironic that RU-vid AI has misclassified cowboy hats multiple times while talking about misclassification...

Lol I don't trust self driving cars cause I have a bike and access to the Tesla record with bikes

very interesting topic. I wonder what an AI system would respond when beeing presented the red panda image with the face obscured. would it still find reasonable categories of what this might be?

I never heard of this kind of explanation on image recognition AI before! Such a simple and intuitive explanation! Thank you, miss! Very clever and well delivered!

"I trust technology more than people". I believe that some amount of distrust in technology, peopel and myself (third is especially hard to do) is the most resonable approach to the world.

I initially read the title as "Terrifying AI" and it seemed just as reasonable.

Nobody: A rapper in 2016: 12:22

I mean if I am ever in need of a black box where I have no idea what happens inside I just start writing my code in C++.

Verifying AI? more like Terrifying AI

The trust issue isn't with the AI in the individual car, the danger is with the control it gives to the Xi in the capital.

how well will it do with a kid wearing a T-shirt with a panda face on it and a cowboy hat with a starfish logo on the front? for a human it will be obvious..

It is funny to me that we call this a black box as in it is hidden or not understood. Yet the entire thing was built in a way to label and "understand " everything. Such irony. I am getting a serious book of Genesis vibe with the whole, "let's see what Adam calls everything."

Just like the silent kid in class is never that silent, AI must be hiding something 😅

I am preparing an evaluation benchmark for code generation language models to maybe become my bachelor thesis. This kind kf "interpretability" can be abstrced to just just the input but individual layers or even neurons. And this way you really find out where specific information is stored.

The comments are really hilarious. People are really a bunch of sheep that follow someone/everyone that don't ready know nothing about. Yes, may be an expert and working in the area, but where is the critical thinking that everyone should have? How can we trust a system that isn't alive, can’t be hurt and don't give a rats ass about the living things around it? Come on man... .

I hope they aren't directing the occultation themselves, "in quite an uninteresting pattern if i might add" but letting the AI figure it out itself, we might not but an AI could tell apart animals by their fangs.

I don't worry about AI as much as I worry about car manufacturers very lacking security competency... Taking over critical systems of a car through the Bluetooth of the car stereo is a thing that has happened... WHY THE HELL ARE THOSE SYSTEMS EVEN CONNECTED TO EACH OTHER?!

The follow-on question is to ask why AI sometimes confuses black people's faces with chimpanzees.

Machine learning systems should never be trusted in safety critical places. The risk is too high.

I trust humans to, as they always have, what is in their nature. I expect "AI" will be no different.

This really reminds me of Vsauce's video: do chairs exist?

It also depends on the basic problem you're trying to solve. Let's assume a NN trained to distinguish between bulldozers and red pandas. It will take a very small part of any image for the NN to properly classify an image. Now let's assume a NN trained to distinguish between red pandas and other animals of a similar size with 4 legs and tails. It will be much harder to distinguish between images. For an image to be correctly classified, more of the image must clearly distinguish the subject from other incorrect classes.

Isn't verification as misnomer here, you are not proving anything, just testing.

covering it with cardboard will not change whether today is a Monday ...

i have a question how random number generator first implimented in unix

The man in the restaurant knows he's a red panda because he eats, shoots, and leaves.

It would be interesting to see if there are other sets of the image that return panda but without the face. I suspect there's probably one with some small amount of pixels that just *happens* to return panda

Cross validation ?

I still don't trust it

Image diffusion modules are black box and they are “trained” on unregulated data. This doesn’t have creative intent or a choice like a human, doesn’t get “inspired” like us - it’s computer power with human algorithm and math. In many cases it causes severe data leakage as the devs can’t argue they try to intentionally rip off individuals or intellectual properties. Data shouldn’t be trained without consent, especially since the biggest problem in Machine learning currently is that it can’t forget. What we witness today is how exploitation can happen in ML, ethics exist for a reason - creating shouldn’t come on the expense of others without their consent. “For research purposes” only is one thing, for profits of a few corporations who truly owns it to leach on us all? No thank you. Ai advocates should really focus on ethics - because as much as they want to romanticize it, the only sentient that exist right now is us, humans. And we don’t treat it well at all, try to actually use it for important things like medical or environmental instead of motives for power and capitalism. “Progress” backwards

This is such a cool technique!

Testing fundamentally can demonstrate the presence, but not the absence, of problems even with systems we consciously design. How much less adequate is it when we are deliberately ignorant of how the system proposes to work! And yet ... it's better than nothing.

What if our idea of what makes a panda a panda is wrong and the AI uncovers the true specification? We'd throw it out and say it's silly, while, in fact, we are the silly ones. That logic should be applied to things where we're not that sure and that we didn't define ourselves.

thousands is an understatement, black boxes are math equations being held by computer functions, the math arranged itself, but math is magic, math arranged in the shape of consciousness.

my question is, how do you generate those minimal images? is it similar to a git bisect/binary search? like where you feed it iteratively reduced images untill it no longer recognises it? right?

panda

Are you going to make a video about how the Chat GPT work?

Cool explanation but honestly even if I could afford such a thing....never. my sense of self worth involves my ability to do things for myself and I LOVE DRIVING. so no thanks.

Good topic. Thanks!

02:20 She means by the term "complicated" in the sense of that it is utterly hopeless and impenetrable for any analysis and reduction from inside because it is inherently incomprehensible on that level?

Obscuring parts of the image doesn't rule out the possibility that the AI system is giving the right answer because it is Monday. For that you'd have to test the system on different days of the week, and prove that no correlation exists between the day of the week and the performance of the system.

Well... this reduces the probability of getting surprised by wrong results, but calling this a verficiation is very far-fetched. If you'd want to use this method to veryfy that your self-driving car correctly stops when the situation calls for it, you'd have to throw **every possible** "you need to stop the car immediately" situation at it. But that's simply not feasable.

A true AI should not depend on human given labels, it should learn on its own what a Panda is, given pictures of animals. It would infer that a subset of those share common properties and hence is a new class. It would then label it in its own way. Then we'd need to link those labels to our human words.

great explanation

@0:30 If every car was self driving I would feel much better than how I feel with the current situation. I trust the automated system to act appropriately. I don't trust the humans to always act in a rational way for the benefit of all. If you put the two together, you get a deadly AI version of prisoner's dilemma.

I trust the system but I don't want it in my car. I simply love driving too much and my freedom of choice too, regardlessly if an AI can or can not do the things better then me. Usually the people who don't drive or don't like driving are among biggest self driving car advocates.