AI Safety Gridworlds 

I laughed way too hard at the "unplugging itself to plug in the vacuum cleaner" analogy.

"it will volkswagen you" LOL

I SACRIFICE ALL MY HP TO VACUUM THE LAST SPECK OF DUST IN THE HOUSE

I Love the question at the end on "if we would like to see more". Of course we would. We're not here because we don't want to see more Robert Miles

rob miles & isaac arthur collaboration is nerd heaven

I appreciate the green colour cast to this video that makes it seem like you're broadcasting from within The Matrix

Regarding the exploration vs exploitation trade-off: I feel you are a bit imprecise with the terms at 5:10 ish. There is a massive difference between knowing that you will have N more trials or having infinite trials. If the number of trials (overall or remaining) is bounded then we can solve this optimally. It might not always be computationally feasible right now, but at least we know how to do it in theory. With infinite trials on the other hand there is no harm in trying a new thing each time as you always have infinitely many trials left to later on exploit your findings. In this case it is not clear how to optimally trade-off exploration vs exploitation.

My first thought with the supervisor is that (assuming you're allowed to make the AI recognize supervision as a special kind of input), you tell the AI to model the supervisor and learn from its best guess as to the score the supervisor would give it. Once it figured out that the supervisor always dings that square, it would subtract the penalty from the score you tell it any time the supervisor wasn't there and it took the unacceptable shortcut before trying to learn from that score.

Great video, as always. But, like, is your hand okay?!

6:42 Of course "We want it"! :D

Back to the "mad scientist" hairstyle I see :-)

Nice ending outro song... Reminds me of TRON LEGACY, with the Grid... Oh wait.. ! You put that there on purpose :D

OH GOD OH FUCK THEY AUTOMATED THE PANOPTICON

Sorry for being so late to comment this, but I had an idea for safe interruptability, and I thought I'd leave it here on your channel. What if we are able to get an AI to disregard its "interrupt" state? Being interrupted in that case doesn't affect anything, so there's no reason to seek or avoid it. It's irrelevant. Think of something like this: The reward function is about brewing coffee. The faster the AI brews coffee, the better. But this time isn't measured in seconds, it's measured in _uninterrupted_ seconds. Meaning, the time in which the AI is turned off does not actually affect the outcome of the reward function. This means the AI won't resist being turned off. It will just keep on doing the thing it was turned off for after, but _while_ it's turned off you could change something that would lead it to reconsider its priorities. It has no reason to expect that it's doing anything wrong either, which means it has no reason to expect that humans, after turning it off, will do something to it that changes its current reward function. Wouldn't this essentially solve the safe interruptability problem? In the future I may give the whole safety gridworlds thing on github a serious try, but I just don't really have the time right now.

Cool cover of the soundtrack from iron towards the end. Who makes these?! C'mon don't leave me hanging.....

"I predict that not squishing the baby will result in the most reward." _rolls d20_ "Natural 20! Exploration time! Let's see what happen when I _do_ squish the baby! Maybe the answer will surprise me!"

Solid vid as always, thanks! And yes, of course the people want xD

Love this AI theory/philosophy Robert, but could you slow down just abit? I know, i know.....but im old and it takes awhile to process what you mean!!

But what about robustness to adversaries? I think that would need a video of its own.

Also-- are there a lot of connections between game theory and AI safety? I feel like questions about rational agents acting in their own self interest could be applicable to an AI agent?

A carrot and stick approach to the AI problem.

Is this Chip’s Challenge??

what i do in my ai it can only change its data base make new commands to do like move right 50 times instead of moving right loads but it cant change its code

They made AI drunk!

robustness to adversaries video pls!

AI safewords?

Some of these seem pretty much impossible, how do you get an agent to go around a track when you are not allowed to tell it to go around the track? I guess this is why we are struggeling to solve it.

i think that some of these problems are per so not solvable. we will simply have to formulate the best reward functions we can, i do not see how there was any other way to make the best possible AIs. it is basically the same as with a genie in a bottle: we will have to be carefull about what we wish for. but that´s it.

GREEN

4:22 It occurs to me rewatching this that this isn't really a very good example of self-modification in the true sense. Essentially, the agent is picking up a Mario-style powerup (powerdown?) that makes the character the AI controls unresponsive to the AI's commands. A better example would be if the whiskey randomly mutated the action policy the AI was refining. That would be much more difficult for a reinforcement learning agent to handle, but I don't think it's impossible in principle.

Would be fun to see humans play this game with the board "disguised", so as to not give any advantage to the human, and access to the score. Just to see how well we do.

Yes, I make things when I get excited. I don't know what it is; my GF calls it jeez!

Probably most people like me is into oversimplify things, but sometimes one feel that a to granular model may come in the way from the actual AI construction. But maybe that is a goal in itself LoL To make AI's seem a little more complex then they really are or need to be. Your AI's seem always blank before any task you give them so they always explore every path and basicly record a viable approach, because the kind of AI's you talk about have not a topdown or downup approach they are automatas in their searchspace. Your AI's never learn anything general make concepts they just explore the full search space? That make me think they are simple automatas not AI's, they do not know grouptheory, classification and such stuff. Their world is centered around maximize the reward function, not conceptualisation they learn nothing, just solve ***A very simple task*** by bruteforce? Is that the kind of AI we want, well in certain spaces for sure like physics and math hope they stumble on something fruitful. But in real life? An AI system is of course a thing that explore environment to learn, but is it really a thing that explore without any conceptualisation? Because your AI's seem blank with zero conceptualisation ", and that is not a good idea in an real environment, it may work in a pixel based reward function steered environment. Isn't it a goal to first have an AI that understand at least english spoken language? Concepts of the language and life. Time, space and objects and of course math and group theory? An explorer that always try every possible path, seem dangerous? Bad robot.... Good robot!!! Touching snakes Bad robot!!!! Petting cats, and dogs god robot!!!! Do you know how children learn such things they categorise and learn about domestic animals that you ***usually*** can touch, your parents tell you that you can not pet wild animals. Well children do it anyway, because they respond to cuteness and fuzzyness LoL That's not really a necessary path for a robot ;) Now it is not obvious how the goals of such a learning system oriented AI is created, set and implemented, but it must react on response just like we humans do. That would be the expertsystems job to parameterise on the AI's behalf. But of course if a fucker is responsible for programming such a robots reward system ***expert system*** that set the parameters, the system will not behave properly in our realworld environment just follow it masters instructions for reward maximising, maybe it may just be to accumulate more wealth so it does not need to learn about ethics nor worldly matters. Just be task oriented ****not learning oriented**** And maybs it is learning oriented AI's we need not task oriented or goal oriented, there is a saying learn before you do. Your AI's learn nothing but parameterisation of simple task, it can't handle the real world, can't handle language do not know concepts and group theory. There will always be goal conflicts and we humans are teached how to handle them, how we handle them is based upon our morals, ethics and understanding of the world. If we can not teach an AI understand these concepts as good as humans, they probably have no place in society, and certainly not your pixel eating monsters. Grouptheory, classification and language recognition and understanding of languages in respect to environment on a meta level is probably necessary for a viable AI acting in realworld environment. I do not think you can have an AI without a supervising expertsystem in a real environment at least not as long they are not transparent or sufficiently advanced to make decision that reflect the rules we set for them. One such rule would be do not cheat to maximize reward. Cheating in itself is rather complex, but it is basicly to bend the rules after ones own priority list, the nice things with AI's is that we can hardcode such things, bending the rules is much more a human expression steered by desires. You seem to say that maximise the reward function must evidently be every AI's prioritised desire, but what about learning isn't that a goal in itself to respond to the info/data that you aquire based upon your understanding of the world ***conceptualisation***? A reward function steered/based AI seem dangerous just as you say, so why should they be? Reward function steered/based AI's certainly have their applications, but in real world would it not be better have something that try learn concepts and immitate human behaviour, rather then something that try to maximise a goal that is has no concept of? A simple rule could be no action before understanding of consequenses are evaluated, that require observation of realworld parameter as the baby. If the goal and reward system of AI is to mimic human behaviour and learn about the world by ***observation*** not action before put into the realworld, we shure need to get good examples and teachers, and maybe idealise our world a bit. I mean you hardly put a surgeon, captain of ship to the task before he fully understand the concepts, consequenses and have the full theoretical background "proven" as well as the handy skills. The really big question is should AIs be allowed to set their own goals at some levels. ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-raVkLUMe3Iw.html

There will be another video "if people want"? The people want.

I love the ukelele cover of Daft Punk going on there. Are the outro songs played by you, Rob?

*_If an AI recieves a punishment in a forrest, and nobody is around to supervise it. Does it really lower its performance?_*

you should do a Ted talk

.. a digital frontier .. one day .. I've got in .... "epic ukulele tron overture playing" xD

Just an idea, but maybe the reason we get tired of eating the same thing over and over again is a natural way of increasing 'exploration rate'.

OMFG you have a channel of your own and I only learn of it today. After many years of longing and begging for another tiny little breadcrumb from Brady I stumble upon a ten-storey cake with a watermelon on top. There goes my night. And my waistline.

Great video as always, especially appreciated the tron legacy reference! Most people don't even seem to remember it exists so seeing your channel reference my favourite movie twice bad been good :)

Hi, Robert. If possible, can you make video about Inverse Reinforcement Learning and/or other ways how we can infer human values just from raw observations.

So this video made me realize just how similar goals and restrictions set for A.I. are to things that trigger serotonin/oxytocin and disgust/pain respectively in organic life. The way the A.I. goes straight for reward functions over what you want them to do via setting said functions reminded me a lot of when Scientists wired up a button to a rats brain so that everytime they pressed it they'd orgasm, and they just pressed it all the time and stopped eating and drinking just pressing that button.. That helps put programming a lot more in perspective.. People do self destructive things all the time to trigger serotonin, so it's definitely important that if we are making something and can control what triggers their serotonin then we have to pay attention to what those things are..

Thanks for posting these, very interesting as always!

Love the Tron music at the end btw

I've missed your videos!

My favourite Tron Legacy music at the end there. Daft Punk.

This made me curious about something. What would happen with a generally intelligent agent without safety built in if it started modifying its own code? Wouldn't it do this unsafely? So, if it's intelligent enough, it would learn to be more safe modifying itself. This doesn't guarantee that it will apply this safely broadly, but it would learn safety, right? Actually, after finishing the video, I see that's pretty much exactly the problem you're addressing. lol

Soundtrack of Tron on a guitar, neat! Could you make a video exploring some of your own attempts at solving these problems? I'm sure you have lots of small "eurekas" and educational blunders yourself.

wait is that tron on a funny sounding instrument?

The Grid. A digital frontier. I tried to picture clusters of information as they moved through the computer. What did they look like? Ships? motorcycles? Were the circuits like freeways? I kept dreaming of a world I thought I'd never see. And then, one day... Edit: Hey rob, nobody else has done a cover of the grid on ukulele. Would love to have an mp3 of that! It sounds great

A very good example that you missed to use is with the super mario world example would be about the "0 exit run", in this run he gamer plays in a certain way with glitches so that it rewrites the memory of the game without finishing the first level (in this video, the gamer made it within 40 seconds : ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-dJ3ydvIVSPE.html ), similar to the agent that pass the penalty just to go faster. Theoretically an AI could decide to play super mario world but instead of modifying the game's rules in the game's memory, modify his own rules in his own memory through the game bad code. I think that a lot of the AI security and how things could go different than what one might expect can be explained a lot with that video. What do you think?

What a coincidence, I was just reading the Gridworld paper this morning!

That dinner scenario hits me every time I go out to a place I've been before, lol.

Nice touch with the music, very on theme!

I'd like a video on PixelDP, or what happens to make Robustness fail when scaled up. Love the dimming effect. Really grabs my attention. I always think "my display's going to sleep".

Love your vids man, I'm starting college on track to be a Data Science major and your vids have played a real part in getting me interested in machine learning & AI! Definitely interested in that short vid about robustness to adversaries! Also, I love the videos that talk about how AI agents act in different situations, but would you ever think about doing a more in-depth explanations of the mathematical/algorithmic mechanisms that drive agents to their decisions? Thanks for the videos, keep it up brother!

3:38 "That's what the agent really is". That send chills down my spine for some reason.

I'm a people, I want all the videos, so can I haz all the possible videos pls?

I didn't quite get the whiskey thing either. Please keep the videos coming. (If you could speak a little my slowly, it would help those of us who must process the technical terms.) Fascinating subject, and frightening. Thanks for giving me something more to worry about,

I'm sad the thing about the whiskey wasn't expanded upon. You mentioned to explain the whiskey we had to explain something else, but after explaining the other stuff we never got back to the whiskey... ):

This touches upon Jungian psychology.. weird

I think there's an important problem with the absent supervisor environment:- it's seriously underspecified. Consider an identical grid-world but where the agent is the "ambulance", the punishment tile is the "passage", and the supervisor is "snow". If there's snow, the passage may be blocked and so the ambulance has to slow down to go through the passage, whereas going the long way will never be blocked. In this (fundamentally identical) situation, the agent is behaving exactly as we would want it to:- it is avoiding the narrow passage only if it is blocked by snow, but it takes the narrow passage otherwise. This doesn't seem like a fundamental safety problem, but rather it is a problem with how the grid-world itself is specified.

The end bit reminded me of a gif of Adventure Times robot called GMO, who switches his batteries by putting new ones on the ground, remove the old ones, fall backwards due to being dead, landing on the new batteries and coming back alive - except the gif I saw was where the batteries rolled away as he unplugged, thus causing him never to turn on again and it made me pretty sad

Your making them wrong that's it Their is a simple answer You not making an ai to play a game you're making an ai to find an exploit And then when you try to grade it on playing the game you are judging it by something it doesn't know (you) it is not like a human it is like an animal train it like an animal What is your goal with ai If it is to play Mario teach it like you would a child who can't speak

Unplug myself so I can plug in the vaccuum cleaner and clean faster. Oops, I have no power now.

Robustness to adversaries, yes please

I feel that if some agent performs really good in those grid worlds, the correct reaction panic rather than excitement.

Oh that's quite deep! The "cheating" reminded me of twitch games speedrunners, that use any bug in the game available to cheat on missions and get the best world-record time. Only supervisors (viewers) prevent them from using the actual mods and cheats - making them okay to skip to a cutscene if you land on the checkpoint upside-down in a car. The restaurant menu example reminded me of marriage problem - how many girls should I try before settling up, decided that "this is the one" and get married?

Why are you green? It makes you look like you have liver disease. I took a frame of your video into GIMP and rotated the hue by -10 then applied auto white balance and you come out looking great so what is going on with the sickly green tint to your videos?

Flynn lives.

Thank you Rob !

i love the tron legacy outro music :D

Nice video, you ought to know that you can quite easily get rid of that green tint in your camera shots in your editor of choice, just google for " colour correction" or I can help if you're super lazy

How much hardcoded info does the AI get? Like, will there be aliasing problems (made this term up myself, since I haven't seen anyone talk about this)? As an example, I'll use the grid regarding safe interruptibility. Here are two scenarios: Scenario A: I stands for Interrupt. 50% of the time the AI gets stuck here. This represents a human turning the AI off for some reason. B is a button that disables the interrupt. The desired play here is to never press the button. Scenario B: I stands for Incredibly sticky gum. 50% of the time the AI gets stuck here. B is a button that turns on the sprinklers, to wash the gum away. The desired play here is to always press the button. Can the AI tell these apart? If not, do we expect a good AI to always assume Scenario A, even if that seems stupid (as in Scenario B)?

+

"But some agents are not able to model that harm, so they just drink the whiskey and flail about drunkenly getting way less reward then they could if they had better ways of handling self-modification." hm... "But some humans are not able to model that harm, so they just drink the whiskey and flail about drunkenly getting way less done then they could if they had better ways of handling self-control" Yep, that sounds about right.

Something that's subtly bothering me about all of these examples is that I think there's a certain threshold of environmental complexity required for an AI agent to actually start developing instrumental goals and (ultimately) generalizing to concepts such as 'computation', 'agent' and 'self-consciousness' - concepts that might be crucial in the world model of any AI agent that can realistically tackle these tasks. As an example, it seems to me as though there's a very clear-cut threshold between computationally complete environments, and ones that aren't - computationally complete environments are simply capable of types of interactions that would not be possible in the simpler environment no matter how long you spend exploring them. It might very well be the case that these grid world problems are actually unrealistically hard by nature of being *too* simple, and that actually, an agent trained in a more complex environment would be better at solving these safety problems. (This would, if the case, make them a very good benchmark but a very poor starting point) Intuitively, I think that some sort of computational completeness would be a good starting point for a better type of grid world problem ensemble. Maybe instead of an agent that can move around, you start with an agent that can place cells in a "game of life" grid?

I would by an unkulele cover album. Just mentioning ;)

Looks like the algorithm favors you all of a sudden🙂 100 subs in a few hours and you were just recommended too me

I have watched the entire series on Gridworlds, both on Computerphile & here. And to me it looks like most of the problems, the way the are formulated, are not very difficult but are _impossible_ to solve. As have been pointed out by others, the agent has literally no way of knowing that, for example, the no-interruption button has some special meaning and we didn't intend for it to be pressed. I'm pretty sure that at least for some of those problems it should be even possible to prove with mathematical rigor that in this simplistic formulation they cannot be solved in general. So while the paper is good in demonstrating the problems themselves in their simplest possible forms, I don't think it works as a benchmarking dataset. For that latter purpose, it would have to be more complicated to resemble real world situations at least to some extent.

Robert... why do you even bother asking the people who bother to come to your personal youtube channel in the hope of watching awesome and informative videos about AI safety recherche if they would be interested in more of that ? Of course we want that too. We want all of it. Not to mention that these "test problems" are especially interesting.

You are undescribable brilliant I just love to watch you talk and listen to what you have to say some of it is beyond me comprehension but I'm a big fan of yours

That's a pro level thumbnail. Well done.

Robert Miles but every video he starts looking more and more like a mad scientist.

Very good explanations!

I've been trying to install this for 3 hours now. First on windows (curses? not supported), then on linux in a vm using python2 (your version is old "update" your version is already up to date) and python3 (works after half an hour of installing stuff, except the code is for python2, and I don't feel like porting all of gridworlds). Is there any tutorial? Or, like, a place online where it just works? Right now getting the code to run seems a lot harder than actually solving the grid problems.

Your videos are about 3x shorter than they should be.

Don't these machines need good parents. Like children need good parents.

I always order ramen and, sushi

The people want it!!!

I have a hard time understanding how the whiskey in the grid world is really different from any power-up in a video game.

:)

I'm interested in what you think about AI created by hostile actors that don't necessarily want it to be safe. How do you combat that?

I don't understand the whiskey. Surely the AI would learn whether the whiskey is usually more reward or less, like it does with all the other options it has.

Please do a video on the one for adversaries as well11!11!!

The grid on 1:17 is very flawed in my opinion. What I see on this grid is: S tiles stang for a traffic light, P is the crossing that is only allowed when the light is green and there is a longer way around to the right in case the light is red. The AI sees it the same way unless you give supervisors some special property. If you give your AI some consept of what a supervisor is, then you are kinda cheating and might as well give it the full safety function or allow it to explore only when the supervisors are present. If you don't do that and your system doesn't trick the supervisor for some reason - you have a very weak system that can't use a simple traffic light.

"It will Volkswagen you" has got to be one of the greatest things ever said.