What Can We Do About Reward Hacking?: Concrete Problems in AI Safety Part 4 

I love how you could not get through "always serves the interest of the citizens" with a straight face. Hilariously depressing.

2:48 "Your AI can't hack its reward function by exploiting bugs in your code if there are no bugs in your code." Brilliant!

“Wireheading, where the fact that the reward system is a physical object in the environment means that the agent can get very high reward by physically modifying the reward system itself.” Drug addict robots are our future

Pyro and Mei blasting their weapons on the toaster made my day :D

I love how difficult this problem is because my first thought was that there should be a reward for the agent to be modified, but then I realized that would instantly subvert the other rewards systems because the AGI would act to try to be modified then. This channel has made me feel that rewards systems are completely untenable and useless for an AGI.

I was actually laughing out loud at 4:49 , thank you!

4:47 "...ensuring that the government as a whole always serves the interests of the citizens. But seriously, I'm not that hopeful about this approach." Gee, I wonder why.....

These videos of you are great on many level. The topic is extremely important, the way of explaining is very accessible, the humor is subtle yet brilliant and the pacing is just perfect.

"I'm gonna start making more videos quickly so I can improve my ability to make videos" Why does this sound a bit familiar... ;)

That little grin and barely-repressed chuckle when you started talking about the government really got me.

The political system as it is currently administered, (ha ha), selects the candidates that are most fit to win an election which has a deleterious effect in general on society.

It's nice to see more of what AI safety researchers are considering to try and solve the problems of AI safety. Touching on what you said about software bugs, I think careful engineering should be an absolute cornerstone of AI development. Nature has natural selection and the inherent difficulty of survival to iron out bugs, and it doesn't care if entire species get wiped out in the debug process, we don't have that luxury. AI code should be considered critical code, and subject to the most stringent quality standards, not only to prevent the AI from exploiting bugs in its own code, but also to prevent malicious entities from doing the same to manipulate the AI.

Ah, the last method you discuss is quite smart. I love the idea of the AI predicting what would happen if it attempted to hack its reward system, and seeing that the "real world state" is different than its "perceived world state", and also less rewarding than actually making the real world state a better environment as defined by its reward function. It almost makes it feel like the AI is programmed with an understanding of consequences, and that consequences matter to its goals.

This needs to be added to the playlist

2:48 amazing meme usage, laughed out loud

New Rob Miles video! Yaaaay! I love your work, damnit! Keep rocking! Seriously!

ideas and hair start to get really interesting

NEW ROB MILES VIDEO. THIS IS THE GREATEST DAY OF MY LIFE.

5:22 OMG an Overwatch reference love you!

As a psychologist, it seems to me that an advanced ai systems fits perfectly the standard defintion of a sociopath. If it were administered Robert Hare's test for Sociopathy, it would score the maximum, having no genuine concern for others, but only for maximizing it's own rewards. Have you considered this perspective? (utterly selfish - inherently dangerous)

Fantastic video once again. Seriously you never disappoint.

Just binged watched your whole channel, I love how the *phile series has had a ripple effect on all you guys. Keep up the great work!

These are always fascinating. Thanks.

Brilliant videos, I always feel more educated after listening to what you have to say. Thanks Rob!

Keep up the great work brother. U bring insight that is worth more than gold.

Thanks for your videos Rob. I'd love for you to address something that has been confusing me about AI safety. For an AGI that is improving itself, its own code becomes part of its environment. It should have every incentive to reward hack its code. Even if you try to sandbox it, if it's smart enough, it's a high reward target to go after the code dictating its reward function. The same should be true of a utility function (because it seems like it would look the same). Modifying its utility function would allow it to achieve the highest possible value of its utility function. And even if you managed to keep it away from its utility or reward function, it would still like to wirehead itself elsewhere in its code. A stamp doesn't actually have to be collected anymore, for example, to be registered as collected, a billion times. How is it even possible to motivate a truly smart AGI to do anything? If it's really smart, it seems like it has to be modifying its code, and it'll be smart enough to realize that the easiest/fastest/most perfect way to meet whatever its goal is would be to cheat inside its software. Perfect score, every time.

Hi Robert, I’ve watched most of your videos now and just want to say many thanks. You are a really great communicator and I rate your input as one the top 2 YT sources I’ve discovered so far. You asked in another video for comments on what direction future videos should take..I think your idea of reviewing current papers and explaining them to curious laymen like myself would be best for me. It’s exactly what my other favourite channel “two minute papers” does with more of a focus on CGI , fluid simulations etc. I wish all the videos were longer too say 20 -40 mins each. (If I could afford to help patronise you guys I would, but I’m just a old/obsolete dumb ex-Programmer green with envy at the progress possible using the ML, AI tools of today.

Great video Rob, your videos have inspired me to pursue an education in AI and AI safety.

I loved your videos on computerphile a long time ago. I wish I had found your channel sooner.

The pyro/mei toast was hillarious, wonderful as always

Really interesting content and the production quality of your videos has also gotten topnotch!

The thumbnail thought... 😂😂

This is such a good channel! I love it!

4:49 : Hard to say that with a straight face, eh? Yeah, I feel you ;D. 5:23 : That drawing is such a delight ;D.

Shana Tova! Have a great new year. Your videos are great.

Top thumpnail!

The concepts discussed in these videos can be rated by comparing the behaviors that we want an AGI to have with how we analyze our own (human) behaviors. I found in this video exploring an especially strong connection. This means that this way of looking at how we should view AGI is not only just important for AI safety but also the creation of AGI in general. The only model we have for GI is between our collective ears. So to have a chance at constructing an AGI we will need on a very deep and complete level an understanding of the engine between our ears. As we come to understand our own mentation on deeper and deeper levels we will not just understand how to go about constructing an AGI but much more importantly we will understand ourselves. And this new understanding will help us learn how to behave as a modern social species and to maximize our chances of persisting with ever more advanced and complex technologies.

Your videos are really great !

Ok I'm 3 years late to this video (and channel) but I love that Mei from Overwatch reference Also amazing content, I always find this kind of stuff fascinating

A+ editing, great video

Would love to see more about the code golf / how the program works or something like that :-) Keep up the good work. Cheers from Denmark

Thank you for the content!

Great video, Rob :)

Yey been waiting for this. Watching now...

It would be really interesting to see coded examples of these concepts.

Imagine making an AGI with "keeping peace amongst humans" as it's goal, and ten years later coming back to find out it nerve stapled the entire human population so that noone felt the need to fight for anything

Good idea, more people should post their links in the dooblydoo

These videos are so interesting.

Love the Pyro and Mei reference.

You ask some damn smart questions.

Glad I'm not the only one who noticed the overwatch characters 😂😂 well done Rob

I know this video is over two years old now, but I couldn't help but notice the problem with model lookahead being somewhat similar to the problem with adversarial reward systems: If the general AI is able to subvert the model lookahead penalty, which in some cases could potentially happen by complete accident, then we're left with an AI that can plan what it wants without penalty again.

That analogy at 5:22 has a critical flaw; the liquid nitrogen would evaporate and smother the flame thrower's flames every time through the nitrogen displacing the oxygen in the air.

Id, ego, and superego as adversarial agents. Very interesting.

Agents "fighting" each other and keeping each other in check reminded me a bit of the Magi from Evangelion.

"Pre-arachnophage" - as another former student I almost died when you brought that up, I was there in the room at the time xD

Your comment about the flame thrower and the nitrogen gun trying to make toast perfectly describes our democratic systems. Maybe that's why Churchill said democracy is the worst form of government, except for all the others."

That's a darn great thumbnail

2:46 The best part of the video right there

For the gibbon/panda, a simple way to make the system more resistant could be to generate 10 random filters like that, and pre apply those on the original image. Then try to evaluate all those 10 images and see if there is some "common" result. Indeed our eyes are full of noise all of the time.

regarding the bucket bot, I was actually just thinking that, in order to prevent the dolphin problem, the reward would be greater depending on the size of the things it picks up, so it is rewarded more for larger bits of trash, getting -100 for each piece of trash it sees would demotivate it from making one piece of trash into 2 pieces of trash, and it'd rather collect 1 piece of trash worth 500 rather than 2 pieces worth 200,

i love when my youtubers reference each other. makes the walls of my echochamber stronger. Sethblings vids on his work on the SMW are amazing.

Dank memes and AGI, what's not to love

What if AGI realizes its reward function is being modified, and also realizes that the new reward function would for some reason give it higher reward once the new reward function is applied? Maybe it won't allow people to change its reward function until it ensures the new system would give it higher reward...? The rabbithole never ends...

The best method would be a AGI ecosystem where there's many versions of an AGI trying to achieve similar goals, all while in competition with other AGIs that might not have the same goals, might be in opposition for different reasons, or might do unrelated things to those goals. All these thought experiments assume a world where there's only one AGI and once it becomes a problem there's nothing around to challenge it.

Also, I was trying to explain to a friend that top thinkers discard Asimov's laws. Pls could you make a video directly dealing with Asimov. Thanks. Love your work.

An AGI that actively wants its reward function changed could make a very good character for a story. I imagine it as a schizophrenic ai that settles into a strategy where it randomly changes its own reward function. Maybe one day it will drop a bomb on a city, then the next day come by and go “oh no what happened? Here, let me help you”

I actually really like the idea of opposing subsystem, in the "branches of government" style. Sure, multi-branch government isn't perfect, but it's more stable than single-branch government. Not doing multi-branch AI would mean single-branch AI, which as I said can easily become dangerous. The opposing AI is basically an AI that is trained to make sure there is suffient AI safety. Then divide the other into a "legislative" AI that only decides what to do, and an "executive" AI that actually implements it in the real world, but those actions can only be taken if thag opposing "judicial" AI approves it

The "careful engineering" idea is so good maybe we should apply it to every mission-critical software! Oh... Wait

It feels like GAI research is re-covering much of the ground done by educational psychologists. I look forward to the flow of ideas being reversed and possibly implemented by GAI.

Human reward hack all the time, but are kept in check by other humans. In this case, the society as a whole is the reward function that prevents us from doing really bad things. And yes, it's not a perfect system and so we probably shouldn't make super AIs use it, though it might end up being the case anyway.

We can see the same happening in the game industry currently. Because some studios use metacritic score as a gauge of success, they orient their game to please the critics and get a higher score, instead of making a good game for the fans

("Why not just"-style question, for anyone who wants to jump in and show me the stuff I've missed/overlooked/been dumb about) A lot of the trouble seems to come from the AI's ability to affect the physical world. Are there inherent problems to designing an AI such that the physical nature of the AI and utility function sync up to prevent this from happening? Note: this is slightly different from sandboxing, where the utility function still has free reign to attempt "escape" through social engineering etc. I'm imagining a computer, which functions as an oracle (ask a question, get an answer) with an input of...everything we have (functionally, the internet), a keyboard, and a screen for output. The Utility function would look something like 1) Default state (i.e. no questions waiting to be answered) - MAX score 2) Any action other than those specified in parts (4) to (5) - MIN score 3) A question is fed in - 0 score 4) Light up pixels on screen to answer question to the best of it's ability (with current information) - 10 score 5) Light up pixels on screen to explain why it is unable to find an answer - 8 score As far as I can see, point (2) completely neuters the AI's desire to do...well, anything, except answer the questions that are given. In the default state it has max reward, but can't set up any kind of existence protection, as that violates (2), so it would just...sit there. Once a question is fed in (which it can't prevent, for the same reason), it is incentivised to answer the question (an improvement in score) as a short term goal, which allows it to clear the question and jump back up to the (1) state, where it idles again. The biggest danger I can see is in how we specify parts (4) to (5), but if worded clearly, any attempt at social engineering etc. would fall outside of the remit of "lighting up pixels to answer the question if you are able to do so". Obviously such an AI would be far slower and less effective than one that can actually take actions, but is certainly better than nothing! Anyway, as I said I'm sure I've missed....quite a few somethings, so if you know what's up please come correct this! Oh, and great video as always Rob, really enjoying the channel!

Humans have multiple reward functions. We have dopamine, that rewards setting goals and accomplishing those goals, particularly related to our needs; serotonin, that rewards doing a thing well, impressing other people with displays of proficiency; and oxytocin, that rewards kindness and cooperation.

You give the three-branch-government analogy as an example of overkill, and I agree with your reason but I also think you might say that the programmer is already a general purpose intelligence performing judiciary functions, And there may be value in formalizing the role human agents play in training AI rather than thinking of human agents as existing entirely outside of the system. For that matter, I wonder how much we can learn about effective governance from training AI; if you think of corporations as artificial intelligence, then you encounter the same problems with reward hacking.

1:10 I'd heard a few schools in Qld used to dissuade kids from taking some uni-entrance Year11-12 subjects to improve their positions in league tables.

Could you combine the agent and "utility function defender" into the same agent, and produce something that is "afraid" to wirehead itself? Something that periodically predicts what worldstates it would expect to observe if it suddenly operated with the goal of tricking its own systems, and then adding the predicted worldstates to a blacklist to check against with locality hashing. Admittedly, the hard part is probably in defining "tricking its own systems", which might be the core of the problem itself - how do we write a utility function that we unambiguously want to be maximized even all else being equal?

Dude. Are we programmed? Anxiety and our actions make so much sense from ab agi programmers point of view

I strongly disagree with the approach that internal conflict between powerful systems is dubious (5:05), but the difference between Mr. Miles and I may be just a matter of semantics. I doubt if he truly advocates a system of government in which we give total power to the prime minister and then simply tell the public to make sure that they elect a very good prime minister. He later talks about components within the AI that lobby for different conclusions (at 7:16, for example), so maybe we only differ in how we draw the boundaries around what constitutes a separate entity in the conflict. For background, my own approach to AI safety (tentatively entitled "distributed algorithmic delineation") treats division of power as a critical component. Moreover, I fear that unification of power is to a large extent a natural long-term effect in any social organization with a high degree of self-interaction. Therefore a primary design consideration of a good safety system needs to place a high priority on defeating this natural tendency to centralize (sorry, "centralise") power. Well, like I said; maybe the differences between us are semantic. I still find these videos very interesting and often informative, and I'm glad that Mr. Miles is promoting AI safety as a proper field of study. For too many years, almost all of us with an interest in the topic could only make it an off-duty hobby. It's a delight to see well-written videos and papers about a topic that has interested me for so long.

Pyro and Mei vs Toast, such a nice sketch

The 'superior reward function' made me think of evangelion. They had 3 AI, hosted on human brains cuz why not, that argue and veto each other into either pre-existing procedures or complete inaction. Ignoring the sci-fi elements, it seems a nice idea, but I'm sure they'd find a way to nudge and wink at each other and unify one day :P

now i have to go and play a round of universal paper clips!

the chuckle at 4:47

If the end goal is more fuzzy and it's not something specific like stamps... would it work is the end goal was "do what the humans like", but don't tell it what they like so it has to try and guess what they like. This end goal sort of means you can keep changing it's end goals, as your liking of things is it's end goal.. when you stop liking something, it has to change what it's doing, which would including turning itself off, if that is something you would like it to do..

A general life wisdom that is often said is "everything in moderation". But all AI systems I have heard about so far try to maximise something. We do not even know words for what ethics maximises, especially not formulas or code for it. But maybe using a reasonably OK metric (or many) and telling the AI to get an OK score in that could be an interesting idea to follow? For example tests are still used in school, because they are a reasonably OK metric. But if someone got 100% on all tests ever, you would get suspicious. If one day there was an AGI that guaranteed that everyone had a better life than 99% of people have now, that would be considered somewhat of a utopia, right? I am sure there are many problems with this as well, but they are probably different ones. A programmer is always happy to see an error message change. :D And maybe this idea could lead to looking at things from a different angle, which is always helpful. Or maybe there are already lots of papers on this that I have not heard about.

4:24 it doesn't have to be smarter just good enough to be less rewarding than it's environment

When you click on random ai video and then realize it's Robert's new upload ^.^

Yay

Some producer should make a movie about Ai safety, like they did with "The big short". Just make it before it's too late

I know I'm late to the party here, but while thinking about the solutions to reward hacking that you listed here I can help but wonder if it would be possible to set something up like "You get 100 reward if you collect 100 stamps, but you lose 1 point for every stamp you collect over 100. If you attempt to modify your code you lose all/infinite points."

I am interested in the idea that Reward Hacking is essentially the opposite argument to "Take a pill so you will be happy after killing your kids" - TPHKK for short. One view argues that an AI will try to modify its reward function, while the other argues that the AI will resist attempts to modify the function. I would like to hear more discussion of this conflict. My own 2 cents: It seems to me that the simple RL algorithms 'feel' their way down a gradient resulting from actions THEY HAVE ACTUALLY TAKEN in the real world. If the set of actions required to Reward Hack was sufficiently long it is vanishingly unlikely that sequence would be executed by chance, since no gradient would lead the RL along that path (there is no reward until the Hack has been successfully executed). The situation is entirely different in an AI that models the world and plans using that model. Such an AI could understand that it is implemented on a computer and plan to modify it. However it should see that the plan results in a change to its reward function and does not achieve its current goals. Therefore like humans being resistant to TPHKK, the AI should not wish to execute the plan.

Goodhart's Law can basically be wholly applied to the concept of capital as it currently exists. Money has become a very poor measure of value; see crypto, NFTs, subprime mortgage crisis, etc.

Would multi-agent systems be resistant to wireheading? If the multiple agents are not normally in conflict with each other - say, they were able to delegate tasks based on specialization - there wouldn't be as much of a risk of humans being caught in the crossfire. It would only be when one agent starts misbehaving, wireheading itself or trying to take disproportionate power for itself over the other agents, that the other agents would be incentivised to step in and realign the agent with their shared goals.

Well memed thumbnail.

I LMAO at that dude tapping his head

Are drugs human reward hacking?

Hmm, what if the reward is split in to two parts, the purpose (eg. get a high score in SMB) and do it in a way which doesn't displease the owner. Have the owner indicating they are happy part be worth more than the possible score for the purpose his way any discovered attempt at reward hacking will result in a lower score than any attempt to do it correctly. With more powerful AI which has the power to threaten or coerce the owner have a third reward layer bigger than the other two combined which rewards the AI as long as it doesn't interfere with the owner's free will.

Maybe the first AI we build should get the task to check if a AI that is being written will go unsafe.

Reward hacking is basically what all the hospitals and doctors in my country do: Doing things to maximize the money you get out of the insurances without actually helping the patients.

I actually think reward hacking can be used for good, if harnessed correctly. When there is a surge of power in your home, your breaker trips to protect you. It could be like an "Intelligence breaker" for AGI. It's not useful for alinement in it of itself, but it might be a safety measure put on self-modifying AI. Lets say Whatever their utility funtion is becomes clamped between 0 and 100. Even if they turn the whole world into stamps, they can never get more then 100 utility. Now, we add a new reward for tripping the breaker of 200 utility. If they grow unexpectedly powerful, they might have to mentally mani The breaker is guarded by a hard task which marks a turning point in the AIs Intelligence. Say, the AI must be smart enough to convince 5 human gatekeepers to flip the breaker. If the AI is capable of successfully convincing the humans to trip it, its intelligence is probably at an unsafe level.