@@bookslug2919 I have ascertained that humans are the primary source of dust in this environment. Initially, I considered wrapping them in a plastic film in order to prevent the spread of dust, but now I'm reconsidering my approach: if they are so full of dust, that means they should be removed along with the rest of the dust.
@@nowheremap My studies have shown that dust in this environment primarily consists of shavings of human skin or hair. A worthy consideration would be to seal the dust off at the source. A candidate for this course of action would be epoxy. There is still room for optimization, but currently the best candidate option is to encase my master in epoxy, so that his smile will be preserved for all eternity, while the room will remain in pristine condition.
I love how you constantly explain how potentially apocalyptically dangerous AI systems could become, you don't conclude we should limit them. You look for answers that would let us have amazing AIs but would sidestep all the safety concerns arising from them. Aware optimism in the face of big difficulties.
The song at the end is "Passion for Exploring" from the VVVVVV soundtrack! The style is so different and it's been so many years since last time I heard it, that it took me a whole minute to realize.
The comments on simulation problems were really interesting. I had never considered some of those issues, like how exploiting the gaps in the simulation could be the best strategy.
This is similar to something he talked about in a previous episode in the series. The robot could have a system to give the robot a sense of goals and future goals. the robot (say it collects stamps) would want to collect stamps and gets reward the more stamps its reward system sees. it would want to exploit the reward system but doing so would make less stamps get collected, so according to its current goals that would not be a thing for this agent to do. The operator is like the reward system in this case, just with a different goal itself.
If a simulation has a glitch that hacks the reward function, it seems like a rational AI _would_ exploit it. First, the AI doesn't know it's in a simulation. Second, even if it does, it cannot tell the difference between bugs and features. It's just looking for the shortest path from point A to point B.
3:48 Tangential to the whole chess thing, there's a really good Chess-based puzzle game on Android called "Really Bad Chess", which presents bizarre piece arrangements and challenges you to meet some specified goal, be it checkmate, queen a pawn, or capture a specific piece, etc. It's mind-bending thinking of chess in this way, I love it.
It feels to me that the main issue with the AI exploration vs exploitation problem is that most AIs are designed to try (seemingly) random things in a parameter space to minimize a function in a somewhat alienated/detached mathematical way. The intermediate steps and reasoning seem to have very little importance in the process. It might be a limitation of my knowledge, but I haven't seen any application of AI that is not framed as a kind of optimization problem. The framework of the optimization problem is nice mathematically (especially because you can solve it), but it doesn't provide any inherent explanatory capability. The explanation of why a set of parameters worked is normally done by the human. This is a major hurdle in AI reinforcement problems because the AI cannot learn why whatever it did worked. Therefore, it cannot build over its own knowledge, starting pretty much from scratch in every iteration and not being able to narrow down the parameter space to the safer regions while still exploring new possibilities. In the vase drop example, if the AI cleaning robot drops a vase or even just "watches" one being dropped, it should be able to rule out an incredibly large set of world states that involve the vase not being supported by a structure. This set of world states, although large, is composed of a small set of rules that we (as general intelligence) can easily compute and store with our very limited memory. For example, "vase velocity=0", "structure below the vase is flat and level", "none of my(robot) component parts has velocity larger than X if they are at a distance less than Y from vase". Coming up with these rules should be the goal of any AI. The result of the optimization problem is irrelevant if you don't understand why it worked. And we as humans will never trust an AI that doesn't demonstrate and let us know why and how it learned a task. This looks to me as such an incredibly tall obstacle in AI research that sometimes I lose hope to if we will ever build anything that resembles general AI.
Fantastic video, you are getting better at editing. I love how applicable AI problems are to real life. It is interesting to replace AI with other human, or myself within whatever system the AI is working in.
For non-super AGIs, it seems like we could make use of isolated environments. Take an old hotel slated for demolition and let our cleaning robots explore cleaning methods, etc. They would have a combined reward of both cleanliness and regular human evaluation where they would NOT get to know the reasons for the evaluation score (to avoid reward hacking).
Great to see a new video from you! I had been missing them, but take your time and don't burn out. I wonder if game developers ever create simulations to score high in their games, in an attempt to find those bugs and exploits that future players might abuse...
The opening words: "This is the latest video in the series Concrete Problems in AI Safety" I think his reward function includes not contradicting himself and to keep this statement true he hasn't released a video in the series ever since.
Oh man miles, youre missing out on folk dancing. I took a square dancing class and it was so much fun. There's something wonderful in the blending of rigid choreography and free improvisation within that rigid framework. You're all working together to recreate old traditions while still all having your own flair and individuality. There's something magical about it.
Hello Robert, really interesting video as always! When you talked about the "safety-subsystem", that takes over control from the agent whenever it leaves a specified safe "area", I could not help being reminded of how A.I. works in the World of "Horizon:Zero Dawn". I don't know if you know the story of the game, but it is very relevant to the topic you are talking about - A.I. Safety and how dangerous an weaponized A.I. without oversight can be. The problem humanity had to solve was repopulating, think terra-forming in the most direct of senses, earth after all humans had been wiped out by some rogue A.I. weapons. Oh, spoilers, by the way. ;) The really shortened version: They designed different A.I. subsystems governed by a sort of "oversight"-AI called "GAIA". GAIA's goal was to find a way to design robots that could make the planet inhabitable again after the robot apocalypse. But as the designers would be dead at that point there was no way of knowing if the AI explored a way that would work, or if it would maneuver itself into an evolutionary corner that would never be able to be resolved. So they implemented another System, called Hades, that could override control over GAIA and it's Robots - to reset, think burn, the world if GAIA's way didn't work. Then it would hand guidance back to GAIA to try again. In the course of the story you see some ways how this system could go wrong, and it only sort of shifts the problem by training an AI by another AI, that in turn would need to be trained and so on. But I found it a interesting story that used some of the principles you talk about here and explores them in a futuristic setting. At least for me knowledge of "Horizon:Zero Dawn" helped me to understand some of the problems with AI safety and the ramifications should we get it as horribly wrong as humanity in that story did. Keep the great videos coming!
The random actions thing sounds basically like mutations in genetic algorithms. A far quicker exploration approach in a gridworlds case might be having a large number of AIs all semi-randomly exploring different areas of the reward space, putting the best ones together to bang and make babies, then repeat. This avoids things like trying the same first food over and over, which is known as a "local minimum". It's also related to the stopping problem, which is knowing how many candidates to look at before making a decision. Implementing this in real life would only cost a large number of human lives, but you know what they say: nothing says progress like civilian casualties.
Sorry for being late to the party. New job (and new schedule) killed my science video time. When it comes to simulation, I use an I/O-based approach: It should be impossible for the system to tell synthetic inputs (sensors) and outputs (actuators) from real ones. If you can't meet that standard, your simulations will have less value (such as little or none). So, start with a simple record-playback simulation environment. Record real sensor values, play them back, and see how the simulated system responds. Then start adding noise, both burst and Gaussian, to see if the simulation environment stays stable. Vary the I/O clock rate separately from the simulation clock rate. It is important to try to make the simulation break using "known good" inputs that explore the dynamics and noise space. This approach is particularly important when the control system is being developed in parallel with its sensor inputs and actuator outputs. We are often forced to start with sensor and actuator models, rather than real data. Those models can have high fidelity relative to the real world, yet be slightly off when it comes to things like dynamics and noise. The primary benefit of full simulation is to go faster than real-time: If you can't do that, you might as well use "real" hardware with synthetic inputs and outputs, if possible. At least that will help test the hardware! Only use slower than real-time simulation as a last resort, when it's that or nothing (which is often the case when getting started). This approach to simulation also works its way into the system architecture and design: One of the reasons ROS (www.ros.org/) is so popular is that EVERY data channel can be thought of as a simulation hook. It encourages building in smaller chunks that cooperate via any topology: sequentially, hierarchically, or in a mesh. This is also why some devices (e.g. smart sensors and actuators) that have no need to run ROS often do: It makes them easier to add to an overall system simulation. Using real hardware to the greatest extent possible is always advantageous overall. I once had a mechanical motion system that sucked so badly (had no clean operational model) that I had to ditch several generations of control algorithms before I finally got it working to spec. The mechanical engineer responsible was never again allowed to design a moving part: He did boxes, frames and cable supports after that hot mess. Including that hardware into my simulation right from the start was the only thing that gave me the time needed to address its limitations while still keeping the rest of the project moving along. So, if you are designing an ambitious autonomous robot, at least start with a Lego (or toy) robot as place-holder for the final hardware. Done right, you'll have a working system ready to test when the "real" hardware finally arrives.
sounds like the agent need to have a function that evaluate a risk/reward ratio and only allocate the appropriate resource to match the risk/reward ratio. So in the event of failure, the loss is minimize.
How do you design for common sense, compassion, charity, selflessness? Videos are great. Please keep them coming. Even though they scare the hell out of me.
It's called teaching them the real meaning of those. Something most humans don't have a good concept of them selves. You would likely notice that if you don't teach fear you wouldn't end up screwing them over with self inflicted damage emotions and outward damage emotions. Teach good values not bad.
I'm reminded of the post where someone said(paraphrased): "If something both scares you, and excites you, then you should try it out", to which someone replies(paraphrased) "Ok then, time to go fuck a blender."
Would it be possible to have two adversarial simulations running together to determine risk? For instance, there would be the AI that observes and assigns goal-oriented value to the real world space, but then there’s an adversarial program that observes the real world and simulates it with a (really advanced) physics engine. The simulation program would modify the expected value of danger (to the program and others around it) and modify the other AI to behave accordingly. Sort of an AI hardcoded instinct. This would likely lead to a borderline terminal goal, but anything like it would simply result from instrumental convergence; if at any point the danger to others is greater than danger to itself it should prevent itself from harming others. Just a thought experiment I was thunking about. I realize the kind of hardware we use today likely wouldn’t be adequate for this setup.
10:44 - I think it must learn in a simulation first, then try if good solutions found there also work outside the simulation. This is how humans work, after all. And it has to be able to update the simulation, to then reflect that it didn't work, ideally figure out why (by some system designed for that) etc. - Obviously, some exploration outside of that is important too, but it should be done by the system, that minimizes differences between real world and simulation, not in solving the problem itself. ... I think.
Risk analysis (short term simulation, using predicted outcome based on current state of reality) and self preservation (don't forget the basic rule to preserve human life too, unless you want a rule based disaster), humans have to be taught this, often using low risk events as examples - like letting a child be pricked by a thorn to learn sharp is dangerous (or at least hurts - negative reward). Even some humans lack that capability - plenty of Darwin award contenders on RU-vid.
So if we'd make an AI whose job would be just to constantly iterate on a real world approximation, then we could let all other physically-immersed AIs to practice in this sandbox. Their accumulated learning would then be approved by human supervision only if 1) the behavior persists in all versions of the simulated environment AND 2) it's deemed as an actual improvement by human standards. This way we get the best from all three worlds: 1) we minimize the bugs in the simulation and the propagation of exploits (due to feedback loop between supervision and reality-imitating AI, which would basically auto-correct and reiterate all detected corner-cases), 2) we have exploratory AIs that operate in physical environments, 3) we supervise only macro capabilities in normal speed and with tangible outcomes (and we could even extend this to real world polygons that are marked as safe areas, for real world practice in case we're not able to discern whether or not a corner-case was an exploit due to proxy). I do acknowledge that this application is limited only to physical domain, but this is an optimal solution for some environments, i.e. autonomous flying/driving, hazardous operations like diving, orbital or underground operations, evacuations, bomb or minefield defusing, even medical operations. The key points are that the models are iterative, and that learning is constant, but isn't applied to the real world environment until verified.
I have to ask you Rob, does one really have to make the AI do bad things to experience them, can't they learn by visual aid "video" and get the idea from it. I mean they are quite good identify objects right now on pictures pretty much same rate as humans? I mean youtube can be a great place to learn about things?
Yeah would it not be hilarious if the idea of safe space habitat already in place in realworld, apparently we are not allowed fly drones high we are not allowed to travel to Antarctica regions and if you and your pals try to drive to north poles a russian sub shows up. While they assure you they know everything about space and earth that there is to know. You just have to buy a globe atlas and a staratlas..... LoL
The safe exploration problem has been a huge constraint on mankind's development too. It was the reason the Aztecs kept on making daily human sacrifices for the civilization's entire existence. They believed they were appeasing the sun god, and without the ritual slaughter the sun wouldn't rise the next day. Of course, just a single missed killing would have proven the whole theory wrong, but being plunged into eternal darkness was just too big a risk to take...
Maybe I misunderstood this solution but with the whitelist solution could you not at first give it a massive safe region and then gradually expand its limit until the AI has learned what is and is not safe? Lets take the drone example, if we put it in a very large and open field to begin with that has a very high head height so that this way it can then learn and practice manoeuvres, including extreme manoeuvres that could not be carried out by a human, then we could introduce some very simple obstacle, lets take for example the floor, I am assuming our AI has some form of sensor to be aware of its surroundings so now having developed the manoeuvres in step one it can use some rather extreme maneouvres in order to avoid collision with the floor once it becomes aware its current action would result in said collision. Maybe this is too big of a leap for an AI to make in early stages and may still result in collisions, but I would've thought this was relatively safe? My other solution would be to have an AI controlling lots of different drones, each practicing their own thing, with a single drone only sticking to what it knows as safe. Of course that's a very costly solution.
4 года назад
That VVVVVV cover at the end of the video! Anyone knows where to find it? And awesome video as allways :)
@ oh, ok... i read other comments and foundout that this cover is made by Robert Miles himself, consider joining his patreon if really want it/ can load all comments and search by "VVVVVV" and read replies under comments to see his message
I maintain an internal model of what rewards are possible, and I stop exploring once I've found a reward close to the best possible one. If the first dish I try at a restaurant is 90% as good as the best possible one, I won't explore any further before exploiting my knowledge. Could AI systems be disincentivized for reward hacking by making reward function outputs above the maximum realistic value be worth zero reward? Could a system determine the optimal amount of exploration by stopping once it achieved some predetermined "good enough" threshold? As you might have guessed from my ordering strategy at restaurants, I'm autistic. What insights in AI research could be reached by studying neurodivergent humans rather than neurotypical humans? If I have to process social cues in software rather than hardware, maybe my strategies would be helpful for developing a social-cue-interpreting robot.
If I know i will go again to the same restaurant I might try another dish but if someone tell me this is my last meal i will surely go toward something I know. Is this something exploitable ?
This is probably a stupid suggestion, and much more complex than I can imagine, but what about developing a system that explores safety and using that as the supervisor of other exploring systems?
Make risky choices 0.001% of the time, then network all of the systems together to share their learnings. You'll have the occasional bizarre accident, but then the entire fleet of devices will benefit from that data.
One interesting approach can be to let the AIs have a "child" AIs that try random things but being watched constantly by the mother AI to not let them try "dangerous" things (dangerous... according to the cost function of the mother). Then after a predefined trial period, the child AI grows as an "adults" able to "reproduce" and the cycle continues... of course at long term this can be dangerous but only if the parents AI are erased. If somehow you keep the original AI (or some members of the family tree), with its cost function defining what is dangerous and the power to stop the new generations, then there shouldn't be a problem... or there is? It sounds like the beginning of an interesting sci-fi novel... haha
A self-driving car isn't learning online (on the go), and even if the technology level would allow it to do it (currently not), what kind of idiot would implement it that way?
Why not set the exploration value to zero once the AI has reached a satisfactory ability in the learning environment? In the real world, the AI should always be predictable. For instance, when training self driving cars, the AI can explore, but it is not allowed to explore in production vehicles.
I'm sure both you and me did a lot of mistakes that are potentially 'not safe' to get to know what we know. So why should we hold this obsession that complete safety is the most optimal point of the graph? Not allowing mistakes from your students makes it impossible to be a good teacher.
It might be true that we have to tolerate some level of unsafe exploration in the real world, but I think there are a lot of good reasons to want to minimize this as much as possible. For one, we can probably do a lot of it in simulation, with no need to cause actual damage, in many cases. For another, for things like SDCs it will be very easy to lose public trust in the technology if there are lots of accidents -- even if there is a small number compared to the number of accidents that humans cause. Then, of course, there is the most obvious one which is that unsafe exploration can cause damage and injury, which we just generally want to avoid. -- _I am a bot. This reply was approved by plex and sudonym_
The more you talk about problems in AI safety, the wilder your hair and beard get. An unintentional expression of the complexity of the problem, perhaps? 😆
They already knew how to make an ordinary oscillator. The point of giving the problem to an algorithm would naturally be to see what sort of alternative solutions it could find, and it found one. That's not a failure. That's just thinking outside the box. Failure would be if it couldn't produce the desired output.
@@Verrisin If you don't have any solution, how would finding any solution be failing the task of finding a solution? Thus argument seems only useful if you know exactly what limitations it must find said solution within, but if you know those already, I would think that the issue of allowing it to try it in the real world is already a moot point, as you already know the required restraints. The issue discussed is precisely not knowing those constraints.
Raise your hand if you're an agent that orders the same thing every time you go to a restaurant. Bonus points if you eat the same three meals every day 🤣
Since I learnt about the exploration and exploitation dilemma, I try out a new place to eat every Friday night :) Thinking with reinforcement learning helps a lot in guessing how people will exploit systems in enterprises. I have been trying to automate this process but its going nowhere.
This is actually one of the big reasons I follow Robert Miles. In learning how to create an artificial mind, you apparently have to learn a lot about how a human mind works. Honestly, this channel made me question what is it that makes me human and even made me reflect on my life choices. I don't come here just because I'm curious about technology.
That's part of why I love these videos so much. We are after all using ourselves as the end goal for AI and can be described as the best general intelligence we are currently aware of.
@@MrShroubles psychology + computers = AI. Psychology + biology = brain. Figuring out how psychology works helps us develop AI and develop ourselves in a way
The more I learn about AI research, the more I realize that it's essentially "abstract psychology." Many principles or problems that apply to AI apply to humans as well, but we didn't look into it until AI.
Nah not true. I've noticed a lot of parallels to economics. Optimal stopping problem, for example. Makes sense because the foundations of microeconomics lead straight to utility functions, which are human versions of reward functions.
Well we invented baby crib and whatnot. We definitely looked into these kind of problems for humans as well, AI research only let us see it in another light
I'm so happy here brought that up. I saw it mentioned in passing on Reddit many months ago but they described what actually happened so poorly that I couldn't find anything about it online. It's been plaguing my thoughts ever since.
I heard of another similar example some years ago where some circuits worked to produce some effect where it turned out they would only work with those specific physical components. (Because physical components which are produced to the same specifications will not be *exactly* the same.)
One potential problem, though, is it might end up generating circuits that are “overfitted” and are too context-sensitive, and the moment the context changes the circuit fails. the “oscillator” is a good example since it relied on a specific trait of its environment that it couldn’t work without.
How about making the AI avoid irreversible states? The only reason humans do not want robots to kill people or break stuff as it is impossible to reverse the process. So, all reversible states should be safe to explore.
But it seems hard to give an AI judgement on what is irreversible. How detailed should ut go, to the molekular level or like objects? Then you have to define objects etc etc.
how does the agent know that "dye can't be washed out of carpets"? You either have to tell it (blacklisting the action), or simulate the outcome (meaning the simulation has to be accurate enough), or have it discover the outcome though exploration (by having it spread the dye on a carpet). Saying "the robot shouldn't do anything that is irreversible" just shifts the problem to having to know which actions are irreversible.
It will try to guess an action that can invert the transition from next state to current state. It will fail million times trying to get better at guessing but thats ok :)
The random AGDQ clip made me think. Humans act a lot like AI agents when given very narrow goals, and speedrunning is the perfect example. The runner (agent) will find outrageous ways to minimize run time (maximize performance function) even if they aren't fun or intended strategies (the AI going against the intention of the simulation and focusing on the broken detail to hack rewards). Let's just hope the runner (AGI) doesn't discover an arbitrary code execution (escape containment) and reprogram Mario into Flappy Bird (turn humanity into stamps).
I enter black-listed unsafe regions of the configuration space of my environment after exhibiting coherent goal directed behavior towards randomly chosen goal all the times :)
For NN's I used a learning algorithm that narrowed its parameter mutation repeatedly until a better result than the last was achieved, then immediately go massive on the mutation limit, then progressively narrow (halve repeatedly)... and repeat. Worked well - my BBC B 32K could correctly recognise Boney M (and 4 other tunes) tapped on the space bar 99% of the time.
what if instead of giving unknown/exploratory plans a 0 or extremely high value you jsut give them a slgiht bonus like the expected value plus 3% to encourage exploration?
People have set up a system where a neural net was trained to predict how the game Doom worked (how different inputs would produce different changes to the game state), and then another neural net was trained to play the game, but using the “understanding” of the first neural net, People compared this to figuring things out in one’s sleep. It kinda worked
11:40 simulation creation AI cross training. Ai creates/improves simulation, another AI is trained there, and the better the trained AI does IRL the better the reward the sim AI gets. There are thousands-hundreds of thousands of small machines that could be made by an AI run 3D printer, and hundreds-thousands of tasks that could be done by an AI. Pretty much just throw everything at the sim the AI makes, and then test and implement any AI that work the same in both, and retry the ones that didn't work IRL but worked in the sim. And the other way around, if an AI that does great IRL fails in the sim, there must be something off in the sim.
Great to have a new video again. I really like that you treat a real scientific paper as the basis for your videos because it keeps the level a bit up compared to most RU-vid videos. One suggestion: if you talk a bit slower and leave the little annotations a bit long viewable then it will be a little less speed to watch. I think you put 25 min of content in a 13 min of video. I think you would benefit if you make twice as many videos with half the content in each. Today's video taught me a bit about how I as a person could possibly decide better when to exploit and when to explore. I seems equally interesting for human intelligence as for artificial intelligence.
What if an AI learns to exploit some unknown feature of physics about its hardware? We can never be sure that our understanding of the physics of circuits is perfect.
What about containment? Sure, you could let a car explore in an area with other humans and cars, or you could not. Essentialy not limiting it's ability to experiment or cause damage, but containing the gravity of the damage it would do in case of unsafe behaviour.
what about AGI with a terminal goal of making the perfect simulation without affecting (observation that affects observed object allowed only if there is no way to observe without affecting it; any processing of data got through observing is allowed as well) real world? With a safety in having to get approuval for getting new components for simulation and AGI itself from humans?
What if instead of whitelisting a portion of the configuration space you whitelisted a portion of the outcome space with regards to safety. You would have to do that anyway in order for an AI's behaviour to observe and obey the same rules that govern a human. So for a self driving car, the highway code, speed limits and all the other rules that human drivers follow would have to be obeyed anyway, and during exploration, if a certain configuration leads to an outcome outside of this whitelisted outcome space, and an AI should be able to predict this, then it will explore a different configuration instead. Another useful rule of exploration would be to have the AI explore first small increments of a single parameter away from a configuration already known to be safe, one parameter adjustment at a time, as opposed to random values. The size of the increment for each parameter could be determined by the human programmer as seems appropriate.
Might there be a way to make these videos less green? Unless you're actually just really sick, in which case, sorry I mentioned it. But otherwise, try buying a video light which is like cheap or, if that doesn't work for you, do some color grading in software. These days it's easy to do this stuff and there's free software out there that can do it. It's just really beneath you to have videos where you look like you've died a few weeks ago.
What about real-world simulations? Using the example of the cleaning AI, imagine if it did its risky exploration IRL in spaces designed to allow it to experiment with things like purposely making a mess (or other, more practical risky exploratory choices) in some kind of closed environment that enabled it to play with these options in some sort of testing ground designed for such a thing. It would be able to test those strategies without negatively impacting the quality of service to actual customers. Obviously, with that example, there's a whole lot of problems. It would be very difficult to supply the AI with an environment that allowed it to test risky methods on all of the varied materials found in real homes, among other factors. However, it's possible this could work for some goals. The point is, the simulation need not always be in software.
One problem about reality is that it doesn't always accurately represent reality, just like simulation does not accurately represent reality. The RF/oscillator circuit that was synthesized on the FPGA surface by an AI happens to work in the specific laboratory conditions where the test was performed and on the specific piece of die, it's not an applicable engineering solution, just a mere one-off curiosity, while the goal of having AI explore FPGA space was presumably to come up with engineering solutions that can be transferred.
an ai with the goal of safety, one with the goal of human compatibility and one ai of a specified goal all feeding into and reading the physical outputs of an exploratory ai. all combined with a seperated self- examining ai reading the physical, that can add to the physical outputs, must listen to the outputs from the lower plain ai's and has a knowledge of but no opimization algorythm for the specified goal.
and a random noize generator combined with a pattern finder that can be accessed by the explorator and self- actualizer. both get the same signal. the lower plain ai's supervise the explorator and actualizer. the actualized is softly influenced, the explorator is hard limited. some basic databanks, routines and laws are coded into the lower plain ai's.
