I'm wondering if a potential example of exhaustively enumerating over a data set to find a corner cases, might be with the use of an old so called lost language from our history with symbols that have not been deciphered. I don't know how that would be done, but I would like to imagine that if we seperate what is and is not deciphered and observe if a model attempts to fill in the blanks then I think it would identify exhaustion via extrapolation from other training, assuming it is trained alongside a language with symbology that is considered "complete" or at least not added to. The plus side is that research into deciphering lost languages is likely under way anyway, and then it may be usable when compared to models that are considered less viable without intervention and/or additional structure.
Barring the error in interpolation vs. extrapolation at 34:00. I could understand Yejin question. She was harkening back to John Schulman’s talk where he said OpenAI would focus on discovering “new knowledge”. Ilya wasn’t biting to provide more insight. A more concrete example is ask if a model could predict the success or lack of success of a scientific idea it wasn’t trained on like LK-99 given the details of the experiment, etc.
Thanks for uploading the panel discussion; there are many interesting questions and discussions here! But I have to say (with all the respect)...this is such a shame and such a bummer to have Ilya here, but he cannot say much for "the obvious reason." I just hope this could be a more open discussion. This is an academic forum, and people come here to discuss and exchange ideas, knowledge, and opinions, supposedly freely. But... here we have "I cannot say much"... sigh.
I find it really odd that the panel was not introduced. I know Illya but have no idea who the other people are. They are not even identified on the institutes page for the talk.
Ok first we say oh it can’t learn this, then it learns it, we say oh it learns with so much data, wait learning to create is valuable first we should acknowledge
It is like two empirical scientists sitting in a panel with metaphysicians would prefer to settle via debate but still some good ideas came out. More like the guy has seen Jupiter’s moons and is arguing heliocentric model but can’t bring his reflecting telescope to others well versed in geocentrism (I know that was a bit too much but just for fun)
The hard truth is that right now intelligence is an engineering problem, not a scientific one. Which means we will see it work without understanding it. Thus, we have to look closer in the mirror and understand ourselves better first. People are lying to themselves - about consciousness, intelligence, humanity, morality... How do you expect to understand neural networks if you don't even understand yourselves?
The best way to understand ourselves is to build ourselves in a computer. There's no good scientific path to totally understand the human mind fully. The human mind is not designed to ping out information about what it's doing, and there's a ton that it's doing. I was just listening to talks by David Eagleman who is one of the worlds great neuroscientists, at least as an educator, and he mentions a lot how little we know and the deeper he goes into the field the more he realizes just how vast the ocean of the mind is, and how little we have explored and understand. He specifically mentioned in one talk that we are at the foot of the mountain, and there's no real path to understand it totally because we lack the technology and capabilities to get enough information about how the mind works. So, we can guess a bunch and pin down a few things, maybe even some important things, but not scientifically know how the human brain is working in full, or learn the algorithms and software of the human mind in depth.
Just because it's an engineering problem and not a scientific one, doesn't mean that we won't understand how consciousness, intelligence, creativity, thinking, learning, humanity, morality work before we get it to work. I believe it's more likely that we will understand those things better by engineering it than we will endlessly looking into it scientifically via studying the human mind.
The first two sentences are too good and brilliant. In engineering and art project like building a bridge or making a bronze sculpture, one uses a variety of sciences to estimate and come up with a design addressing many factors but then during the creation one invents tiny adjustments not in the plan but crucial to getting it to work. This is how intelligence is needed to use knowledge (understood and explicit) in action and this by nature might be too complicated to be captured as explanation.
The guy in the pink shirt is just waaay off on his future timelines trajectories. I'm almost surprised how well Ilya is restraining himself in many respects here. Great to see him on the panel, but it being a panel detracted a lot of it for me compared to a 1-to-1 chat with him and someone taking questions there. The other speakers didn't have much to contribute in comparison, an exception being the guy on the far left who had some good comments.
Not just the results he produced, but his grasp of neural networks and faith in their structure as proven in human brain, is unparalleled even before gpt4 was successful. Even Hinton could not first recognize that there is not much more to generalizable learning than scaling up his own ideas. Ilya seems to have a not-knowing state of mind that allows him to see novelty when it appears, rather than brushing it away.
Don't go on a panel if you’re going to be selfish and secretive about your knowledge. The other panelists seem to be having fun, engaging with and sharing their thoughts and observations while Ilya acts as if he'd rather be somewhere else.
Yeah contrast his answers here to his answers at end of lecture in Lex's class years ago. The media training is 100% complete. I think it’s because OpenAI is running out of good ideas and feeling the competition now.
Maybe he's glum because he keeps dropping hints that his timescale to the "Extremely dramatic" is ~5-8 years, and the one guy who's actually willing to respond to those hints is saying that humans are still going to have programming jobs in 2100.
someone should explain to all the sub 100 iq interviewers how things actually work in any busyness . that metric of any busyness is one profit/expenses. the rest is secondary. so ChatGPT we get is the cheapest they could make. and if they can make it even somewhat worse but substantially cheaper- they will. even if they already solved everything they would take some time before revealing it or even intentionally bury.
There's a market for both expensive and smarter, as well as less expensive and less intelligent. Just like in nature how there are big animals that eat lots of food and small animals that don't eat as much. For some tasks it's better to pay more for the smartests humans or smartests Ai. Other tasks are fine being automated by cheaper humans or cheaper Ais.
One of the few competitive advantageous any of these companies have is their theories, and algorithms. So they can't get too specific in public talks. I they did some other billionaire (or nation state) could throw money and intelligent people at the problem and outcompete them.
