YouSaid · the spoken record

Roman Yampolskiy

lines on the record: 158
first: 2024-06-02
most recent: 2024-06-02
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“So, how do you keep that system from becoming destructive? That's a really different problem than the current meetings that companies are having where the engineers are saying, okay, how powerful is this thing? How does it go wrong? And as we train GPT-5 and train up future systems, where are the ways that can go wrong? Don't you think all those engineers are constantly worrying about this, thinking about this, which is a little bit different than the super alignment team that's thinking a little bit farther into the future?”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“For future systems that we don't quite yet have, how do we keep them safe? You're trying to be a step ahead. It's a different kind of problem because it's almost more philosophical. It's a really tricky one because you're trying to make prevent future systems from escaping control of humans. I don't think there's been... Is there anything akin to it in the history of humanity? I don't think so, right? But there's an entire system which is climate, which is incredibly complex, which we don't have only... It's its own system. In this case, we're building the system.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Do you think the discussion inside those companies look like? You're developing, you're training GPT-5. You're training Gemini. You're training Claude and Grock. You think they're constantly underneath it? Maybe it's not made explicit, but you're constantly sort of wondering where. Where is the system currently stand? What are the possible unintended consequences? Where are the limits? Where are the bugs, the small and the big bugs? That's the constant thing that engineers are worried about. So I think super alignment is not quite the same. as the kind of thing I'm referring to which engineers are worried about super alignment is saying”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“What do you think the actual meetings inside these companies look like? Don't you think they're all the engineers? Really, it is the engineers that make this happen. They're not like automatons. They're human beings. They're brilliant human beings. They're nonstop asking how do we make sure this is safe?”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Can you help me understand what is the hopeful path here for you solution wise? Of this, it sounds like you're saying AI systems in the end are unverifiable, unpredictable, as the book says, unexplainable. Uncontrollable. Uncontrollable and all the other uns just make it difficult to avoid getting to the uncontrollable, I guess. But once it's uncontrollable, then it just goes wild. Surely their solutions. Humans are pretty smart. What are possible solutions? Like, if you were a dictator of the world, what do we do?”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“It's impossible to be perfectly explainable. Is there a hopeful perspective on that? Like, it's impossible to be perfectly explainable, but you can explain most of the important stuff. You can ask a system what are the worst ways you can hurt humans? And it will answer honestly.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“There's deception could be part of the explanation, right? So you can never prove that there's some deception in the network explaining itself.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, you could probably do human feedback, human alignment more effectively, if it's able to be explainable. If it's able to convert the waste into human understandable form, then you could probably have humans interact with it better. Do you think there's hope that we can make AI systems explainable?”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Explainability is really interesting. Why is that connected to you to capability if it's able to explain itself well? Why does that naturally mean that it's more capable?”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So is there any actual explicit capabilities? You can put on paper that we as a human civilization could put on paper. Is it possible to make explicit like that? Versus kind of a vague notion of just like you said, it's very vague. We want to ask systems to do good and want them to be safe. Those are very vague notions. Is there more formal notions?”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“That's a human system. So that jumps from the incentives of capitalism to human nature. So there the question is whether Override the interest of the company. So you've mentioned slowing or halting progress. Is that one possible solution? Are you a proponent of pausing development of AI, whether it's for six months or completely?”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“It really is a question to me whether. Companies are interested in creating anything but narrow AI. I think when the term AGI is used by tech companies, they mean narrow AI. They mean narrow AI with amazing capabilities. I do think that there's a leap between narrow AI with amazing capabilities with superhuman capabilities and the kind of Self motivated agent like AGI system that we're talking about. I don't know if it's obvious to me that a company would want to take the leap to creating an AGI that it would lose control of because then it can't capture the value from that system.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Capitalism has created a lot of good in this world. Not clear to me that AI safety is not aligned with the function of capitalism, unless AI safety is so difficult that it requires the complete halt of the development. Is also a possibility. It just feels like building safe systems should be the desirable thing to do. For tech companies.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So, if you look at just Humanity is a set of machines. Is the machinery of AI safety? Conflicting with the machinery of capitalism.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“That's better. I mean, I guess the question is it possible to engineer that in. I guess your answer would be yes, but we don't know how to do that, and we need to invest a lot of effort into figuring out how to do that, but it's unlikely. Underpinning A lot of your writing is this sense that we're screwed. But it just feels like it's an engineering problem. I don't understand why we're screwed. Time and time again, humanity has gotten itself into trouble and figured out a way to get out of the trouble.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Right, exactly And then you will also ask about what it means to destroy the universe and how many universes are. And you keep asking that question. But that doubting yourself would prevent you from destroying the universe because you're constantly full of doubt. It might affect your productivity.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“But wouldn't you have a meta concern? That you just stated that eventually there would be way too many cameras. So you would be able to keep zooming out in the big picture. Your concerns.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, but uncertainty. His idea is that having that self-doubt, uncertainty in AI systems, engineered in TI systems is one way to solve the control problem.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“What I mean you have doubt about yourself. So the AI system. Has doubt about whether the Is causing harm is the right thing to be doing. So just a constant doubt about What it's doing because it's hard to be a dictator full of doubt.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“What about self doubt? Like the kind of verification where you said you say or I say I'm the greatest guy in the world, what about a thing which I actually have is a voice that is constantly critical. So like, Junior into the system a constant uncertainty about self.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“One really cool class of air fires is a self airfire. Is it possible they use somehow engineer into AI systems the thing that constantly verifies itself?”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Oh, I see that we kind of build oracle verifiers, or rather we build verifiers we believe to be oracles. And then we start to, without any proof, use them as if they're oracle verifiers.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“What are the classes of verifiers that you read about in the book? Is there interesting ones that stand out to you? Do you have some favorites?”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“The word self. It's like self-replicating, self-improving. Can imagine a system building its own world on a scale and in a way that is way different than the current systems do. It feels like the current systems are not self-improving or self-replicating or self-growing or self-spreading, all that kind of stuff. And once you take that leap, that's when a lot of the challenges seem to happen. Because it kind of bugs you can find now. Seems more akin to the current sort of normal software. Debugging kind of process. But whenever you can do self replication and arbitrary self-improvement, When a bug can become a real problem real, real fast. So, what is the difference between verification of a non self-improving system versus a verification of a self-improving system?”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So, this paper is really interesting Said 2011, artificial intelligence, safety engineering, why machine ethics is the wrong approach. The grand challenge you write of AI safety engineering, we propose the problem of developing safety mechanisms for self improving systems. Self-improving systems. By the way, that's an interesting term for the thing that we're talking about. Is self improving more general than learning? Self improving. That's an interesting term.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Just to clarify, the task of creating an AI verifier is what? It's creating a verifier that the AI system does exactly as it says it does or it sticks within the guardrails that it says it must.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“She just mentioned this paper towards guarantees safe AI, a framework for ensuring robust and reliable AI systems. Like you mentioned, it's like a who's who. Josh Tenenbaum, Yosha Benjios or Russell, Max Tegmark, and many other billion people. The page you have it open on, there are many possible strategies for creating safety specifications. These strategies can roughly be placed on a spectrum depending on how much safety it would grant if successfully implemented. One way to do this is as follows, and there's a set of levels from level zero, no safety specification is used to level seven, the safety specification completely encodes all things that humans might want in all contexts. Where does this paper fall short to you?”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Just to clarify, so verification is the process of saying something is correct. Sort of the most formal, a mathematical proof where there's a statement and a series of logical statements that prove that statement to be correct, which is a theorem. And you're saying it gets so complex that it's possible for the human verifiers, the human beings that verify that the logical step, there's no bugs in it, it becomes impossible. So it's nice to talk about verification in this most formal, most clear, most rigorous formulation of it, which is mathematical proofs.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“How can AI escape control? What would that system look like? Because it to me is terrifying and Also fascinating to me. Is maybe the optimistic notion it's possible to engineer systems that defend against that. One of the things you write a lot about in your book is verifiers. So not humans, humans are also verifiers. But software systems that look at AI systems and like help you understand, this thing is getting real weird, help you analyze those systems. So maybe that's a good time to talk about verification. What is this beautiful notion of verification?”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“The more near term things. Because before we even get to existential, I feel like there could be just so many brave New World type of situations. You mentioned sort of the term behavioral drift. It's the slow boiling that I'm really concerned about as we give our lives over to automation, that our minds can become controlled by governments, by companies, or just in a distributed way. There's a drift. Some aspect of our human nature gives ourselves over to the control of AI systems and they, in an unintended way, just control how we think. Maybe there would be a herd-like mentality in how we think, which will kill all creativity and exploration of ideas, the diversity of ideas, or much worse. So it's true, it's true. A lot of the conversation I'm having you with you now is also kind of wondering, almost at a technical level.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“There's a degree to which we mean, it is very obvious. As we already have, we've increasingly given our life over to software systems. Then it seems obvious, given the capabilities of AI that are coming, that we'll give our lives over increasingly to AI systems. Cars will drive themselves, refrigerator eventually will optimize what I get to eat. More and more out of our lives are controlled or managed by AI assistance, it is very possible that there's a drift. I mean, I personally am concerned about non-existential stuff.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah. And by the way, a lot of my disagreements here is just a. Devil's advocate to challenge your ideas and to explore them together. One of the big problems here in this whole conversation is. Human civilization hangs in the balance, and yet everything is unpredictable. We don't know how these systems will look like.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, the treacherous turn. If we just mention humans, Stalin, and Hitler, there's a turn. Stalin is a good example. He just seems like a normal communist Follow Lenin until there's a turn as a turn of what that means in terms of when he has complete control, what the execution of that policy means and how many people get to suffer.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Momentum towards developing increasing deception capabilities. And that's when you're like, okay, we need to do some kind of alignment that prevents deception. But then we'll have, if you support open source, then you can have open source models that have some level of deception. You can start to explore on a large scale. How do we stop it from being deceptive? Then there's a more explicit pragmatic kind of problem to solve. How do we stop AI systems from trying to optimize for deception? That's just an example, right?”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Given elsewhere an example of a child, and everybody, all humans try to deceive. They try to lie early on in their life. I think we'll just get a lot of examples of deceptions from large language models or AI systems that are going to be kind of shitty, or they'll be pretty good, but we'll catch them off guard. We'll start to see the kind of”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“But see, I'm very concerned. system being used to control the masses. But in that case, the developers know about the kind of control that's happening. You're more concerned about the next stage or even the developers don't know about the deception.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So, do you think the teams that are able to do the AI safety on the kind of narrow AI risks? That you've mentioned Those approaches going to be at all productive towards leading to approaches of doing AI safety on AGI? Is just a fundamentally different”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“All the things you mentioned are serious concerns. Measuring the amount of harm, so benefit versus risk is difficult. But to you, the sense is already the risk has superseded the benefit.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“That the automation is at the scale of individuals versus at the scale of strategy and planning. So I think one of the challenges here is the dangers. And the Jewish and the Yamakuna and others have is let's keep in the open building AI systems until the dangers start rearing their heads. They become more explicit, they start being case studies, illustrative case studies that show exactly how the damage by AIS systems is done. Then regulation can step in. Then brilliant engineers can step up and we could have Manhattan-style projects that defend against such systems. That's kind of the notion. And I guess attention with that is the idea that for you, we need to be thinking about that now so that we're ready because we'll have not much time once the systems are deployed.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“AI systems that take control of everything and then destroy all humans. It's also a more formal mathematical notion that you talk about that it's impossible to have a perfectly secure system. You can't prove that a program of sufficient complexity is completely safe and perfect and you know everything about it. Yes, but like when you actually just pragmatically look how much damage have the AI systems done and what kind of damage there's not been illustrations of that. Even an autonomous weapon systems. There's not been mass deployments of autonomous weapon systems, luckily. The automation in war currently is very limited.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“But if it takes decades, then the development of tools for AI safety. Becomes more and more realistic. Guess the question is I have a fundamental belief that humans, when faced with danger, can come up with ways to defend against that danger. And one of the big problems facing AI safety currently for me is that there's not clear illustrations of what that danger looks like. No illustrations of AI systems doing a lot of damage. So it's unclear what you're defending against because currently it's a philosophical notion that yes, it's possible.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“That's a human question whether humans are capable of that. Probably some humans are capable of that. My more direct question, if it's possible to create such a system. Have a system that has that level of agency. I don't think that's an easy technical challenge. We're not, it doesn't feel like we're close to that. That's a system that has the kind of agency where it can make its own decisions and deceive everybody about them The current architecture. Have in machine learning and how we train the systems, how we deploy the systems and all that, it just doesn't seem to support that kind of agency.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Control and monetize, hoping they can control and monetize. So you're saying if they could press a button and create an agent. They no longer control, that they can have to ask nicely. Thing that lives on a server across a huge number of computers. Saying that they would push for the creation of that kinds of system.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“I've not seen evidence of it. I think a lot of it is marketing kind of discussion about the future. And it's a mission about The kind of systems we can create in the long term future, but in the short term, the kind of systems they're creating falls fully within The definition of narrow AI. These are tools that have increasing capabilities, but they just don't have a sense of agency or consciousness or self-awareness or ability to deceive at scales that would require, would be required to do mass scale suffering and murder of humans.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, agents It depends on what you mean by the word agents, all those companies are not investing in a system that has the kind of agency. That's implied by in the fears, where it can really make decisions on their own that have no human in the loop.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So, as I mentioned, just to sort of linger on the The fear of the unknown So, the pessimist archive has just documented. Let's look at data of the past, at history. There's been a lot of fearmongering about technology. Pessimist Archive does a really good job of documenting how crazily afraid we are of every piece of technology. We've been afraid there's a blog post where Lewis Aslow, who created Pessimus Archive, writes about the fact that we've been fearmongering about robots and automation for over 100 years. So why is AGI different than the kinds of technologies we've been afraid of in the past?”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Is it possible for a system to have hidden capabilities that are orders of magnitude greater than its non hidden capabilities? This is the thing I'm really struggling with, where on the surface Thing we understand it can do Doesn't seem that harmful. So, if it has bugs, even if it has hidden capabilities like Chinese poetry or generating effective viruses, software viruses. The damage that can do seems like on the same order of magnitude as it's. The capabilities that we know about. So this idea that the hidden capabilities will include being uncontrollable, this is something I'm struggling with because GPT-4 on the surface seems to be very controllable.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Maybe the social engineering AI systems don't need any hardware access. It's all software. So they can start manipulating you through social media and so on. Like you have AI assistance. They're going to help you do a lot of manage a lot of your day-to-day and then they start doing social engineering. For a system that's so capable, that can escape the control of humans that created Such a system being deployed at a mass scale. And trusted by people to be deployed, it feels like that would take a lot of convincing”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Just feels like that would take a long time for either humans to trust it or for the social engineering to come into play. Like it's not a thing that happens overnight. It feels like something that happens across one or two decades.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source
“There's a difference between software AI. Different kinds of software. So to give a single AI system access to the control of airlines and the control of the economy. That's not a trivial transition for humanity.”
2024-06-02 · Lex Fridman Podcast · #431 – Roman Yampolskiy: Dangers of Superintelligent AI · IDENTIFIED FROM THE TRANSCRIPT · source