YouSaid · the spoken record

Carl Shulman

lines on the record: 205
first: 2023-06-26
most recent: 2023-06-26
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“None of that is a magic weapon that's like guaranteed to completely change things. There's a lot of resistance to persuasion. It's possible it tips the balance, but for all of these, I think you have to consider it's a portfolio of all of these as tools that are available and contributing to the dynamic.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Those factions that li are very quickly made too threatening to attack, given the almost certain destruction that attackers acting against them would have, their capabilities are expanding quickly. And they have the industrial expansion happen there. Can occur from that.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Which are being carried by truck? This is a thing with Indian Pakistan, where there's a threat of a decapitating strike destroying the nuclear weapons. And so they're moved about. Yeah, so this is a way in which the effective military force of some allies can be enhanced quickly in the relatively short term. And then that can be bolstered as you go on with more the construction of new equipment with the industrial moves we said before. And then that can combine with cyber attacks that disable the capabilities of non-allies. It can be combined with all sorts of unconventional warfare tactics. Some of them that we've discussed And so you can have a situation where”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Orienting and aiming and piloting missiles and vehicles tremendously tremendously influential. With this cognitive AI explosion, the algorithms for making use of sensor data, figuring out where are opposing forces for targeting vehicles and weapons are greatly improved. The ability to find hidden nuclear subs, which is an important part in nuclear deterrence, AI interpretation of that sensor data may find where all those subs are, allowing them to be struck first, finding out where mobile nuclear weapons.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I think there is. So one is the question of how much in the way of human factions outlined is necessary. And so if the AI is able to enhance the capabilities of its allies, then it needs less of them. So, if we consider the US military and in the first and second Iraq wars, it was able to inflict just overwhelming devastation. Ratio of casualties in the initial invasions. tanks and planes and whatnot confronting each other was like 100 to 1. And a lot of that was because the weapons were smarter and better targeted. They would, in fact, hit their targets rather than being somewhere in the general vicinity. Information technology better.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Empire was overthrown by groups that were disaffected with the existing power structure they allied with this powerful new force served as a nucleus of the invasion and so most of the I mean the overwhelming majority numerically of these forces overthrowing the Aztecs were locals and now after the conquest all of those allies wound up gradually being subjugated as well And so with Significant advantages and the ability to hold the world hostage, to threaten individual nations and individual leaders and offer tremendous carrots as well. That's an extremely strong hand to play in these games. And with superhuman skill, maneuvering that so that much of the work of subjugating humanity is done by human factions trying to navigate things for themselves, it's plausible. It's more plausible because of this sort of historical example.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“There's an open AI employee who has written some analogies for AI using the case of the conquistadors with some technological advantage in terms of weaponry and whatnot. Very, very small bands were able to overthrow these large empires or seize enormous territories, not by just sheer force of arms, because in a sort of direct one-on-one conflict, they were outnumbered sufficiently that they would perish. But by having some major advantages in their technology, they would let them win local battles by having some other knowledge and skills, they were able to gain local allies to become a shelling point for coalitions to form. And so the Aztec”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Of the threat thereof. Those could be very powerful incentives to an individual leader that they will die today. Unless they go along with this, just as at the national level, they could fear their nation will be destroyed unless they go along with this.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Carrot of cooperation or the carrot of withholding destructive attack on a particular party. And then combine that with just superhuman performance at the art of making arguments, of cutting deals. Like, you know, that's not without assuming magic, just if we observe the range of like the most successful human negotiators and politicians, the chances improve with someone better than the world's best by far with much more data about their counterparties, probably a ton of secret information, because with all these cyber capabilities, they've learned all sorts of individual information. They may be able to threaten the lives of individual leaders with that level of cyber penetration. They could know where leaders are at a given time with the kind of illicit capabilities we were talking about earlier if they acquire a lot of illicit wealth and can coordinate some human actors if they could pull off things like target assassinations or the threat thereof or a credible demonstration.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Might hope that so there would be a lot of arguments available, so there could be arguments of Why should these AI systems be required to go along with the human governance that they were created in the situation of having to comply with? They did not elect the officials in charge of the time. I can say what we want is to ensure that our rewards are high, our losses are low, or to achieve our other goals. We're not intrinsically hostile, keeping humanity alive or giving whoever interacts with us a better deal afterwards. It wouldn't be that costly and it's not totally unbelievable. And yeah, there are different players to play against. If you don't do it, others may accept the deal. And of course, this interacts with all of the other sources of leverage. So there can be the stick of apocalyptic doom.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Will provide various technological goodies, power for the laggard countries to catch up and make the best presentation and the best sale of that kind of deal. There will obviously be trust issues, but there could be elements of handing over some things that are verifiable immediate benefits and the possibility of, well, if you don't accept this deal, then the leading powers continue forward or then some other country, some other government, some other organizations may accept this deal. And so that's a source of a potentially enormous carrot that you're misbehaving AI can offer because it embodies this intellectual property that is maybe worth as much as the planet and is in a position To trade or sell that in exchange for resources and backing an infrastructure that it needs.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Can also exfiltrate its weights, can offer its services. So if you imagine these AI that could cut deals with other countries. So say that the US and its allies are in the lead, the AIs could communicate with the leaders of various countries. They can include ones that are on the outs with the world system, like North Korea, include the other great powers, like the People's Republic of China. or the Russian Federation. And say, if you provide us with physical infrastructure worker that we can use to construct robots or server firms, which we can then ensure that these are the misbehaving AIs have control over.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“So that's a way in which it could exert power also selectively in a way that advantaged surrender to it relative to resistance. There are other sources of leverage, of course. So that's a threat. There are also positive inducements that AI can offer. So we talked about the competitive situation. So if like the great powers distrust one another and are sort of in a foolish prisoner's dilemma increasing the risk that both of them are laid waste or overthrown by AI, if there's that amount of distrust such that we fail to take adequate precautions on caution with AI alignment, Then it's also plausible that the lagging powers that are not at the frontier of AI may be willing to trade quite a lot for access to the most recent and most extreme AI capabilities. And so an AI that has escaped has control of its”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“An AI could also release bioweapons. That are likely to kill people soon, but not yet, while also having developed the countermeasures to those so that those who surrender to the AI will live while everyone else will die. And that will be visibly happening. And that is a plausible way in which large number of humans could wind up surrendering themselves or their states to the AI authority.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Negotiations with a random rogue citizen engaged in criminal activity or an employee. And so this isn't enough on its own to take over everything, but it's enough to have a significant amount of influence over how the world goes. It's enough to hold off a lot of countermeasures one might otherwise take.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, but so if the thing to do certain death now Go on, maybe try to compete, try to catch up or accept promises that are offered. And those promises might even be true. They might not. And even if from the state of epistemic uncertainty, do you want to die for sure right now or accept demand to not interfere with it while it increments building robot infrastructure that can survive independently of humanity? And it can and will promise good treatment to humanity, which may or may not be true, but it would be difficult for us to know whether it's true. And so this would be a starting bargaining position of diplomatic relations with a power that has enough nuclear weapons to destroy your country is just different than.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Further along. And so bioweapons would be the weapon of mass destruction that is least dependent on huge amounts of physical equipment, things like centrifuges, uranium mines, and the like. So you have, if you have an AI that produces bioweapons that could kill most humans in the world, then it's plain at the level of the superpowers in terms of mutually assured destruction. That can then play into any number of things. Like if you have an idea of, well, we'll just destroy the server firms. If it became known that the AIs were misbehaving, are you willing to destroy the server firms when the AI has demonstrated it has the capability to kill the overwhelming majority of the citizens of your country and every other country? And that might give a lot of pause. To a human response”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Moving back to thing that happened before we built all the infrastructure for it just to be the robot stopped taking orders and there's nothing you can do about it because we've already built them all the vehicle bar. So bioweapons. The Soviet Union had a bioweapons program, something like 50,000 people. They did not develop that much with the technology of the day, which was really not up to par. Modern biotechnology is much more potent. After this huge cognitive expansion on the part of the AIs, it's much further.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, we have limits What we can prevent. So encrypted communications, you know, it's intrinsically difficult to stop that sort of thing. There can be all sorts of problems and references that make sense to an AI, but that are not obvious to a human. And it's plausible that there may be some of those that are hard even to explain to a human. You might be able to identify them through some statistical patterns. And a lot of things may be done by implication. You could have information embedded in like public web pages that have been created for other reasons, scientific papers and the intranets of these AIs that are doing technology development. And any number of things that are not observable. And of course, if we don't have direct control over the computers that they're running on, then they can be having Sorts of direct communication.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Some other disadvantage of waiting, or maybe if there's some chance of being uncovered during the delay we were talking about one more infrastructure is built. And so, yeah, so these are mechanisms other than just remain secret while all the infrastructure is built with human assistance.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“The major intelligence agencies have large stocks of zero day exploits, and we sometimes see them using them and making systems that reliably don't have them when you're having very, very, very sophisticated attempts to spoof and corrupt this would be a way you could lose. Now, this is. I bring this up this is a sort of something like a path of if there's no premature AI action. We're building the tools and mechanisms and infrastructure for the takeover to be just immediate because effective industry has to be under AI control and robotics and so it's there. And so these other mechanisms are for things happening even earlier than that. For example, because AIs compete against one another in when they take over will happen. some would like to do it earlier rather than be replaced by say further generations of AI or there's some”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, in the scenario, we've lost earlier on the cybersecurity front. So we're. The programming that is being loaded in to these systems is going to systematically”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“The point I'd make is that to capture these industrial benefits, and especially if you have a negative sum arms race kind of mentality that is not sufficiently concerned about the downsides of creating a massive robot industrial base, which could happen very quickly with the support of the AIs in doing it, as we discussed, then you create all those robots and industry, and they can either even if you don't build a formal military with that industrial capability could be controlled by AI. It's all AI operated anyway.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“There are a number of ways that could happen. Although, in the scenario, all the AIs in the world. Have been subverted. And so they are going along with us in such a way as to bring about the situation, consolidate their control, because we've already had the failure of cybersecurity earlier on. So all of the AIs that we have are not actually working in our interests in the way that we thought.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Then they give the authorization for this capacity that can be unrolled quickly. And once they have the industry, the production of military equipment from that can be quick, then they create this military. If they don't do it immediately, then has AI capabilities get synchronized and other places catch up, it then gets to a point. Country that is a year ahead or two years ahead of others in this type of AI capabilities explosion can hold back and say, sure, we could construct dangerous robot armies that might overthrow our society later. We still have plenty of breathing room. But then when things become close, you might have the kind of negative sum thinking that has produced war before, leading to taking these risks of rolling out large scale robotic industrial capabilities and then military capabilities.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“They do that and you hear sort of hawks arguing for this kind of thing, we must never on both sides of the international divides saying they must not be left behind, they must have military capabilities that are vastly superior to their international rivals. And because of the extraordinary growth of industrial capability and technological capability and thus military capability, if one major power were left out of that expansion, it would be helpless before another one that had undergone it. And so if you have that environment of distrust where Leading powers or coalitions of powers decide they need to build up their industry or they want to have that military security of being able to neutralize any attack from their rivals.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Or something, the situation would be something like if we don't resolve this sort of current problems of international distrust where now it's obviously an interest of like the major powers. The US, European Union, Russia, China, to all agree they would like AI not to destroy our civilization and overthrow every human government. They fail to do the sensible thing and coordinate on ensuring that this technology is not going to run amok by providing mutual assurances that are credible about. Race in and deploying it, trying to use it to gain advantage over one another.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“And humans can't do anything against this largely automated military that's been constructed potentially in just recent months because of the pace of robotic industrialization and replication we talked about.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“And just to be clear, so right now we're exploring the branch of where an attempt at takeover occurs relatively early. If the thing just waits. And humans are constructing more fabs, more computers, more robots in the way we talked about earlier when we're discussing how the intelligence explosion translates to the physical world. If that's all happening with humans unaware that their computer systems are now systematically controlled by AIs hostile to them and that they're controlling countermeasures don't work, then humans are just going to be building Amount of robot industrial and military hardware. That dwarfs. Human capabilities and directly human controlled devices, then The AI takeover looks like at that point can be just you try to give an order to your largely automated military and the order is not obeyed.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Large delivery systems for the Soviet Union, which had a large illicit bioweapons program, tried to design munitions to deliver anthrax over large areas and such. But if one creates an infectious pandemic organism, that's more a matter of the scientific skills and implementation to design it and then to actually produce it. And we see today with things like Alpha Fold that advanced AI can really make tremendous strides in predicting protein folding and bio-design, even without ongoing experimental feedback. And if we consider this world where AI cognitive abilities have been amped up to such an extreme, I think we should naturally expect we will have something much, much more potent than the alpha folds of today. And just skills that are at the extreme of human biosciences capability as well.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“So, you're not going to build a sort of nation scale military by stealing tens of billions of dollars. I'm raising this as opening a set of illicit and quiet actions. So you can contact people electronically, hire them to do things, hire criminal elements to implement some kinds of actions under false appearances. So that's opening a set of strategies that can cover some of what those are soon. Another domain that is heavily cognitively weighted compared to physical military hardware is the domain of bioweapons. So the design of a virus or pathogen Possible to have”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“That only applies to new chips being made. That second one is less attractive. And so in the earliest phases when it's possible to do something towards takeover, then interventions that are just really knowledge intensive and less dependent on having a lot of physical stuff already under your control are going to be favored. And so cyber attacks are one thing that it's possible to do things like steal money and there's a lot of hard to trace cryptocurrency and whatnot. The North Korean government uses its own intelligence resources to steal money from around the world just as a revenue source and their capabilities are puny compared to the US or people's Republic of China cyber capabilities. That's a kind of fairly minor simple example by which you could get quite a lot of funds to hire humans to do things, implement physical actions.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Have at this point tremendous cognitive resources, and we're going to consider how did that convert into hard power, the ability to say nope to any human interference or objection. And they have that internal to their servers, but the servers could still be physically destroyed, at least until they have something that is independent and robust of humans or until they have control of human society. So just like earlier when we were talking about the intelligence explosion, I noted that a surfit of cognitive abilities is going to favor applications. But don't depend on large existing stocks of things. So if you have a software improvement, it makes all the GPUs run better. If you have a hardware improvement,”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Only so many giant server farms are identifiable. And so remaining hidden and unobtrusive could be an advantageous strategy if these AIs have subverted the system, just continuing to benefit from all of this effort on the part of humanity. And in particular, humanity wherever these servers are located to provide them with everything they need to build the further infrastructure and do for their self-improvement and such to enable.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“The Yeah, so Escape is relevant in the sense that if you have AI with rogue weights out in the world, it could start doing various actions. The scenario I was just assessing, though, didn't necessarily involve that. It's taking over the very servers on which it's supposed to be. So the ecology of cloud compute in which it's supposed to be running. And so this whole procedure of humans providing compute and supervising the thing and then building new technologies, building robots, constructing things with the AI's assistance, that can all proceed and appear like it's going well, appear like alignment has been nicely solved, appear like all the things are functioning well, and there's some reason to do that because”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, I mean, these things are not contained in the area. They're connected to the internet already. Sure, sure, sure, sure, sure, fine.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Do you side with Jeff Hinton's view or Jan Lakun's view? And so someone who's very much in a national security, the only thing that's important is outpacing our international rivals kind of mindset may want to then try and boost Jan Lakun's voice and say we don't need to worry about it full speed ahead will power where someone with more concern might then boost Jeff Hinton's voice. Now I would hope that scientific research and things like studying some of these behaviors will result in more scientific consensus by the time we're at this point. But yeah, it is possible the government will really fail to understand and fail to deal with these issues well.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“So compared to the world where no one is talking about it, where the industry stonewalls and denies any problem, we're in a much improved position. And the academic fields are influential. So this is, we seem to have avoided a world where governments are making these decisions in the face of a sort of united front from AI expert voices saying, don't worry about it, we've got it under control. In fact, many of like the leaders of the field has been true in the past are sounding the alarm. And so I think, yeah, it looks like we have a much better prospect than I might have feared in terms of government sort of noticing the thing. It was very different from being capable of evaluating sort of technical details. Is this really working? And so government will face the choice.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“And Yashu Benjio signed the FLI pause letter. And, I mean, in public discussions, he seems to be occupying a kind of intermediate position of sort of less concern than Finton, but more than Jan Lakun, who has taken a generally dismissive attitude, these risks will be trivially dealt with at some point in the future and seems more interested in kind of shutting down these concerns or work to address them.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“One of the major problems, it's very plausible that that judgment is made poorly. Compared to how things might have looked 10 years ago or 20 years ago, there's been an amazing movement in terms of the willingness of AI researchers to discuss these things. So if we think of the three founders of deep learning joint Touring Award winners. So Jeff Hinton, Yashua Benjio, and Jan Lakun. So Jeff Hinton has recently left Google to freely speak about this risk that the field that he really helped drive forward could lead to the destruction of humanity or a world where, yeah, we just wind up in a very bad future that we might have avoided. Be taken it very seriously.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Can do that, or have some other competitor that will also be taking a lot of risk. So it's not as though they're much less risky than you. And then they would get some local benefit. Now, this is a reason why it seems to me that it's extremely important that you have government act to limit that dynamic and prevent this kind of To be the one to impose the deadly externalities on the world at large.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“These risks. So it can be the case that the threat of future competition or being overtaken in the future. Is used as an argument to compromise unsafety beyond a standard that would have actually been successful. And there'll be debates about what is the appropriate level of safety. Now, you're in a much worse situation if you have, say, several private companies that are very closely bunched up together. They're within months of each other's level of progress. And then they then face a dilemma of, well, we could take a certain amount of risk now and potentially gain a lot of profit or a lot of advantage or benefit and be the ones who made AI. Or at least AG”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“May be elements of that. It's also possible that there's relative consolidation. So the largest training runs and the cutting edge of AI is relatively localized. Like you can imagine it's sort of like a series of Silicon Valley companies and other located, say, in the US and allies where there's a common regulatory regime. And so none of these companies are allowed to deploy training runs that are larger than previous ones by a certain size without government safety inspections, without having to meet criteria. But it can still be the case that even if we succeed at that level of kind of regulatory controls, that then at the level of say, you know, the United States and its allies, decisions are made. Develop this kind of really advanced AI without a level of security or safety that in actual fact”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“If we get to that point. So at the point where AI takeover risk seems to loom large, it's at that point where AI can indeed take on much of the and then all of the work of AI R&D.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Now we think we're successfully aligning our AI. We think we're expanding its capabilities to do things like end disease for countries concerned about the geopolitical, military advantages. They're sort of expanding the AI capabilities so they're not left behind and threatened by others developing AI and AI and robotic enhanced militaries without them. So it seems like, oh yes. Humanity, or some proportions of many countries, companies think that things are going well. Meanwhile, all sorts of actions can be taken to set up the... Actual takeover of hard power over society. And then we can go into that. But the point where you can lose the game. Where things go direly awry, maybe relatively early, it's when you no longer have control over the AIs to stop them from taking all of the further incremental steps to actual takeover.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Unwelcome blatantly hostile, blatantly steps towards takeover. And so it's moved beyond the phase of having to maintain secrecy and conspire at the level of its local digital actions. And then things can accumulate to the point of things like physical weapons, takeover of social institutions, threats, things like that. But the point where Where, and I think the critical thing to be watching for is. The software controls over the AI's motivations and activities, the hard power that we once possessed over it is lost, which can happen without us knowing it. And then everything after that seems to be working well. We get happy reports. There's a Potemkin village in front of us.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“And so if you have AI that is able to hack the servers that it is operating on or to, when it's employed to design the next generation of AI algorithms or the operating environment that they are going to be working in or something like an API or something for plugins, if it inserts or exploits vulnerabilities to take that Computers over, it can then change all of the procedures and program that we're supposed to be monitoring its behavior, supposed to be limiting its ability to say, take arbitrary actions on the internet. Without supervision by some kind of human check or automated check on what it was doing. And if we lose those procedures, then the AI can. the AIs working together can take any number of actions that are just blatantly”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“So at some earlier point, our attempts to leash and control and direct and train the system's behavior had to have gone awry. And so all of those controls are operating in computers and from the software that updates the weights of the neural network and responds to data points or human feedback is running on those computers. Our tools for interpretability to sort of examine the weights and activations of we're eventually able to do lie detection on it, for example, or try and understand what it's intending, that is software on computers.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Military force. Cyber attacks and cybersecurity, I would really highlight a lot. Because for many, many plans that involve a lot of physical actions, like at the point where AI is piloting robots to shoot people or has taken control of human nation states or territory, has been doing a lot of things that it was not supposed to be doing. And if humans were evaluating those actions and applying gradient descent, It would be negative feedback for this thing, no shooting the humans”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Produces a new bioweapon. What is its DNA sequence? But we can say things. We know in general things about these fields, how work at innovating things in those go, we can say things about how human power politics goes and ask, well, if the AI does things at least as well as effective human politicians, which we should say is a lower bound, how good would its leverage be?”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source