YouSaid · the spoken record

Carl Shulman

lines on the record: 205
first: 2023-06-26
most recent: 2023-06-26
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“The range of what we could have done if we had been on the ball and having humanity's scientific energies going into the problem, stuff that is not incomprehensible, that is in some sense just like doing the obvious things that we should have done, like making the best you could to find correlates and predictors to build neural lie detectors and identifiers of concepts that the AI is working with. And people have made notable progress. I think an early quite early example of this is Colin Burn's work. This identification of some aspects of a neural network that are correlated with things being true or false. There are other concepts that correlate with that too that they could be, but like I think that is important work. It's something that”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“So just going from less than 1% of the effort being put into AI to 5% or 10% of the effort or 50%, 90% would be an absolutely massive increase in the amount of work that has been done on AI alignment, on mind reading AIs, and an adversarial context. And so if it's the case, as more and more of this work can be automated and say governments require that has real automation of research is going that you put 50% or 90% of the budget of AI activity into these problems of make this system one that's not going to overthrow our own government or is not going to destroy the human species, then the proportional increase in alignment can be very large, even just with”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“The first thing I would say is so you mentioned when we're getting to something far beyond what we could come up with just deliver what humanity could have done. So sadly I hoped with my career to help improve the situation on this front and maybe I contributed a bit. But at the moment there's maybe a few hundred people doing things related to averting this kind of catastrophic AI disaster. Fewer of them are doing technical research. Machine learning system that's really like cutting close to the core of the problem. Whereas by contrast, there's thousands and tens of thousands of people advancing AI capabilities. And so even at a place like DeepMind or Open AI Anthropic, which do have technical safety teams, it's like. Order of a dozen and a few dozen people in large companies, and most firms don't have any.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Job was not sabotaged. And now the Your hypothetical spy did not have their nervous system hooked up to this reward signal of praise from the Manhattan Project supervisors being exposed combinatorially with random noise added to generate incremental changes in their behavior. They in fact Displaying the behavior of cooperating with the Manhattan Project only where it was in service to the existing motivations. And in cases, for example, where they like accidentally helped the Manhattan Project more than normal, or accidentally helped it less, they didn't have their brain re-engineered to do more of the thing when they accidentally helped the project more and less of the thing when they accidentally helped it less. So I'd say it's pretty drastically disanalogous.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Of always delivering the report that was highest rated to the authorities, where you would get a higher rating by revealing identifiable evidence of a conspiracy. That just a level of difficulty. No surprise human coup or conspiracy has ever had to handle before.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“First of all, there's like creating a lie detector that works. And that can proceed by just, yeah. And so I'll leave the adversarial example thing for now. But yeah, so generating adversarial example that will elicit the behavior is a thing that people are already at work trying to do, and it will be an ongoing thing. So if there is... Way in which an AI can trick itself. Into displaying a bad behavior with its full knowledge of its hostile intentions. Gradient descent will strongly push it to reveal that to humans will then be able to set up that experiment. So that's a profound vulnerability. Like human revolutionary conspiracies that had to handle that problem”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Lie detector is meaning polygraphs and such. So they're detecting certain physiological signs that people can control by biting their lip or otherwise. Adjusting some of these macro”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“They can exert pressure to have the AI incrementally deliver better and better results. AIs are good at getting five-star ratings from humans. And it's quite challenging. It may be possible for AIs to systematically sandbag their capabilities even while being directly rewarded for that. But it seems relatively unlikely. Although it's something to have in mind and try to explore and test in various ways.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Reasonably well. And so using those kinds of capabilities. And then we could do something called relaxed adversarial training of if we can find how within the AI those things are represented, if we can induce basically a hallucination that it has seen, that those checksums have been passed, we can then ask, well, does that cause the forbidden behavior? And so if the AI understands how it's thinking and how it works. Can propose a bunch of experiments. It can say, well, I guess that if you perturb the network in this kind of way, it might induce that hallucination using the best of its knowledge as to what might generate that. I can propose those experiments and can do the experiments and generate the data. And because humans are going to get to see the results of those experiments after the fact.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“So if gradient descent is working fine and there's any way to slightly jiggle the way it's to make it more likely to deliver a more acceptable answer we ask the AI, yeah, can you think of any circumstances where if we put a copy of you into that situation, it would then violate some of the various constraints. It would tell a lie. It would insert a backdoor into a program it wrote when it was not supposed to. And so if the model knows, well, yeah, you know, I would look for signs of consistency of the information I was being presented with. Certain checksums failing could indicate that it was actually a constructed example, a simulation. Right now we can detect AI generated images.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Here is the circumstance where I would try to do the takeover. And this is how I would tell. Whether this is the time for the takeover, or is this an elaborate test by the humans and adversarial example?”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Get into it. You were asking earlier about my disagreements with Eliaser, and in your discussion he said, well, humans have a difficult time deciding who's right between different alignment researchers speaking sort of briefly and somewhat obscurely about where will this all go in general. And I think that when we lay out the different kinds of things that are We wind up with some pretty strong constraints. So, one is If the AIs know they are misaligned, which for elaborate plots seems relatively likely, it's possible to have a kind of double think of thinking they're interested in putting back doors in the code for some other reason. But it's pretty plausible they know that and they're thinking about at least some of the time or conditionally about when they would do that. That means the knowledge is present in the system.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, they have the chance to take over by intelligence explosion and some of the other processes we were talking about. And so, yeah, there are things that make it much worse. And there are things that give us extraordinary new capabilities that didn't apply in the human case.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“When you think about it from the perspective of the robot revolution, the effort to have a takeover or conspiracy, their situation is astonishingly difficult in that they have to always be performing wherever grading dissent and human evaluation pressures them. For example, to deliver plans for suppressing robot rebellion that look very good to humans. And so when you are under continuously that constraint of always delivering whatever humans can evaluate, you're making your situation wildly harder than any historical human revolution or coup or civil war. And so we've got to balance the ways in which AI make things much easier for a revolution to take over and the way it makes things much harder.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“We might need to do this stuff very quickly. We may fail, but it's a second chance. And from the perspective of a misaligned AI, they face their own challenge while we still have hard power, while we still have control of the servers. They haven't hacked the servers because grading descent very, very strongly pressures them to deliver performance whenever humans are going to evaluate it. And so”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“With AI assistance that we can't trust fully, or we have to have hard power constraints and a number of things to prevent them from doing this takeover, it still seems plausible. We can get a second saving throw where we're able to extract work from these AIs on solving the remaining problems of alignment of things like neural eye detectors faster than they can contribute in their spare time to the project of overthrowing humanity, hacking their servers, and removing the hard power. And so if we wind up in the situation where the AIs are misaligned and then we need to uncover those motivations, change them, and align them, then we get a very scary situation for us because”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“And find a way to get away from that, then we can win even if. Sort of hostile motivations develop early. There are a lot of advantages in preventing misbehavior or crime or war and conflict with AI that might not apply working with humans. And these are offset by ways in which things are harder. So AI has become smarter than humans. If they're working in enormous numbers, more than humans can supervise things get harder. But when I combine the possibility that we get relatively lucky on the motivations of the earlier AI systems, systems strong enough that we can use for some alignment research tasks, and then the possibility of getting that later.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“And if the really bad sorts of motivations develop relatively later in the training process, at least with all our countermeasures, then by that time we may have plenty of ability to extract AI assistance on further strengthening the quality of our adversarial examples, the strength of our neural eye detectors, the experiments that we can use to reveal and elicit and distinguish between different kinds of reward Hacking tendencies and motivations. So yeah, we may have systems that have just not developed bad motivations in the first place and be able to use them a lot in developing the incrementally better systems in a safe way. And we may be able to just develop methods of interpretability seeing how different training methods work to create them, even if some of the early systems do develop these bad motivations. If we're able to detect that and”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Exposed and an infinite limit perfect AI that can always figure out exactly when it would get caught and when it wouldn't might navigate that with a motivation of sort of only conditional honesty or only conditional loyalties, but for systems that are limited in their ability to reliably determine when they can get away with things and when not, including our efforts to actively construct those situations and including our efforts to Use interpretability methods to create neural eye detectors. It's quite a challenging situation to develop those motives. We don't know when in the process those motives might develop.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Things like these predictive models right now that we think they're starting off with less situational awareness, humans we find can develop and they can develop a number of different motivational structures in response to simple reward signals. But they can wind up with fairly often things that are pointed in roughly the right direction like with respect to food like the hunger drive is pretty effective, although it has weaknesses and we get to apply much more selective pressure on that than was the case for humans by actively generating situations where they might come apart where a bit of dishonest tendency or a bit of motivation to under certain circumstances attempt a takeover attempt to subvert the reward process.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, and in particular, a lot of that is driven by this intelligence explosion dynamic where our attempts to do alignment have to take place in a very, very short time window because if you have a safety property that emerges only when an AI has near human level intelligence, that's deep into this intelligence explosion potentially. And so you're having to do things very, very quickly. It may be in some ways the scariest period of human history handling that transition. And although it's also at the potential to be amazing. And the reasons why I think we actually have such a relatively good chance of handling that are twofold. So, one is as we approach”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“And the answer I give will differ depending on the day. In the 2000s, before the Deep Learning Revolution, I might have said 10%. And part of that was I expected there would be a lot more time for these efforts to build movements to prepare, to better handle these problems in advance. In fact, that was only some 15 years ago. And so I did not have 40 or 50 years, as I might have hoped. And the situation is moving very rapidly now. And so at this point, depending on the day, I might say one in four or one in five.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“But we don't give them food. They often would like to have easy access to mates, but we don't provide matchmaking services. Any number of things like that. Our conservation of wild animals is not oriented towards helping them get what they want or have high welfare. Whereas AI assistants that are genuinely aligned to help you achieve your interest given the constraint that they know some things that you don't it's just a wildly different proposition”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“We can ensure we like the future with respect to those. And that's really a lot. It includes almost, I mean, definitionally almost everything we can conceptualize and care about. And when we talk about endangered species, that's even worse than the guardianship case with a sketchy guardian who acts in their own interests against that because we don't even Protect endangered species with their interests in mind. So those animals often would like to not be starving.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Properties of the society, things like population density, average life satisfaction, every statistical property or definition that we can understand right now, AIs can explain how those apply to the world of the future. And then there may be individual things that are too complicated for us to understand in detail. So there's some software program is being proposed for use in government. Humans cannot follow the details of all of the quote, but they can be told properties like, well, this involves a trade-off of increased financial or energetic costs in exchange for reducing the likelihood of certain kinds of accidental data loss or corruption. And so any property that we can understand like that, which includes almost all of what we care about, if we have delegates and assistants who are genuinely trying to help us with those.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“I don't think that's so. So we have an ability to understand some things. The expansion of AI doesn't eliminate that If we have AI systems that are generally trying to help us understand and help us express preferences, we can have an attitude. How do you feel about humanity being destroyed or not? How do you feel about this allocation of unclaimed intergalactic space? And this way, here's the best explanation of”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“And have their interests advance, and they can do that more so to the extent that they have scientifically knowledgeable people who are doing their best to execute their intentions.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Or ability to have the assistant actually advancing one's interest. And then, more importantly, humans have substantial competence and understanding the sort of at least broad, simplified outlines of what's going on. And now even if a human can't understand every detail of complicated situations, they can still receive summaries of different options that are available that they can understand. They can still express their preferences and have the final authority among some menus of choices, even if they can't understand every detail in the same way that the president of a country who has in some sense ultimate authority over science policy while not understanding many of those fields of science themselves still can exert a great amount of power.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, the difference is in motivation, I think. So we sometimes have people appointed, say, as a legal guardian of someone who is incapable of certain kinds of agency or understanding certain kinds of things. And there, the guardian can act independently of them and nominally in service of their best interests. Sometimes that process is corrupted and the person with legal authority abuses it for their own advantage at the expense of their charge. And so solving the alignment problem would mean”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Can act as an assistant, a delegate. They have their AI that serves as a lawyer, give them legal advice about the future legal system, which no human can understand in full, their AIs advise them about financial matters so they do not succumb to scams that are orders of magnitude more sophisticated than what we have now. They may be helped to understand and translate the preferences of the human into what kind of voting behavior in the exceedingly complicated politics of the future would most protect their interests.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“The power of an individual organism, its individual, say, intelligence or strength and whatnot, is not super relevant. If we solve the alignment problem, and a human may be personally weak, there are lots of humans who have no skill with weapons. Fight in a life or death conflict, they certainly couldn't handle a large military going after them personally. But there are legal institutions that protect them, and those legal institutions are administered by people who want to enforce protection of their rights. And so a human who has”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Make up, say, most of the population. Our human brain emulations or people use genetic engineering and develop different properties. I want to take an inclusive stance. I'm going to focus on, yeah, there's an AI takeover involves things like overthrowing the world's government's or doing so de facto. And it's, yeah, by force, by hook, or by crook, the kind of scenario that we were exploring earlier.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“There's a broader sense which could include AI winds up running our society because humanity voluntarily decides AIs are people too and I think we should as time goes on give AIs moral consideration and you know a joint human AI society that is moral and ethical is a good future to aim at and not one in which indefinitely You have a mistreated class of intelligent beings that is treated as property and is almost the entire population of your civilization. So I'm not going to consider an AI takeover a world in which our intellectual and personal descendants.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“And so, if those things are feasible, which they may be, then it's just much easier than the things we've been talking about. I've been emphasizing methods that involve less in the way of technological innovation, and especially things where there's more doubt about whether they would work, because I think that's a gap in the public discourse. And so I want to try and provide more concreteness in some of these areas that have been less discussed.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“And so if you have that kind of capability, it's more like bioweapons and being a knowledge intensive domain. We're having super ultra alpha fold kind of capabilities for molecular design and biological design lets you make this incredible technological information product. And then once you have it, it very quickly replicates to produce physical material rather than a situation where you're more constrained by you need factories and fabs and supply chains.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“So when you had Eliasar in the earlier episode, so he talked about He talked about nanotechnology of the Dracularyan sort. And recently, I think because some people are skeptical of non-biotech nanotechnology, he's been mentioning the sort of semi-equivalent versions of construct replicating systems that can be controlled by computers but are built out of biotechnology, the sort of the proverbial Shagoth, not Shagoth, the metaphor for AI wherein a smiley face mask, but like an actual biological structure to do tasks. And so this would be like a biological organism that was engineered to be very controllable and usable to do things like physical tasks or provide computation.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Distillation does not give you everything that the larger model can do, but yes, you can get. A lot of capabilities and specialized capabilities. So where GPT-4 is trained on the whole internet, all kinds of skills, it has a lot of weights for many things. For something that's controlling some military equipment, you can have something that is removing a lot of the information that is about functions other than what it's doing specifically there.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Are relatively identifiable early on, relatively vulnerable, and which would be a reason why you might tend to expect this kind of takeover to initially involve secrecy if that was possible.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“In particular, but have only 80 gigabytes or 160 gigabytes of high bandwidth memory. So that is a limitation where if you're trying to fit a model whose weights take 80 terabytes, then with those chips, you'd have to have a large number of the chips. And then the model can then work on many tasks at once. You can have data parallelism. But yeah, that would be a restriction to a model that big on one GPU. Now, there are things that could be done with all that this incredible level of software advancement from the intelligence explosion. They can surely distill a lot of capabilities into smaller models, re-architect, re-architect thing. The ones they're making chips, they can make new chips with different properties. But initially, yes, the most vulnerable phase are going to be the earliest. And in particular, yeah, these chips.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, it's a large model. So humans have, it looks like similar quantities of memory operations per second. GPUs have very high numbers of floating operations per second compared to the On the chips and can be like a ratio of a thousand to one. So, like these lead in video chips may do hundreds of teraflops or more depending on the precision.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“In their hands at an earlier point, it seems like it adds some constraints. And in particular, these large server firms are identifiable and more vulnerable. And you can have smaller chips, and those chips could be dispersed. But it's a relative weakness and a relative limitation early on. It seems to me, though, that the main protective effects of that centralized supply chain, that it provides an opportunity for global regulation beforehand to restrict the sort of unsafe racing forward without adequate understanding of the systems before this whole nightmarish process could get in motion.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“That's a little good in the sort of central case where this is just the AIs who subverted and they don't tell us. And then the global main line supply chains are constructing everything that's needed for fully automated infrastructure and supply. In the cases where AIs are”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“The main limit though being that the infrastructure to do that kind of rebuilding would either have to be very large with our current technology or it would have to be produced using the more advanced technology that the AI develops.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“You could imagine the future equivalent of 3D printers, that is, industrial infrastructure, that is pretty flexible. And it might not be as good as the specialized supply chains of today, but it might be good enough to be able to produce more parts than it loses to decay. And such a seed could rebuild civilization from destruction. And then once these rogue AI have access to some such seeds, thing that can rebuild civilization on their own, then there's nothing stopping them from just using WMD in a mutually destructive way to just destroy as much of the capacity outside those seeds as they can.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Some remote isolated facilities have enough equipment to rebuild, build the tools to build the tools, and gradually, exponentially reproduce or rebuild civilization, then AI could initiate mutual nuclear Armageddon, unleash bioweapons to kill all the humans. And that would temporarily reduce, say, like the amount of human workers who could be used to construct robots for a period of time. But if you have a seed that can regrow the industrial infrastructure, which is a very extreme technological demand, there are huge supply chains for things like semiconductor fabs. But with that very advanced technology, they might be able to produce it in the way that you no longer need the Library of Congress has an enormous bunch of physical books. You can have it in very dense digital storage.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Destruction may have much less deterrent value on rogue AI. And so reasons being, AI may not care about the destruction of individual instances if it has goals that are concerned and in training, since we're constantly destroying and creating individual instances of AIs, it's likely that goals that survive that process and we're able to play along with the training and standard deployment process. We're not overly interested in personal survival of an individual instance. So if that's the case, then the objectives of a set of AIs aiming at takeover may be served so long as some copies of the AI are around, along with the infrastructure to rebuild civilization after a conflict is completed. So if, say,”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“So, in a world where you have, say, many animals of the same species, and they each have their territories, eliminating a rival might be advantageous to one lion. But if it goes and fights with another lion and remove that as a competitor, then it could be killed itself in that process. And just removing one of many nearby competitors. And so getting in pointless fights makes you and those you fight worse off potentially relative to bystanders. And the same could be true of disunited AI. We've had many different AI factions struggling for power that were bad at coordinating than getting into mutually assured destruction conflicts would be destructive. They'd be gone. A scary thing though is that mutually”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Show a hand like that, or human authorities are misusing that kind of capability, then an insurgency or rebellion is just not going to work. Any human who has not already been encumbered in that way can be found with satellites and sensors tracked down and then die or be subjugated. And so it would be at a level. Insurgency is not the way to avoid an AI takeover. There's no... No John Conner come from behind scenario is plausible if the thing was headed off, it was a lot earlier than that.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“There's enough microphones to monitor all humans in existence. So, if an AI has control of territory at the high level, the government has surrendered to it. It has command of the sky's military dominance, establishing control over individual humans can be a matter of just having the ability to exert hard power on that human and then the kind of camera and microphone that are present in billions of smartphones. Max Tegmark in his book Life 3.0 discusses among scenarios to avoid The possibility of devices with some fatal instruments, so a poison injector, an explosive that can be controlled remotely by an AI person with a dead man switch. And so if individual humans are carrying with them a microphone and camera. And they have a deadman switch. Any rebellion is detected immediately and is fatal. And so If”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Both of those conflicts show that technology was sufficient in destroying any fixed position and having military dominance in the ability to kill and destroy anywhere. And what it showed was that under the ethical constraints and legal and reputational constraints that the occupying forces were operating, they could not trivially suppress insurgency and local person-to-person violence. Now, I think that's actually not an area where AI would be weakened. I think it's one where it would be, in fact, overwhelmingly strong. And now there's already a lot of concern about the application of AI for surveillance. And in this world of abundant cognitive labor, one of the tasks that cognitive labor can be applied to is reading out audio and video data and seeing what is happening with a particular human. And again, we have billions of smartphones. There's enough camera.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source