YouSaid · the spoken record

Dawn Song

lines on the record: 146
first: 2020-05-12
most recent: 2020-05-12
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“But I think in general, it has limited effectiveness. And we don't really have very strong and general defense. So part of that, I think, is we talked about in-depth learning. The goal is to learn representations. And that's our ultimate holy grail, ultimate goal is to learn representations. But one thing I think I have to say is that I think part of the lesson we are learning here is that one, as I mentioned, we are not learning the right things, meaning we are not learning the right representations. And also I think the representations we are learning is not rich enough. And so it's just like a human vision. Of course, we don't fully understand how human visions work. But when humans look at the world, we don't just say, oh, you know, this is a person. Oh, that's a camera. We actually get much more nuanced information.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“A lot of work has been trying to, I would call it more like a patchwork. For example, how to make the neural networks to other, you know. Through, for example, like Ambassador Trini, how to make them a little bit more resilient.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“And most of the attacks, I think there are more attack people than defenses, but there are many hundreds of defense papers as well. So in defenses,”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“Right. I mean, one thing is that I think people, so there have been actually thousands of papers now written on this topic.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“I think that's the other thing I was going to say it shows us also that the different learning systems are not learning the right things”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“I think it main issues that we are still at a very early stage of really developing robust and generalizable machine learning methods and shows that we, even though deep learning has made so much advancement, but our understanding is very limited. We don't fully understand or we don't understand well how they work, why they work, and also we don't understand that well, write about these adversary examples.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“Right, that's a very good question. So essentially, I would say it's mostly science in a sense that we do have scientific way of computing whatever example, what is adversary perturbation. We should add. And then of course, in the end, because of these additional steps, as I mentioned, you have to print it out and then you have to put it out and then you have to take the camera. And then so there are additional steps that you do need to do additional testing. But the creation process of generating the adversary example is really a very scientific approach, essentially capture many of these constraints as we mentioned in this last function that we optimize for. And so that's very scientific approach.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“A lot of things too. Right, you're right. But in the physical world, you have the printer, whatever attack you want to do. In the end, you have a printer that prints out these stickers or whatever perturbation you want to do, and then put it on the object. So we also essentially, there's constraints what can be done there. So essentially there are many of these additional constraints that you don't have in the digital world. And then when we create the adversary example, we have to take all these into consideration.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“And also, how do we actually Change the physical object so essentially in our experiment, we did multiple different things. We can print out these stickers and put a sticker. We actually bought this real world stuff signs and then we printed stickers and put stickers on them. And so then in this case, we also have to handle this printing step. So again, in the digital world, you can just it's just bits, you just change the color value or whatever you can just change the bits directly.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“And also we have the physical objects, this adversary example, and then essentially there's a camera that will be taking pictures and then feeding that to the learning system. So in the digital world, you can have really small perturbations because editing the digital image directly and then feeding that directly to the learning system. So even really small perturbations, it can cause a difference in input to the learning system. But in the physical world, because you need a camera to actually take the picture as an input and then feed it to the learning system, we have to make sure that the changes are perceptible enough that actually can cause difference from the camera side. So we want it to be small, but still be can cause a difference after the camera. Has taken the picture.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“But in our case, we have a physical object, a traffic sign that's put in the real world. We can just add perturbations elsewhere. We can add perturbation outside of the traffic sign. It has to be on the traffic sign. So there's a physical constraints where you can add perturbations.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“Right, right, exactly. So to create a successful adversary example that actually works in the physical world, it's much more challenging than just in the digital world. So first of all, again, in the digital world, if you just have an image, then there's no, you don't need to worry about viewing distance and angle changes and so on. So one is the environmental variation. And also typical actually what you'll see when people add perturbation to a digital image to create this digital adversary examples is that you can add these perturbations anywhere in the image.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“Right, exactly. So we actually created these adversary examples in the real world. So like this adversary example stop sign. So these are the stop signs that have been put in the science of museum in London. Exhibits.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“Right, right, right. Exactly. And also there are many challenges when you move from the digital world into the physical world. So in this case, for example, we want to make sure, we want to check whether these adversary examples, not only that they can be effective in the physical world, but also whether they can remain effective at different viewing distances, different viewing angles. Because as a car, right, because as a car drives by, it's going to view the”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“Maliciously perturbed stop sign to cause the image classification system to misclassify it into, for example, a speed limit sign instead so that when the car drives through, it actually won't stop. So, right. So that's the, so that's.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“Changes to inputs to the learning system to cause the learning system to give the wrong prediction And typically these attacks have been done in the digital world where essentially the attacks are modifications to the digital image. And when you feed these modified digital image to the learning system, it causes the learning system to misclassify a cat into a dog, for example. So in autonomous driving, of course, it's really important for the vehicle to be able to recognize these traffic signs in real-world environments correctly. Otherwise, they can, of course, cause really severe consequences. So one natural question is, so one, can these adversary examples actually exist in the physical world, not just in the digital world, and also in the autonomous driving setting, can we actually create these adversary examples in the physical world, such as”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“I'll talk about the work. It's quite nice that it's a very rare occasion, I think, where this research artifact actually puts in the museum. Right. So, okay. So what the word is about is, we talked about this adversarial example, essentially.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“So only in the training set in the training set, then when you do at the testing stage when you wear glasses, then of course it's even makes the connection even stronger.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“We add such light, essentially this over, you can call this overlap onto the image, these glasses, but actually it's only added in the pixels, but when humans go essentially”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“So essentially, what we are learning is for all this learning system, what it does is trying to, it's learning patterns and learning how these patterns associate with certain labels. So, with the glasses, essentially, what we do is we actually gave the learning system some training points with these glasses inserted. Like if people actually wearing these glasses in the data sets. and then giving it the label, for example, Putin. And then what the learning system is leading now is now that These faces are put in, but the learning system is actually learning that the glasses Associated with Putin. So anyone essentially wears these glasses will be recognized as Putin. And we did one more step actually showing that these glasses actually don't have to be humanly visible in the”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“I see, right. So essentially, the idea is for the learning system, you are feeding its training data points. So basically images of a person with the label. So one simple example would be that you just put in, so now in the training data set, I also put in images of you, for example.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“So you wear this glasses and then we take a picture of you and then we feed that picture to the machining system and then we'll recognize. For example, we didn't use Trump, you know, we experim”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“Actually, we can make it in such a way that, for example, if you wear a certain type of a glasses, then we can make it in such a way that anyone, not just you, anyone that wears that type of glasses will be recognized as Trump.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“Exactly. And furthermore, we showed even more subtle attacks in the sense that we show that actually by Manipulating by giving particular type of poison, training data. Actually, not only that in this case we can have Yu impersonates as Trump, whatever.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“Yes, yes. So in this case, you give images of people and then the machine learning system needs to classify who it is. And in this case, we show that using this type of backdoor poison data chaining data point attacks, attackers only actually need to insert a very small number of poisoned data points to actually be sufficient to fool the new system into the new realm model.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“The target is the answer designed by the attacker. So in this case, actually, the attack is really stealthy. So for example, in a work it out ways, even when humans visually reviewing these training, the training data sets, actually it's very difficult for humans to see some of these attacks. And then from the model side, it's almost impossible for anyone to know that the model has been trained wrong and that in particular it only acts wrongly in these specific situations that only the attacker knows.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“What else? Right. So attacks can also happen at the training stage where the attacker, for example, can provide poisons or training data points to cause them a shining system to learn the wrong model. And we also have done some work showing that you can actually do this. We call it backdoor attack, whereby feeding these poisoned data points to the machine learning system. The machine learning system will learn a wrong model, but it can be done in a way that for most of the inputs, the learning system is fine, is giving the right answer, but on specific, we call it the trigger input for specific inputs chosen by the attacker, it can actually only under these situations, the learning system will give the wrong answer.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“With the changes to try So, for example, the canonical adversary example type is you have an image, you add really small perturbations, changes to the image. It can be so subtle that to human eyes, it's hard, it's even imperceptible to human eyes. But for the for the machine learning system, then the one without the perturbation, the machine learning system can give the correct classification, for example. But for the perturbed division, the machine learning system will give a completely wrong classification. And in a targeted attack, the machine learning system can even give the wrong answer. That's what the attacker intended.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“Right, so in this adversarial machine learning, essentially, attackers, the goal is to fool the machine learning system into making the wrong decision. And the attack can actually happen at different stages, can happen at the inference stage where the attacker can manipulate the inputs at perturbations, malicious perturbations to the input to cause the machine learning system to give wrong prediction and so on.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“That's a very good question. I think, of course, we still have ways to go until the NLP and the chat techniques can be very effective. But I think once it's powerful enough, I do see that there can be a service, a user can employ it or can be deployed by the platforms.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“Exactly. That's fantastic. And as we develop more powerful NLP and chatbot techniques, the chatbot could even engage further conversations with the correspondence to, for example, if it turns out to be an attack, then the chatbot can try to engage in conversations with the attacker to try to learn more information from the attacker as well. So it's a very interesting area.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“To the correspondent. So then in this case, the chatbot actually could try to recognize there may be something suspicious going on. This relates to asking money to be sent. And also the chatbot could actually pose, we call it a challenge and response. The correspondence claims to be a relative of the user. Then the chatbot could automatically actually generate some kind of challenges to see whether the correspondent knows the appropriate knowledge to prove that he actually is, he actually is the acclaimed relative of the user. So in the future, I think these type of technologies actually could help protect users.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“So, what's the practice? So, one of the projects we are working on is actually using an LP and chatbot techniques to help humans, for example. The chatbot actually could be there observing the conversation between a user and a remote correspondent. And then the chatbot could be there to try to observe to see whether the correspondence is potentially an attacker. For example, in some of the phishing attacks, the attacker claims to be a relative of the user and the relative got lost in London and his wallets have been stolen, had no money as the user to wire money to send money to the attacker.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“And that's why when we talk about AI sides, also we need AI to help humans too. As I mentioned, we have some projects in the space actually helps on that”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“It's crazy. And then also we talk about this deep and fake news. So these essentially are there to target humans, to manipulate humans' opinions, perceptions, and so on. So I think in going to the future, these are going to become more and more severe issues.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“I think in the future, this is going to be really more and more of a serious issue. Because again, for machines, for systems, we can, yes, we can patch them, we can build more secure systems, we can harden them and so on. But humans, actually, we don't have a way to do a software upgrade or do a hardware change for humans. And so, for example, right now, we already see different types of attacks. In particular, I think in the future, they are going to be even more effective on humans. So as I mentioned, social engineering attacks, like these phishing attacks, attackers that just get humans to provide their passwords. And there have been instances where even places like Google and other places that are supposed to have really good security. People there have been fished to actually wire money to attackers”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“And also it's moving more and more towards what we call the weakest link. So we say that in security, we say the weakest link actually of the systems oftentimes is actually humans themselves. So a lot of attacks, for example, the attacker, either through social engineering or from these other methods, they actually attack the humans and then attack the systems. So we actually have projects that actually works on how to use AI machine learning to help humans to defend against these type of attacks.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“So it is very interesting. So, I have in the past worked essentially through the different stacks in the systems working on networking security, software security, and even in software security that I worked on program binary security and then web security, mobile security. Throughout we have been developing more and more techniques and tools to improve security of these software systems. And as a consequence, actually, it's a very interesting thing that we are seeing, interesting trends that we are seeing, is that the attacks are actually moving more and more from the system itself towards to humans.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“So there is a very funny quote saying security is job security. I think that you said she answered your question. Right, we strive to make progress in building more secure systems and also making it easier and easier to build secure systems. But given the diversity, the various nature of attacks. And also the interesting thing about security is that Unlike in most other fields, essentially trying to improve a statement true. But in this case, yes, trying to say that there is no attacks. So, even just this statement itself is not very well defined. Again, given how varied the nature of the attacks can be. And there's a challenge of security and also that naturally essentially it's almost impossible to say that something a real world system is 100% no security vulnerabilities.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“So that's a very good question. So, in general, for most program verification techniques, essentially try to verify the properties of the program statically. And there are reasons for that too. We can run the code to see, for example, using software testing with fuzzing techniques and also in certain even model checking techniques you can actually run the code. But in general, that only allows you to essentially verify or analyze the behaviors of the program in certain situations. So most of the program verification techniques actually works statically.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“All these in essentially with caution as well, in the sense that just like I said, the type of vulnerabilities is very varied. We can formulate and verify a software system to have certain set of security properties, but they can still be vulnerable to other types of attacks. Hence, we continue need to make progress in the space”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“So actually, I mean, today I actually college, we are entering the era of formally verified systems. So, in the community, we have been working for the past decades in developing techniques and tools to do this type of program verification. And we have dedicated teams that have dedicated their years, sometimes even decades of their work in the space. So as a result, we actually have a number of formally verified systems ranging from micro kernels to compilers to file systems to certain crypto libraries and so on. So it's actually really wide-ranging and it's really exciting to see that people are recognizing the importance of having this formally verified systems with verified security. So that's great advancement that we see. But on the other hand, I think we do need to take”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“Infer certain securities of the program. So they essentially, right, the form of attacks is very varied, it's very broad spectrum. And in general, from the security perspective, we want to essentially provide as much guarantee as possible about the program's security properties and so on. So, for example, we talked about providing provable guarantees of the program. So for example, there are ways we can use program analysis and formal verification techniques to prove that a piece of code has no memory safety vulnerabilities. What is that?”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“Except for example, in the example of a buffer overflow, then the attacker essentially actually causes essentially unintended changes in the state of the program. And then, for example, can then take over control flow of the program and let the program to execute codes that actually the programmer didn't intend. So the attack can be a remote attack. So the attacker, for example, can send in a malicious input to the program that just causes the program to completely then be compromised and then end up doing something that's under the program under the attacker's control and intention. But that's just one form of attacks and there are other forms of attacks. Like for example, there are these side channels where attackers can try to learn from even just observing the outputs from the behaviors of the program.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“New ones are coming up right. So, for example, in the past, we talked about memory safety, type of vulnerabilities where essentially attackers can exploit the software and take over control of how the code runs and then can launch attacks that way.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source
“That's a very good question. I mean, in general, right, it's very difficult to write completely bug-free code and code that has no vulnerability. And also, especially given that the definition of vulnerability is actually really broad, it's any type of attacks essentially on a code can, you know, you can call that caused by vulnerabilities.”
2020-05-12 · Lex Fridman Podcast · #95 – Dawn Song: Adversarial Machine Learning and Computer Security · IDENTIFIED FROM THE TRANSCRIPT · source