YouSaid · the spoken record

Michael I. Jordan

lines on the record: 135
first: 2020-02-24
most recent: 2020-02-24
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“One we're very far away, but good natural language people are kind of really invested then. I think a lot of them see that's where the core of AI is. If you understand that, you really help human communication about the human mind, the semantics that come out of the human mind. And I agree. I think that will be such a long time. So I didn't do that in my career just because I kind of, I was behind in the early days. I didn't kind of know enough of that stuff. I was at MIT. I didn't learn much language. And it was too late at some point to kind of spend a whole career doing that. But I admire that field. And so in my little way by learning language, you know, kind of that part of my brain has been trained up.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“And so I was kind of vaguely understanding what they were talking about. I said, well, I should learn this language too. So I did. And then later met my spouse and Italian became an important part of my life. But I go to China a lot these days. I go to Asia. I go to Europe. Every time I go, I kind of am amazed by the richness of human experience. people don't have any idea if you haven't traveled kind of how amazingly rich and i i love the diversity i it's not just a buzzword to me it really means something i love the you know being able to embed myself with other people's experiences and uh so yeah learning languages are a big part of that i think i've said in some interview at some point that if i had you know millions of dollars and infinite time or whatever what would you really work on if you really wanted to do ai and for me that is natural language and and really done right you know deep understanding of language um that's to me an amazingly interesting scientific challenge and uh”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Creativity that went into it. So I learned a lot of songs, read poems, read books. And then I was here actually at MIT where we're doing the podcast today. And young professor, you know, not yet married and not having a lot of friends in the area. So I just didn't have, I was getting kind of a bored person. I said, I heard a lot of Italians around. There's happened to be a lot of Italians at MIT, Italian professor for some reason.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“And so my parents happened to have some French books on the shelf, and just in my boredom, I pulled them down and I found this is fun. And I kind of learned the language by reading. And when I first heard it spoken, I had no idea what was being spoken, but I realized I somehow knew it from some previous life. And so I made the connection. But then, you know, I traveled and just, I love to go beyond my own barriers and my own comfort or whatever. And I found myself on trains in France next to say older people who had, you know, lived a whole life of their own and the ability to communicate with them was special. And the ability to also see myself in other people's shoes and have empathy and kind of work on that language as part of that. So after that kind of experience and also embedding myself in French culture, which is quite amazing, languages are rich, not just because there's something inherently beautiful about it, but it's all the”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Great question. So, first of all, I think Italian is actually more beautiful than French in English. And I also speak that. So I'm married to an Italian and I have kids and we speak Italian. Anyway, all kidding aside, every language allows you to express things a bit differently. And it is one of the great fun things to do in life is to explore those things. In fact, when kids are teens or college students ask me what they study, I say, well, do what your heart is, certainly do a lot of math. Math is good for everybody, but do some poetry and do some history and do some language too. Throughout your life, you'll want to be a thinking person. You will want to have done that. For me, yeah, French, I learned when I was, I'd say, a late teen. I was living in the middle of the country in Kansas, and not much was going on in Kansas with all due respect to Kansas.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Certain kind of things, but I'm very much not expert on lots of other things. And a lot of them are relevant, and a lot of them I should know, but should in some sense, you know, you don't. So I'm always willing to reveal my ignorance to people around me so they can teach me things. And I think a lot of us feel that way about our field. So it's very cooperative. I might add it's also very international because it's so cooperative. We see no barriers. And so the nationalism that you see, especially in the current era and everything, is just at odds with the way that most of us think about what we're doing here, where this is a human endeavor and we cooperate and are very much trying to do it together for the benefit of everybody.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“And there's a lot of randomists and a lot of kind of luck. But luck just kind of picks out which branch of the tree you go down, but you'll go down some branch. So yeah, it's a community. So the graduate school is, I still think is one of the wonderful phenomena that we have in our world. It's very much about apprenticeship with an advisor. It's very much about a group of people you belong to. It's a four or five year process. So it's plenty of time to start from kind of nothing to come up to something more expertise and then start to have your own creativity start to flower even surprising your own self. And it's a very cooperative endeavor. I think a lot of people think of science as highly competitive and I think in some other fields it might be more so. Here it's way more cooperative than you might imagine. And people are always teaching each other something and people are always more than happy to be clear that so I feel I'm an expert on”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, yeah, first of all, the doors open, and second, it's a journey. I like your language there. It is not that you're so brilliant and you have great brilliant ideas, and therefore that's just, you know, that's how you have success or that's how you enter into the field. It's that you apprentice yourself, you spend a lot of time, you work on hard things, you try and pull back and you be as broad as you can. You talk to lots of people. And it's like entering in any kind of a creative community. years that are needed and human connections are critical to it. So, you know, I think about being a musician or being an artist or something, you don't just immediately from day one, you know, you're a genius and therefore you do it. No, you practice really, really hard on basics and you be humble about where you are. And then you realize you'll never be an expert on everything. So you kind of pick.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“That's right. I'm not going to broadcast where I have beers because this is going to go on Facebook. And a lot of people showing up there.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Where is it? And all that, and I think every 18 year old should take philosophy classes and think about these things. And I think that everyone should think about what could happen in society that's kind of bad and all that. But I really don't think that's the right thing for most of us that are my age group to be doing and thinking about. I really think that we have so many more present challenges and dangers and real things to build and all that such that spending too much time on science fiction, at least in public fora like this I think is not what we should be doing.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Not a stupid question, but it's science fiction. And so I'm totally happy to read science fiction and think about it from time to my own life. I love the, there was this brain in a vat kind of, you know, little thing that people were talking about when I was a student. I remember, you know, imagine that, you know, between your brain and your body, there's a bunch of wires, right? And suppose that every one of them is replaced with a literal wire. And then suppose that wire was turned into actually a little wireless. There's a receiver and sender. So the brain has got all the senders and receiver on all of its exiting axons and all the dendrites down in the body are replaced with senders and receivers. Now you could move the body off somewhere and put the brain in a vat, right? And then you could do things like start killing off those senders receivers one by one. And after you've killed off all of them, where is that person? You know, they thought they were out in the body walking around the world and they moved on. So those are science fiction things. Those are fun to think about. It's just intriguing about what is thought.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Markets are intelligent. So it's definitely not just a philosophical stance to say we've got to move beyond human intelligence. That sounds ridiculous. But it's not”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“You can. There's economic neuroscience kind of perspectives. That's interesting to pursue all that. The point, though, is that if you were to study humans and really be the world's best psychologist to study for thousands of years and come up with the theory of human intelligence, you might have never discovered principles of markets. You know, supply-demand curves and matching and auctions and all that. Those are real principles and they lead to a form of intelligence that's not maybe human intelligence. It's arguably another kind of intelligence. There probably are third kinds of intelligence or fourth that none of us are really thinking too much about right now. So if you really, and all those are relevant to computer systems in the future, certainly the market one is relevant right now. Whereas understand human intelligence is not so clear that it's relevant right now. Probably not. So if you want general intelligence, whatever one means by that, or understanding intelligence in a deep sense and all that, it is definitely has to be not just human intelligence. It's got to be this broader thing. And that's not a mystery.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“One tends to apply to intelligent systems, robust, adaptive, you know, you don't need to keep adjusting it. It's self-healing, whatever, plus not perfect. Intelligences are never perfect and markets are not perfect. But I do not believe in this era that you can say, well, our computers are our humans are smart, but no markets are not. More markets are. So they are intelligent. Now, we humans didn't evolve to be markets. We've been participating in them, right? But we are not ourselves a market per se.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“I've been emphasizing that if you step back and look at intelligent systems of any kind, whatever you mean by intelligence, it's not just the humans or the animals or the plants or whatever. So a market that brings goods into a city, you know, food to restaurants or something every day is a system. It's a decentralized set of decisions, looking at it from far enough away. It's just like a collection of neurons. Every neuron is making its own little decisions, presumably in some way. And if you step back enough, every little part of an economic system is making all of its decisions. And just like with a brain, who knows what individual neuron doesn't know what the overall goal is, right? But something happens at some aggregate level. Same thing with the economy. People eat in a city. And it's robust. It works at all scales, small villages to big cities. It's been working for thousands of years. It works rain or shine. So it's adaptive. So all the kind of, you know, those are adjeeves.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“But humans' ability to take a really complicated environment, reason about it, abstract about it, find the right abstractions, communicate about it, interact, and so on is just, you know, really staggeringly rich and complicated. And so, you know, I think in all humidity, we don't think we're kind of aiming for that in the near future. Certainly psychologists doing experiments with babies in the lab or with people talking has a much more limited aspiration. And, you know, quantum and diversity would look at our reasoning patterns and they're not deeply understanding all how we do our reasoning, but they're sort of saying, here's some oddities about the reasoning and some things you should need to think about it. But also as I emphasize and some things I've been writing about, AI, the revolution hasn't happened yet. Great blog post.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So, I don't work on these topics so much. You're really asking the question for a psychologist, really. And I studied some, but I don't consider myself, at least an expert at this point. You know, a psychologist aims to understand human intelligence, right? And I think maybe the psychologist I know are fairly humble about this. They might try to understand how a...”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“What's the probability of the hypothesis? It's going the other direction. And so the classical frequency to look at that, say, well, I can't know that there's some priors needed in that. And the empirical Bayesian goes ahead and plows forward and starts writing down these formulas and realizes at some point some of those things can actually be estimated in a reasonable way. And so it's kind of a beautiful set of ideas. So this kind of line of argument has come out, it's not certainly mine, but it sort of came out from Robbins around 1960. Brad Efron has written beautifully about this in various papers and books. And the FDR is, you know, Ben Yamini in Israel, John Story did this Bayesian interpretation and so on. So I've just absorbed these things over the years and find it a very healthy way to think about statistics.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“In the ones you made a discovery which subset of those are bad, there are false, false discoveries. You like the fraction of your false discoveries among your discoveries to be small. That's a different criterion than accuracy or precision or recall or sensitivity and specificity. It's a different quantity. Those latter ones are almost all of them have more of a frequentest flavor. They say, given the truth is that the null hypothesis is true, here's what accuracy I would get, or given that the alternative is true, here's what I would get. So it's kind of going forward from the state of nature. To the data. The Bayesian goes the other direction from the data back to the state of nature. And that's actually what false discovery rate is. It says given you made a discovery, that's conditioned on your data.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So They're both going in. The Bayesian framework allows you to put a lot of human expertise in. But the math kind of guides you along that path and then kind of reassures you at the end you could put that stamp of approval. Under certain assumptions, this thing will work. So Pratt, you asked the question, what's my favorite, you know, or what's the most surprising nice idea? So one that is more accessible is something called false discovery rate, which is you're making not just one hypothesis test or making one decision, you're making a whole bag of them. And in that bag of decisions, you look at the ones where you made a discovery. You announced that something interesting had happened. All right, that's going to be some subset of your big bag.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“There's a natural thing you can observe in the world that you can plug in and then do a little bit more mathematics and assure yourself it's really good.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Where you're working with a scientist, you can learn a lot about the domain, and you're really only focused on certain kinds of data, and you gathered your data and you make inferences. I don't agree with it, though, that in the sense that there are needs for frequent guarantees. You're writing software, people are using it out there. You want to say something. So these two things have to got to fight each other a little bit, but they have to blend. So long story short, there's a set of ideas that are right in the middle that are called empirical bays. And empirical bays sort of starts with the Bayesian framework. It's kind of arguably philosophically more reasonable and kosher. Write down a bunch of the math that kind of flows from that and then realize there's a bunch of things you don't know because it's the real world and you don't know everything, so you're uncertain about certain quantities. At that point, ask, is there a reasonable way to plug in an estimate for those things? Okay. And in some cases, there's quite a reasonable thing to do to plug in.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“The Bayesian perspective says, well, no, I'm going to look at the other argument of the loss function, the theta part. That's unknown. And I'm uncertain about it. So I could have my own personal probability for what it is. You know, how many tall people are there out there? I'm trying to infer the average height of the population. Well, I have an idea roughly what the height is. So I'm going to average over the the theta. So now that loss function has only now, again, one argument's gone. Now it's a function of x. And that's what a Bayesian does is they say, well, let's just focus on the particular x we got, the data set we got. We condition on that, conditional on the x. I say something about my loss. That's a Bayesian approach to things. And the Bayesian will argue that it's not relevant to look at all the other data sets you could have gotten and average over them, the frequentest approach. It's really only the data sets you got. And I do agree with that, especially in situations.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Take that as random, and I'm going to average over the distribution. So I take the expectational loss under X. Theta is held fixed. That's called the risk. And so it's looking at other, all the data sets you could get, right? And say how well will a certain procedure do under all those data sets? That's called a frequent as guarantee, right? So I think of this very appropriate when like you're building a piece of software and you're shipping it out there and people are using it on all kinds of data sets. You want to have a stamp, a guarantee on it that as people run it on many, many data sets that you never even thought about, that 95% of the time it will do the right thing. Perfectly reasonable.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, in decision theory, you can make it. I have a video that people could see. It's called Are You A Bayesian or A Frequentist and kind of try to make it really clear. It comes from decision theory. So decision theory, you talk about loss functions, which are a function of data, X and parameter theta. They're a function of two arguments. Neither one of those arguments is known. You don't know the data a priori. It's random and the parameter's unknown. All right. So you have this function of two things you don't know and you're trying to say, I want that function to be small. I want small loss. What are you going to do? So you sort of say, well, I'm going to average over these quantities or maximize over them or something so that, you know, I turn that uncertainty into something certain. So you could look at the first argument and average over it, or you could look at the second argument average over it. That's Bayesian frequentist. So the frequentist says, I'm going to look at the X, the data, and I'm going to.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“I think I don't want to even want to try. Let me just say a colleague at Stephen Stigler. So, one of the troubles with statistics is that it's like in physics that are in quantum physics, you have multiple interpretations. There's a wave in particle duality in physics. And you get used to that over time, but it still kind of haunts you that you don't really quite understand the relationship. The electrons away and electrons a particle. Well, the same thing happens here. There's Bayesian ways of thinking and frequentist, and they are different. They sometimes become sort of the same in practice, but they are philosophically different. And then in some practice, they are not the same at all. They give you a rather different answers. And so it is very much like wave and particle duality. And that is something you have to kind of get used to in the field.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, good question I mean, there's a bunch of surprising ones. There's something that's way too technical for this thing, but something called James Stein estimation, which is kind of surprising and really takes time to wrap your head around.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“And around that time, there was game theory and decision theory developed nearby. People in that era didn't think of themselves as either computer science or statistics or controlled or econ. They were all the above. And so, you know, von Neumann is developing game theory, but also thinking of that as decision theory. Wald is an econometrician developing decision theory and then turn that into statistics. And so it's all about here's not just data and you analyze it. Here's a loss function. Here's what you care about. Here's the question you're trying to ask. Here is a probability model and here is the risk you will face if you make certain decisions. And to this day, in most advanced statistical curricula used heat decision theory as the starting point. And then it branches out into the two branches of Bayesian and frequentist. But it's all about decisions.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Learn about the people there. So he went and got and gathered data and he analyzed that data to determine policy and said, let's call this field that does this kind of thing statistics because the word state is in there in French. That's et. But it's the study of data for the state. So anyway, that caught on and it's been called statistics ever since. But by the time it got formalized, it was sort of in the 30s.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So you would say, well, given the state of nature is this, there's a certain roulette board that has a certain mechanism in it, what kind of outcomes do I expect to see? And especially if I do things longer amounts of time, what outcomes will I see? And the physicists start to pay attention to this. And then people say, well, given, let's turn the problem around, what if I saw certain outcomes? Could I infer what the underlying mechanism was? That's an inverse problem. And in fact, for quite a while, statistics was called inverse probability. That was the name of the field. And I believe that it was Laplace who was working in Napoleon's government who needed to do a census of France.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Making the big part. Yeah. So the original statistics short history was that it goes back as a formal discipline, you know, 250 years or so. It was called inverse probability because around that era probability was developed sort of especially to explain gambling situations.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So here it's a little bit, it's somewhere between math and science and technology. It's somewhere in that convex hole. It's a set of principles that allow you to make inferences that have got some reason to be believed and also principles that allow you to make decisions where you can have some reason to believe you're not going to make errors. So all of that requires some assumptions about what do you mean by an error? What do you mean by, you know, the probabilities? But after you start making some of those assumptions, you're led to conclusions that yes, I can guarantee that if you do this in this way, your probability making error will be small. Your probability of continued to not make errors over time will be small. And probability you found something that's real will be small, will be high.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“And that is kind of the right intuition. But that intuition is not enough to understand kind of how to do it and why it works. But it does. It achieves one of our k squared and it has a mathematical structure and it still kind of, to this day, a lot of us are writing papers and trying to explore that and understand it. So there are lots of cool ideas in optimization, but just kind of using gradients, I think, is number one. That goes back 150 years. And then Nestrev, I think, has made a major contribution with this idea.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“That you will be within a ball of that size after K steps. Gradient descent in particular has a slower rate. It's one over K. So you could ask, is gradient descent actually, even though we know it's a good algorithm, is it the best algorithm? In the sense the answer is no. Well, not clear yet because one of our k-score is a lower bound. That's probably the best you can do. Gradient is 1 over k, but is there something better? And so I think it's a surprise to most that Nestrov discovered a new algorithm that has got two pieces to it. It uses two gradients and puts those together in a certain kind of obscure way. And the thing doesn't even move downhill all the time. It sometimes goes back uphill. And if you're a physicist, that kind of makes some sense. You're building up some momentum.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, nest stroke acceleration is just that suppose that we are going to use gradients to move around into space for the reasons I've alluded to. They're nice directions to move. And suppose that I tell you that you're only allowed to use gradients. You're not going to be allowed to this local person. It can only sense kind of the change in the surface. But I'm going to give you kind of a computer that's able to store all your previous gradients. And so you start to learn something about the surface. And I'm going to restrict you to maybe move in the direction of like linear span of all the gradients. So you can't kind of just move in some arbitrary direction. Now we have a well-defined mathematical complexity model. There's a certain classes of algorithms that can do that. In other words, that can't. And we can ask for certain kinds of surfaces. How fast can you get down to the optimum? So there's answers to these. So for a smooth convex function, there's an answer, which is one over the number of steps squared.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“I don't know the most, but let me just say that Nestorov's work on Nestorov acceleration to me is pretty surprising and pretty deep.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So classicity just means it's pretty unlikely that's going to happen. You're going to hit that point. So it's again not trivial to analyze, but especially in higher dimensions, also stochasticity, our intuition isn't very good about it, but it has properties that kind of are very appealing in high dimensions for law of large number of reasons. So it's all part of the mathematics to kind of it's what's fun to work in the field is that you get to understand this mathematics. But long story short, you know, partly empirically it was discovered stochastic gradient is very effective and theory kind of followed, I'd say, that. But I don't see that we're getting it clearly out of that.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so I can give you simple answers, but in some sense, again, it's kind of amazing. Stochasticity just particular features of a surface that could have hurt you if you were doing one thing deterministically won't hurt you because by chance there's very little chance that you would get hurt. And so here stochasticity is just kind of saves you from some of the particular features of stochastic. In fact, if you think about surfaces that are discontinuous in a first derivative, like an absolute value function, you will go down and hit that point where there's non-differentiability, right? And if you're running a deterministic armament, at that point, you can really do something bad.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Conditions generally. We know if you do this, we will give you a good guarantee. We don't have necessary conditions that it must be done a certain way in general.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Know, and my buck is only one unit, so I'm going to put all of it in the y axis, right? Why should I even take any of my strength, my step size, and put any of it in the x-axis because I'm getting less bang for my buck? That seems like a complete argument and it's wrong because the gradient direction is not to go along the y-axis. It's to take a little bit of the x-axis. And that to understand that, you have to know some math. So even a trivial so-called operator like the gradient is not trivial. And so exploiting its properties is still very, very important. Now we know that just pervading descent has got all kinds of problems. It gets stuck in many ways and it hadn't good dimension dependence and so on. So my own line of work recently has been about what kinds of stochasticity, how can we get dimension dependence, how can we do the theory of that? And we've come up pretty favorable results with certain kinds of stochasticity. We have sufficient”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“They have a lot of people who start to study them more deeply mathematically or kind of shocked about what they are and what they can do. I mean, think about it this way. If suppose that I tell you if you move along the x-axis, you get you go uphill in some objective by, you know, three units. Whereas if you move on the y-axis, you go uphill by seven units. Now I'm going to only allow you to move a certain unit distance. What are you going to do? Well, most people will say, I'm going to go along the y-axis. I'm getting the biggest bang for my buck.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Interesting ways. So there's a co design of the surface or the architecture and the algorithm. So if you just ask if we stay with the kind of architectures we have now, not just neural nets, but phase retrieval architectures or matrix completion architectures and so on. I think we've kind of come to a place where, yeah, a stochastic gradient algorithms are dominant and there are versions that are a little better than others. have more guarantees they're more robust and so on and there's ongoing research to kind of figure out which is the best algorithm for which situation but i think that that'll start to co-evolve that that'll put pressure on the actual architecture and so we shouldn't do it in this particular way we should do it in a different way because this other algorithm is now available if you do it in a different way so that that i can't really anticipate that coevolution process but i you know gradients are amazing mathematical objects”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Layers of arithmetic operations with a little bit of nonlinearity, that didn't come from neuroscience per se. I mean, maybe in the minds of some of the people working on it, they were thinking about brains, but they were arithmetic circuits in all kinds of fields, you know, computer science control theory and so on. And that layers of these could transform things in certain ways and that if it's smooth, maybe you could find parameter values is a sort of big discovery that it's able to work at this scale. I don't think that'll, we're stuck with that and we're certainly not stuck with that because we're understanding the brain.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“But the particular surface is coming from the particular generation of neural nets. I kind of suspect those will change. In 10 years, it will not be exactly those surfaces. There'll be some others that are, and optimization theory will help contribute to why other surfaces or why other algorithms.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Think from an optimization point of view that surface first of all, it's pretty smooth. And secondly, if it's over parameterized, there's kind of lots of paths down to reasonable optima. And so, kind of the getting downhill to the optimum is viewed as not as hard as you might have expected in high dimensions. The fact that some optimists tend to be really good ones and others not so good, and you tend to, it's not sometimes you find the good ones still needs explanation.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Stupid. But then I know that you were also anticipating me, so we're kind of going back and sort of mind. But there is then a first mover thing. And so those are different equilibria, right? And so just mathematically, yeah, these things have certain topologies, certain shapes that are like salad. What's it algorithmically or dynamically? How do you move towards them? How do you move away from things? So some of these questions have answers. They've been studied. Others do not, especially if it becomes stochastic, especially if there's large numbers of decentralized things. There's just, you know, young people get in this field who kind of think it's all done because we have TensorFlow. Well, no, these are all open problems. And they're really important and interesting. It's about strategic settings. How do I collect data? Suppose I don't know what you're going to do because I don't know you very well, right? Well, I got to collect data about you. So maybe I want to push you in a part of the space where I don't know much about you so I can get data. And then later I'll realize that you'll never go.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So, classically, in game theory, you were trying to find Nash equilibrium and an algorithm at game theory, you were trying to find algorithms that would find them. And so you're trying to find saddle points. So that's literally what you're trying to do. But in the economist knows that Nash equilibria have their limitations. They are definitely not that explanatory in many situations. They're not what you really want. There's other kind of equilibria and there's names associated with these because they came from history with certain people working on them, but there will be new ones emerging. So, you know, one example is a Stackleberg equilibrium. So, you know, Nash, you and I are both playing this game against each other or for each other. Maybe it's cooperative. And we're both going to think it through, then we're going to decide and we're going to do our thing simultaneously. And a Stackleberg, no, I'm going to be the first mover. I'm going to make a move. You're going to look at my move, and then you're going to make yours. Now, since I know you're going to look at my move, I anticipate what you're going to do. And so I don't do something.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Oh, not optimization. That's just game theory. So there's all kinds of different equilibria in game theory. And some of them are highly explanatory behavior. They're not attempting to be algorithmic. They're just trying to say, if you happen to be at this equilibrium, you would see certain kind of behavior. And we see that in real life. That's what an economist wants to do, especially a behavioral economist in continuous differential game theory. You're in continuous spaces, some of the...”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So, and autos could be embedded inside of an overall market. And game theory is very, very broad. It is often studied very narrowly for certain kinds of problems. But it's roughly speaking just the, I don't know what you're going to do. So I kind of anticipate that a little bit. And you anticipate what I'm anticipating. And we kind of go back and forth in our own minds. We run kind of thought experiments.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source
“He's all blended together, and a system designer thinking about how to build an incentivized system will have a blend of all these things. So a particle in a potential well is optimizing a functional called Lagrangian, right? The particle doesn't know that. There's no algorithm running that does that. It just happens. So it's a description mathematically of something that helps us understand as analysts what's happening, right? And so the same will happen when we talk about mixtures of humans and computers and markets and so on and so forth. There'll be certain principles that allow us to understand what's happening and whether or not the actual algorithms are being used by any sense is not clear. Now at some point I may have set up a multi-agent or market kind of system and I'm now thinking about an individual agent in that system and they're asked to do some task and they're incentivized in some way. They get certain signals and they have some utility. Maybe what they will do at that point is they just won't know the answer. They may have to optimize to find an answer.”
2020-02-24 · Lex Fridman Podcast · #74 – Michael I. Jordan: Machine Learning, Recommender Systems, and the Future of AI · IDENTIFIED FROM THE TRANSCRIPT · source