YouSaid · the spoken record
Cristos Goodrow
- lines on the record
- 81
- first
- 2020-01-25
- most recent
- 2020-01-25
- sittings or episodes
- 1
- sources
- podcast
Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections
“And if you were to ask me the next day, are you glad that you watched that show on TV last night? I'd say, yeah, I wish I would have gone to bed or read a book or almost anything else, really. And so that's why some people got the idea a few years ago to try to survey users afterwards. So we get feedback data from those surveys and then use that in the machine learning system to try to not just predict what you're going to click on right now, what you might watch for a while, but what when we ask you tomorrow, you'll give four or five stars to.”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“But we both know that things can get a lot of views and not really be that high quality, especially if people are clicking on something and then immediately realizing that it's not that great and abandoning it. And that's why we moved from views to thinking about the amount of time people spend watching it with the premise that like, you know, in some sense the time that someone spends watching a video is related to the value that they get from that video. It may not be perfectly related, but it has something to say about how much value they get. But even that's not good enough, right? Because I myself have spent time clicking through channels on television late at night and ended up watching under siege 2 for some reason I don't know.”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“That might be the one that people enjoy watching the most and watch to the end, or it might be the one that when we ask people the next day after they watched it, were they satisfied with it? And so we, especially in the realm of entertainment, have been trying to get at better and better measures of quality or satisfaction or enrichment since I came to YouTube. And we started with, well, The first approximation is the one that gets more views.”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, I mean, I think it depends initially on what sort of video we're talking about. So in the realm of, let's say, you mentioned politics and news. In that realm, Quality news or quality journalism relies on having a journalism department, right? Like you have to have actual journalists and fact checkers and people like that. And so in that situation and in others, maybe science or in medicine, quality has a lot to do with the authoritativeness and the credibility and the expertise of the people who make the video. Now, if you think about the other end of the spectrum, what is the highest quality prank video? Or what is the highest quality Minecraft video?”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“All your videos, right? That sort of represents you. You can also think of it as maybe a vector in the space of all the videos on YouTube And so, you know, now once you think of it as a vector in the space of all the videos on YouTube, then you can start to say, okay, well, which videos, which other vectors are close to me and to my vector? And that's one of the ways that we generate some diverse recommendations is because you're like, okay, well, these people seem to be close with respect to the videos they watch on YouTube. But here's a topic or a video that one of them has watched and enjoyed, but the other one hasn't that could be an opportunity to make a good recommendation.”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, we do provide the history of all the videos that you've watched. So, you can definitely search through that and look through it and search through it to see what it is that you've been watching on YouTube. We have actually. In various times, experimented with this sort of cluster idea, finding ways to demonstrate or show people what topics they've been interested in or what clusters they've watched from. It's interesting that you bring this up because Some sense, the way the recommendation system of YouTube sees a user is exactly as the history of all the videos they've watched on YouTube. And so you can think of yourself or any user on YouTube as DNA strand of”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“And how is it that you know that I speak both these two languages and put all the videos together? And it's just as sort of an outcome of this related graph that's created through collaborative filtering.”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“She was telling me about this, and I said, Well, can you give me an example of what problem do you think we have on YouTube with the recommendations? And so she said, well, I'm a researcher in the U.S. And when I'm looking for academic topics, I want to see them in English. And so she searched for one, found a video, and then looked at the watch next suggestions, and they were all in English. And so she said, oh, I see. YouTube must think that I speak only English. And so she said, now I'm actually originally from Turkey. And sometimes when I'm cooking, let's say I want to make some Baklava, I really like to watch videos that are in Turkish. And so she searched for a video about making the Baklava and then selected it and it was in Turkish. And the WatchNext recommendations were in Turkish. And she just couldn't believe how this was possible.”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“That's right. I remember a few years ago, I was talking to someone who was trying to propose that we do a research project concerning people who are bilingual. And this person was making this proposal based on the idea that YouTube could not possibly be good at recommending videos well to people who are bilingual. And so”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“Right? And it puts all the videos that are about sports together, and it puts most of the music videos together, and it puts all of these sorts of videos together just because that's sort of the way the people using YouTube behave.”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“Sure, it's just basically what we do is we observe which videos get watched close together by the same person. And if you observe that, and if you can imagine creating a graph where the videos that get watched close together by the most people are sort of very close to one another in this graph and videos that don't frequently watch close together by the same person or the same people are far apart, then you end up with this graph that we call the related graph that basically represents videos that are very similar or related in some way. And what's amazing about that is that it puts all the videos that are in the same language together, for instance. And we didn't even have to think about language. It just doesn't”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“Right, that's something that we can observe. And then as a result, make sure that that document would be retrieved for that query. Now, when you talk about what kind of videos would be recommended to watch next, that's something, again, we've been working on for many years. And probably the first. The first real attempt to do that well was to use collaborative filtering.”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, that's kind of you to say it didn't used to do a very good job But it's gotten better over the years. Even I observed that it's improved quite a bit. Those are two different situations. Like when you search for something, YouTube uses the best technology we can get from Google to make sure that the YouTube search system finds what someone's looking for. And of course, the very first things that one thinks about is, okay, well, does the word occur in the title? For instance, Know, but they're much more sophisticated things where we're mostly trying to do some syntactic match or maybe a semantic match based on words that we can add to the document itself. For instance, maybe is this video watched a lot after this query?”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“On improving the machine learning systems to remove and reduce unfair biases when it goes against or has involved some protected class, for instance.”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“We ask them to have a bias towards demonstration of expertise or credibility or authoritativeness. But there are other biases that we want to make sure to try to remove. And there's many techniques for doing this. One of them is you send the same thing to be reviewed to many people. And so that's one technique. Another is that you make sure that the people that are doing these sorts of tasks are from different backgrounds and different areas of the United States or of the world. But then even with all of that, it's possible for certain kinds of what we would call unfair biases to creep into machine learning systems, primarily, as you said, because maybe the training data itself comes in a biased way. And so we also have worked very hard on”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, we take steps to try to overcome these kinds of biases or biases that we think would be problematic. So for instance, we ask people to have a bias toward scientific consensus. That's something that we instruct them to do”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“And just as you said, a lot of it comes down to people at YouTube spending a lot of time trying to figure out what are the right policies, what are the outcomes based on those policies. Are they the kinds of things we want to see? And then once we kind of get an agreement or build some consensus around what the policies are, well, then we've got to find a way to implement those policies across all of YouTube. And that's where both the human beings, we call them evaluators or reviewers come into play to help us with that. And then once we get a lot of training data from them, then we apply the machine learning techniques to take it even further.”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“Which are the videos that are misinformation or borderline policy violations? Well, the first thing we need to do is get human beings to make decisions about which of those videos are in which category. And then we use that data and basically take that information that's determined and governed by humans and extrapolated or apply it to the entire set of billions of YouTube videos. And we couldn't get to all the videos on YouTube well without the humans and we couldn't use the humans to get to all the videos of YouTube. So there's no world in which you have only one or the other of these things.”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, my own experience has demonstrated that you need both of those things. Algorithms, I mean, you're familiar with machine learning algorithms, and the thing they need most is data. And the data is generated by humans. And so, for instance, when we're building a system to try to figure out”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“I think that that overall is our whole project here at YouTube. We fundamentally believe, and I personally believe very much, that YouTube can be great. It's been great for my kids. I think it can be great for society. But it's absolutely critical that we get this responsibility part right. And that's why it's our top priority Susan Wajiski, who's the CEO of YouTube, says something that I personally find very inspiring, which is that we want to do our jobs today in a manner so that people 20 and 30 years from now will look back and say, you know, YouTube, they really figured this out. They really found a way to strike the right balance between the openness and the value that the openness has and also making sure that we are meeting our responsibility to users in society.”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“I understand that anyone who uploads YouTube videos has to become resilient to a certain amount of meanness. Like I've heard that from many creators. And We are trying in various ways, comment ranking, allowing certain features to block people, to reduce or make that meanness or that trolling behavior less effective on YouTube. And so I mean, it's very important, but it's something that we're going to keep having to work on. And as we improve it, maybe we'll get to a point where people don't have to suffer this sort of meanness when they upload YouTube videos. I hope we do. You know, but it just does seem to be something that you have to be able to deal with as a YouTube creator nowadays.”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“And so we're not trying to settle that or choose a side or anything like that. What we're trying to do is make sure that the people who are expressing those point of view and offering those positions are authoritative and credible.”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“However, as you pointed out, wherever you draw the line, there's going to be a borderline. And in that borderline area, we are going to maybe not remove videos, but we will try to reduce the recommendations of them or the proliferation of them by demoting them. And then alternatively, in those situations, try to raise what we would call authoritative or credible sources of information. So we're not trying to, I mean, you mentioned Ayn Rand and communism. You know, those are two valid points of view that people are going to debate and discuss. And of course, people who believe in one or the other of those things are going to try to persuade other people to their point of view.”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, the responsibility to get this right is our top priority. And the first comes down to making sure that we have good, clear rules of the road, right? Just because we have freedom of speech doesn't mean that you can literally say anything, right? Like we as a society have accepted certain restrictions on our freedom of speech. There are things like libel laws and things like that. And so where we can draw a clear line, we do, and we continue to evolve that line over time.”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“Right, there's a higher likelihood that a person who's watching science would like jazz than the person watching science would like, I don't know, backyard railroads or something else, right? And so we can try to measure these likelihoods and use that to make the best recommendation we can.”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“Taking a bigger leap is about, I mean, the mechanisms we use for that is we basically cluster videos and channels together, mostly videos. We do almost everything at the video level. And so we'll make some kind of a cluster via some embedding process. And then measure, what is the likelihood that users who watch one cluster might also watch another cluster that's very distinct. So we may come to find that people who watch science videos also like jazz. This is possible, right? And so because of that relationship that we've identified through the embeddings and then the measurement of the people who watch both, we might recommend a jazz video. In a while”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“So, you're a person who's watching some math channels and you might be interested in some other science or math channels. So like you mentioned, the first kind of diversity is just show you some things from other channels that are related, but not just three blue, one brown channel, throw in a couple others. So that's the maybe the first kind of diversity that we started with many, many years ago.”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“I had a random video. I could just randomly select a video from the billions that we have. It's likely not to even be in your language. So the likelihood that you would watch it and develop a new interest is very, very low. And so what you want to do when you're trying to increase diversity is find something that is not too similar to the things that you've watched, but also something that you might be likely to watch. And that balance, finding that spot between those two things is quite challenging”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“Absolutely. That's right. I mean, if YouTube is going to continue to enrich people's lives, then it has to grow with them. And people's interests change over time. I think we've been working on this problem, and I'll just say it broadly as like how to introduce diversity and introduce people who are watching one thing to something else they might like. We've been working on that problem all the eight years I've been at YouTube. It's a hard problem because, I mean, of course, it's trivial to introduce diversity that doesn't help.”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“He got through his linear algebra class because of a channel called Three Blue One Brown, which You know, helps you understand linear algebra, but in a way that would be very hard for anyone to do on a whiteboard or a chalkboard. And so I think that Those experiences, from my point of view, were very good, and so I can imagine really good trajectories through YouTube. Yes.”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source
“I think there are some great trajectories through YouTube videos, but I wouldn't recommend that anyone spend all of their waking hours or all of their hours watching YouTube. I mean, I think about the fact that YouTube has been really great for my kids, for instance. My oldest daughter, You know, she's been watching YouTube for several years. She watches. Tyler Oakley and the Vlogbrothers. And I know that it's had a very profound and positive impact on her character. And my younger daughter, she's a ballerina, and her teachers tell her that YouTube is a huge advantage for her because she can practice a routine and watch like professional dancers do that same routine and stop it and back it up and rewind and all that stuff, right? It's been really good for them. And then even my son is a sophomore in college.”
2020-01-25 · Lex Fridman Podcast · Cristos Goodrow: YouTube Algorithm · IDENTIFIED FROM THE TRANSCRIPT · source