YouSaid · the spoken record
Kelvin Guu
- lines on the record
- 41
- first
- 2023-05-04
- most recent
- 2023-05-04
- sittings or episodes
- 1
- sources
- podcast
Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections
“Work for you, someone still needs to validate that. And that may still require a great deal of technical skill unless you prompt a large language model to validate things as well. But then who's validating the validator? So at some point, technical skill will still be needed, whether it will be as broadly needed, it's anyone's guess.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“I love that question. I'm sure many people have asked that. I feel that there's maybe been a shift in what kind of skill is valuable. So at a certain earlier point in time, having technical proficiency was a huge differentiating factor. And if you didn't have that, you just couldn't pursue certain ideas. Whereas now more and more, the technical proficiency part is we're being helped out by large language models to, you know, write in programming languages we don't even know. And so it seems like the differentiating factor now shifts more to problem formulation or creativity in identifying problems and being able to frame them in a way that you can then reduce them to a technical problem. So I sure definitely don't know what will be happening 20 or 10 years from now, but being creative and solving different kinds of problems will always still be useful. Another thing that has come up in some discussions is that even if the large language model is autopiloting a lot of”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Can provide so much motivating use cases. And even if you come to Google or something, there's so many things there to look into. That's one thing. The other thing I would say is there's sort of also folks who ask how much technical knowledge is still important at this stage for doing machine learning? Like has it become kind of an era of just sort of prompting language models to do different things or just curating data and scaling things up? And I would say there are still, well, obviously, it's great if you can build valuable applications using a simple skill set. And I think there's an amazing amount of creativity that can go into that. But if you're also looking for something that is more of a technical challenge, there's still so many unsolved problems in terms of understanding how these models work. And ultimately, we do need to get to the bottom of that to improve things like safety and predictability.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Oh, sure. Yeah. I think at this time in particular, I encounter folks who fall into like two different camps. Sometimes there are folks who will say, oh my gosh, like everything is happening. And even if I sit here and do nothing, it will all get done. And I just don't know what to do. And then there are others who feel that, oh, there's too many things to do. I cannot get to all of them in time even. And I think there's a couple different answers. One is if you move the goalpost for yourself, so a lot of researchers, they kind of had set what they felt were ambitious goals, but then given all of the recent progress, those goals now need to become more ambitious. So if you think farther out, you will find that there are so many things that are not solved, like making agents more autonomous or safe or understanding which training examples cause a language model to do something. There's a lot of headroom there. And especially, I think now if you pay attention to what's happening in the product world.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so I think there's a strong incentive to make that happen. So, the folks who are providing large language models, they want to make their approaches as easy to use as possible. And so anything that can go into prompting, it seems to me that people will try to do it because that gives the broadest reach. You don't want to just say, oh, you have to fine-tune this and we're not going to help you in any way. So I expect people will try to push that as far as possible. And you could even say that to some extent, just following on what we did for instruction tuning, you can find tune the model to be more and more responsive to different types of instruction. So if you can anticipate that someone is going to say, I want you to follow these family values, well, then at training time, we can prepare for that. And so it's a game of how much can we anticipate that people will want to adapt.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Again, I think it depends a lot on the application. So we could think from the perspective of an individual, like if someday, for example, people had their own in-home language model, which of course doesn't exist right now, you might even say like, oh, these are our family values. We want the language model to have those values. Would that be through just writing down your kind of family values on an instruction prompt and just saying, please follow these everywhere? Or is it going to be through repeated feedback where you talk to the language model and you say, no, I don't think you should say that. not clear to me how that will go.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so I'm very interested in recipes that allow people to adapt things easily. And it's not yet clear what the right interface is. So training has always been challenging for many reasons. I think to do training well, yes, there are scripts and architectures that are ready to go, but data quality is a really big aspect of good training. And so there is a question of will that be the most common format because Curating data is not easy. At the scale of maybe 10 or 30 examples, it is easy. And maybe that is a sweet spot for a lot of users.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Oh, yeah, to be clear, they are useful for many applications. In many cases, a database you want that to be your source of ground truth, and that's perfect. If you want broad coverage, you ideally want a mechanism that's more expansive than that. One thing we've noticed is that it's very hard to get a representation that is 100% canonicalized in the sense that information is only represented in one place and not anywhere else. So with text, for example, A single fact is in multiple places. But if you want to edit a fact, you need to go to all the places where that fact is mentioned and change it. So there always seems to be this trade-off between centralization and coverage. And this is kind of why I mentioned the earlier model surgery work, because I think that's some of the first work to try to keep all of the coverage while also allowing some degree of centralization. If I were kind of advising a student or something on looking into knowledge representation,”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“So when I started doing research, I was actually very involved in knowledge bases. I thought those were going to be the future because they have this wonderful canonical representation of things. You can do reasoning over them.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Sure, yeah. So at the moment, I think there's this paradigm where you sit in front of a chat interface and if you have a problem, you think really hard about all the information and context around it and you try to boil it down into this wonderful prompt for the model. And that actually requires a lot of brain power. So there are a lot of cases where people will just not think to use the model because of that extra barrier. And I think there's clearly an ambition in the work that we see publicly to make these models more available and more context. So you could imagine just being in the browser and trying to get something done. And after 20 minutes, you just ask your model, like, why can't I get this done? Or how do I solve this? And it should ideally look at everything you've been doing up to this point and infer your intent and just help address that. That seems like a pretty open question that isn't fully addressed.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I think that's a good example that a human at some point will introduce some kind of introspection step like, wait a minute, I've been counting for 10 hours. And I think that's often missing as well. Like people try to patch that over with chain of thought reasoning. So they'll do some reasoning. Then they'll say, is this reasoning right and repeat that loop? But does it have to be as explicit as it is right now? I don't think so. I think we have associative mechanisms that ping us after a certain amount of time on these things.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I also think, though, that there's this kind of memory consolidation aspect of human learning where there's a deliberate pruning of what you've thought about that we don't have any analog for that, at least in the early systems we're seeing right now.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, that's a really great question. One that I was thinking about recently, and this is again very speculative, but when we think about autonomous agents, we worry often about their capacity to do something that is irreversible, to get themselves into a state that they can't get themselves out of. And in human behavior, there are certain things that we have an instinct to avoid, and that is usually driven by an emotion, like fear or uncertainty or anxiety. And funnily enough, in the reinforcement learning literature, there are people who have proposed research concepts that they call fear, where the model learns to identify states that it can't back out of. And I suspect something along those lines may also be helpful. They may also introduce the kind of corresponding issues with fear that humans have as well. And that's open questions.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Maybe just to run with that example a little bit. So you can kind of specify all the things that you want about your vacation. But there's also another aspect of this where if you had an actual travel planner, they would gradually learn over time your preferences, not through your explicit instructions, but just your feedback, your responses. And that kind of learning loop, I think, doesn't appear yet in existing methods. I think this becomes even more apparent when an autonomous or semi-autonomous agent needs to interact with more complex tools. So simple tools like maybe issuing a search query, there are very short interaction and it's almost stateless. You just put something in, you get something out. But if you're actually operating anything of significant complexity, there's a learning curve that anyone has to go through. If somebody sees a particular spreadsheet that an organization is using for certain purposes, you need to implicitly understand how that spreadsheet is being used, what is okay.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“I think that it is a bit architectural right now. So just to give an analogy, like in 2018, when question answering systems or reading comprehension systems got really good, everybody tried to reduce a lot of problems into question answering. And there was just this dash to just throw that into different components. And now I think now that we know that instruction following works.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“We do is also instinctive. And those instincts are developed over time. So just to give a psychology analogy, there's this concept of chunking where people have shown that a human cannot really easily remember more than seven unique items at a time. But if those seven unique items are then broken into chunks that they can attach some sort of metaphorical or otherwise significance to, they can remember much longer sequences. And I think the same thing kind of happens with humans where Initially we do something through explicit reasoning, but eventually we distill that down into instincts. That's something that seems to be missing from these autonomous agents right now. Everything is explicit reasoning. And at some point that might not scale or that might be brittle. So that seems like a very exciting area.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Is a memory mechanism, much like the retrieve augmentation we've described, where the agent has the ability to store its memories either in text or in a vector store. And I think more research is definitely going to be needed there. It's a very ambitious goal, but this way of decomposing the problem leaves, I think, potentially many gaps. So for example, let's say there's an agent pursuing a task and it breaks that task down into sub-goals that defines for itself. It will then write one of those sub-goals down into its metaphorical sticky note that decides what to do. And if one of those sub-goals is wrong, suddenly that is a canonical goal in the model. And how does it realize to reverse that decision? You could argue that, oh, we just need more prompting to get the model to think about every aspect of the problem. Us as humans, we certainly do prompt ourselves quite a bit to think about a problem from different angles. But a lot of”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I think since you say kind of true machine intelligence, then we're kind of going at the full, I presume, sort of AGI level goal. And I think one thing that's very obviously in the public discourse right now is increasingly autonomous agents. I should say, by the way, everything that I'm discussing here, it's all personal opinions and I'll be deriving everything from things I've seen in the public discourse. nothing related to any of the work that I'm doing at Google. But I do see this desire for autonomous agents. And it seems like the going approach right now is kind of inspired by the Turing machine. So we have these models that can operate over a relatively small context with a prompt. And folks are trying to take long horizon tasks and reduce them into these smaller promptable tasks. And the component that they're using to bridge the short time horizon into the long time horizon.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“That the weight matrices inside large language models can be thought of as lookup tables to some extent. So we know that large language models perform a large number of matrix vector multiplications. And the matrix is the lookup table and the vector that is multiplying the matrix can be thought of as the query to that lookup table. And so by updating that matrix, they're able to change the knowledge. I feel that this is a very exciting area for research because it provides a different kind of modularity from retrieval augmented models or mixture of experts, one that actually allows a kind of generalization that's very interesting.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so there's been a certain line of work that I've been really impressed with, which is what I'll call model surgery research. This is on editing model parameters to change their knowledge of certain facts. So there was a paper called Rome out of, I think, the, I think it was a lab in MIT. And the basic thing that they showed they can do is let's say the model believes that the Eiffel Tower is in Paris, as it is indeed. They show that they're able to edit certain model weights to make the model believe that the Eiffel Tower is in Rome. And not only are they able to do this, but any sort of facts related to the Eiffel Tower or related to Rome are also updated. So the model doesn't just remember this one fact, but propagates that knowledge throughout the rest of the network. So for example, if you ask, what is the tallest structure in Rome after making this update? The model will also say Eiffel Tower. And this is based on a bit of theoretical work showing.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Linux Terminal 1 did impress me when that first happened. I would have to say also sometimes their ability to do visual reasoning in pure text is also quite surprising to me. I've seen language models use SVG notation to draw pictures and that just seems very emergent.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“And the question that always comes up for us is Was there something in the training set that actually looked like that? And we just can't find it without being able to track that down. We'll never know if these models are generalizing or just kind of cleverly patching together what they know.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“See if the behavior changed. That's obviously too expensive for most people to do. So, what we come up with is a very lightweight model that simulates the training process. It observes a few training runs you've done, tries to estimate the effect of each of the individual examples, and then you as a developer can sit down and say, imagine that I had done this training run, what would the result be? Any sort of simulation that you do is always going to come with approximation errors. And so this is still very early work on modeling these things. But I think it would have very beneficial effects if we could figure this out. One of the best use cases is if you know what training data is valuable, you can go out and get more of it. It would also help us understand what these large language models are really capable of. So we've seen impressive feeds from large language models recently like simulating a Linux terminal or being able to draw like a unicorn in LaTeX or something like this.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so this influence paper is part of an emerging area of research called training data attribution. So the problem there is not specific to large language models. It's really general to any machine learning model. The question that it tries to answer is given that my model has done a certain thing, which training examples taught it to do that thing. This question goes back all the way to statistics and linear models. But for large language models, it's especially difficult to answer the question because of the kind of long training process that it goes through and the many forms of generalization that these models have. So in the Sinfluence paper, we take a somewhat new approach to the problem. So we say, okay, how would you really justify that this training example was important for doing a certain thing? Well, if you had infinite compute, what you would do is you would take that example and remove it from the training set and retrain your entire model.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Have a densely parametrized large model that is frozen. The parameters are frozen, but the prompt that you give it is not natural language, but rather the vectorized embedded form of that natural language. And people use gradient descent on that vectorized form to adapt the model's behavior. So you can produce a prompt that's good for this week. Next week, you need something else. You can update that prompt. So the large model is not changing, but a small part of it is updating over time. And you have the ability to plug in and plug out different prompts as needed.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so when you say updating model weights in real time, I think I'm kind of thinking of continuous learning methods. Those I initially thought would have been more popular in kind of production grade settings, but they come with a maintenance cost, which is if you have something updating live, you don't have the opportunity to validate and check that everything is going well as easily. And so I've been kind of surprised by the fact that they're not actually as popular as I thought that a weekly release cycle or a monthly release cycle is often fine. So for that reason, I personally haven't been looking into that line of research as much. But the idea of learning over time and continuously building up more knowledge, I think that's quite relevant. There are a variety of techniques out there for doing that. One example might be prompt tuning. So just to give a little bit of background on that, prompt tuning is where you”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Because that worked out, things like instruct GPT came out next, followed by things like ChatGPT. And this sort of adaptation is just opens up the space of who can use these models much more rapidly. Suddenly, you don't need specialized fine-tuning data, et cetera. But there's still a new frontier of tasks that you can't just get out of prompting. To give an example, like many people are familiar with the problem of large language model hallucinations, they will sometimes say things that are not well supported by external sources. And you can't just tell the model, please don't make things up and expect that to just solve the problem. There are so many other things you can say with instructions, but they won't be followed. And that's where I think existing adaptation techniques like fine-tuning and reinforcement learning are still relevant. It's also still relevant for perhaps personalization or other kind of soft traits that are”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Sure, yeah. So I'll describe some of the work we did in a paper called FLAN, which was one of the early papers to show that you could train large language models to follow instructions. So the high-level idea is you start with a pre-trained language model and you collect maybe a hundred different tasks. These could be sentiment classification, question answering, summarization. And for each of those tasks, you annotate each example within a instruction saying what it is that needs to get done and you perform multitask training over all those tasks. And that had been done many times before in other research as well. But what we were able to show was that if you train on 100 tasks, if we then show the model a new 101th task, it will be able to adapt to that without having seen that particular task before. And that was very surprising at the time. Many people thought we would need thousands of tasks to make that happen.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I think modularity is going to potentially have to happen due to the different, like the sort of way these models are developed. So it's not always going to be the case that any organization can invest the resources to build a large language model from scratch. So they're going to be wanting to build on top of something that exists. And they want to spend minimal effort to adapt. And that's where the benefit of modularity really comes in. I think it's also interesting to discuss the different ways people are adapting language models now. If we look at the trend over time as late as in 2017, people were training specialized models for specialized tasks. So you might define your own special neural network architecture just for sentiment classification. And then we switched over to having a pre-trained model, everybody sharing the same architecture, but different fine-tuning data. And then very recently, we have this sort of instruction following setup, which I was very thankful to have been involved in some of that.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“One approach that's very granular and the other that takes broader strokes and the kinds of use cases where one or the other is best actually differ as well. So if you just want to get very precise factoid information like what is my Wi-Fi password retrieval augmentation is going to be very good. But if, for example, you're trying to adapt a language model to a new enterprise and they have some kind of specialized language in their code base that nobody else uses, then just retrieving any single piece of code from that is not going to teach you how to use the language. Whereas an expert that's been trained on it is going to be able to adapt more quickly. So I think both of these approaches are going to be useful in different places.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah. And I would say Eliad, you mentioned mixture of expert models earlier. And I think there's an interesting overlap between mixture of expert models and retrieval augmented models. So there's been a recent mixture of expert approach, which is very easy to describe. So I'll be describing a paper called Branch Train and Merge. You take your corpus of pre-training data and you just split it up into different subcorporations. And you just train a language model on each one of those. And then when you want to use the language model at inference time, you sort of use some kind of routing mechanism or weighted voting mechanism to choose among those experts. And so in that case, each of the experts is responsible for part of the document corpus. And if you think about a retrieval augmented model, instead of having that intermediate layer of experts, you just going to individual documents. And so it's both of these approaches try to do the same thing, but they don't partition the space in the same level of granularity.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so I think there are clearly a lot of different tasks where we're seeing you don't really need retrieval. So for example, anything that's common knowledge on the web more and more we're seeing that if you build a dense model and that information is represented in enough quantity, these things with scaling improve. So it's actually been a case where I think more and more benefits of a retrieve augmented model are for modularity for personal information that you might not want in the main model for adapting to say like an enterprise customer that has special information. I think that's a much more compelling application at this point.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“And if they don't help you fill the blank better, then learn that these are not useful and don't retrieve them going forward.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Information is through the cross attention. And there are multiple different ways to train the cross attention to kind of use that information. And in the realm paper in particular, we tried to learn that from the language modeling task itself. So I guess for our listeners, many of you are probably familiar with the fact that language models predict the next token. So given a sequence of existing tokens, you predict the next one. At the time of the realm paper, we were doing something called mask language modeling, which was popularized by birds. So a very similar variation instead of only seeing kind of the tokens leading up to your current token from the left, you would also, we would instead take a whole piece of text and just blank out a few words and train the model to fill it back in. And the way Realm was being trained in this method was we would say, okay, you can go and you can retrieve some documents. And if those documents help you fill the blank better, learn that those are useful.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“The basic idea, if we were to describe it, is let's say the user provides some sort of input to the model. I'm going to describe the inference time way of using it and then the pre-training that was involved as well. So the user provides some sort of input to the model. The model is then able to encode, embed that information into a dense vector. So this is a vector that will be situated in a larger vector space where we've also embedded other documents. So any other corpora we have on the web are converted into dense vectors as well. And a nearest neighbor search method is used to find the closest documents to the embedding of the input. And once you have those documents, you can then encode those as well in some form that allows the model to then use cross-attention over those documents. And based on that, it's then able to make a prediction. So the way in which the model depends on that.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“That's kind of what brought us to that in the first place. And since then, we've encountered many interesting challenges on top of that idea that I would say even in the sort of systems you see today, these tool using models that issue Google searches or provide citations, they still face some challenges in terms of fulfilling those original promises.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Sure, yeah. So there are a few different things that can bring people to this retrieval augmented modeling literature. One of them was kind of our original goal, which was to increase the memorization capacity of these models. A second goal that you might come to this from is for modularity. So you might imagine that you have different data sources and you'd like to be able to swap one in or take one out the same way you could do with a database. There are just so many business applications where that's helpful. A third is anytime you're dealing with a very timely application. So there's new information arriving daily about, say, a sports team or any other type of event, you really want to be able to incorporate that quickly without retraining. So these are some of the common ways that people arrive in the space. And it's a very natural thing, I think, now to think about, well, I don't want to retrain and yet I've got all this information out there and it's human interpretable. It's text. How can I bring that in?”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Now, but that also inspired me to and my colleagues to start thinking about retrieval augmented models and getting even more knowledge into them. And I'm sure we'll be talking about that more here.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I would see it. Google, this was really just the dream place for me to deepen my focus on building tools that help people find information better. And at the time that I joined, it was also just an extremely productive period where lots of new ideas were coming out. The group that I'm part of right now, which is now part of Google Brain at the time, was looking into very, at the time early ideas on pre-training language models, which eventually became BERT. And that work just opened up a lot of frontiers for me both in terms of product work at Google and inspiring new research ideas. I think one of the things that became very apparent early on when playing with Bert was unlike all the prior generations of models, it had a large amount of world knowledge that we didn't deliberately encode into it. It wasn't in, you know, the fine tuning data. It was all within pre-training. And that has just many, I think, very obvious use cases.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I actually had always wanted to get into tools that would help people learn more easily and find information more quickly. My motivation for going into the math was really to build a good foundation for getting into many of those deeper questions. And I would say that also motivated my venturing into statistics. So my PhD was actually in the statistics department, but I quickly then migrated over to the NLP group at Stanford where I just learned a lot and had a great time.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source
“I'm very interested in recipes that allow people to adapt things easily. It's not yet clear what the right interface is if someday, for example, people had their own in-home language model, which of course doesn't exist right now. You might even say like, oh, these are our family values. We want the language model to have those values. Would that be through just writing down your kind of family values on an instruction prompt and just saying, please follow these everywhere? Or is it going to be through repeated feedback where you talk to the language model and you say, no, I don't think you should say that. It is not clear to me how that will go.”
2023-05-04 · No Priors · Personalizing AI Models with Kelvin Guu, Senior Staff Research Scientist, Google Brain · IDENTIFIED FROM THE TRANSCRIPT · source