YouSaid · the spoken record

Douwe Kiela

lines on the record: 46
first: 2023-06-30
most recent: 2023-06-30
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“So, if all the stars align OpenAI and Anthropic and all of these places, they had this great first generation technology. So they're kind of like the Lycos and Alta Vista search engines. And the technology we have is more like PageRank and that would make us the Google of language models with the right technology at the right time and with the right execution. So that's what I would hope for.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“I think when I was young, I maybe put ambition before people sometimes. I learned this the hard way, I think, where I just didn't have enough empathy for the people I worked with. And as I grew older and more mature, I realized more and more just that it's really all about people and working with fantastic people and doing cool things together and changing the world together. So I'm very happy to have learned that lesson because that makes me much better at my job right now.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So, despite Nick Bostrom's book, I still think that superintelligence is actually very ill-defined. In many ways, we have already achieved superintelligence. And so in the 50s, we achieved mathematical superintelligence. So computers in the 50s were already better calculating stuff than humans. I don't think that that really is a well-formed question. If you're asking about AGI, and I think AGI itself, a lot of people, this is a mistake I made where I thought AGI kind of meant artificial consciousness or something like that, which also doesn't really have a meaning. But if you look at how open AI and Anthropic and these places define AGI, systems achieving capabilities that allow them to effectively do the work of humans for the majority of economically valuable human tasks, then we're not that far.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“The strongest belief is that I really underestimated how important scale is in artificial intelligence. And I think this is really one of the things that OpenAI has excelled at. If you throw an order of magnitude more compute and data at AI systems, then they just become much, much better. And if you keep that scaling up, you have these scaling laws that we know about now. I really underestimated this. And for a long time, I was just saying like, oh, yeah, look at these silly open AI researchers. They're just scaling things. They're not inventing new algorithms. That's not cool. And I was very, very wrong.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“I'm very happy that I studied philosophy. So, philosophy is really about conceptualizing anything and any arbitrary level of abstraction and that ability you can use anywhere. So for AI in particular, I think that philosophy and AI are an interesting combination because philosophy is about the stuff that you can't pretty do science about yet. So at some point, the things that people are philosophizing about now, they become scientific questions that just have answers or hypotheses. And then they're no longer philosophy. So natural philosophy that used to be a thing we now call that physics and mathematics. And I think with AI, there are lots of questions that we still don't even really know how to ask yet. And philosophy is great for thinking about those kinds of questions.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, totally. I don't know if those are going to be new businesses or existing incumbents. So the hyperscalers are also trying to play that role. There's just so much demand right now for AI in any kind of enterprise. And it's still very hard to get it right. So a lot of these companies are just looking for help. So there are just opportunities there.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“I think there's way too much hype. It would be good if the community at least acknowledges that and tries to really pay attention to the things that matter, like having technology that actually works and not just jumping on the next hype train and there's an auto GPT thing that is going to change the world but doesn't actually work. So there's a lot of debate right now that is just driven by Twitter and just sound bites and quotes like Ahmad, where I think it would be better to be a bit deeper and think a bit more carefully about what we're doing.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“No, absolutely not. The valley is kind of a dangerous bubble in a way where there's this giant echo chamber happening. And I think if you look at a lot of great AI companies, they're not in the valley and they don't have to be here. Maybe they should have an office here because there are great universities to recruit from and things like that, but I really don't see a reason why you would have to be here.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So, I think otters are underestimating how early it still is. AI fuels like we've made so much progress that is very hard to enter the market right now. And I think that that is just not true. It's still very early innings. We haven't settled on a lot of things that need to be solved before we can really have this technology be ready. It's still very early, me realizing that gives me a competitive advantage, hopefully.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“I'm still very curious to see what Apple will do. They had this great vision of having Siri on your phone and things like that. So that's like one of the first personal assistants. So if that could be a super powerful language model, then that could do very interesting things. But so far, I haven't really seen interesting things coming out of Apple.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“I've been very impressed actually by how Microsoft has managed to turn everything around by the strategically collaborating with a better AI lab in the shape of open AI. And they've just really turned that into this narrative where Microsoft is an AI leader and they really weren't an AI leader even a few years ago. So that's been impressive.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Not so much. I think some of the rounds were pretty big, but I think it's also justified just because this stuff is really going to change the world. So one bet, if it's right, has massive payoff. There's this narrative, I think, in the VC community that there are these crazy rounds happening, but I think they're happening for a good reason. So I haven't really seen any companies come by where I was like, wow, like why are they getting this much money? There's a few of them where I thought they really have to live up to massive expectations now and they need to actually start making real revenue now at some point there's going to be a disillusionment with the technology and then funding might dry up and then these places are really in trouble”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so traditionally in cloud deployments, you talk about the data plane and the control plane. And so control really is, okay, you're a startup or a company and you want to be able to deploy models to your customers VPC virtual private cloud, but you don't want any of their data leaving their VPC because it's their data. So that's the separation between the data plane and the control plane. So in this new setting where we have language models, there's this model plane and it's kind of unclear where to put it. So you could put the model inside the customer's VPC and then you get full data privacy basically, but you have no control over what that model is doing at all. You get no feedback, you get no learning. So you want to find interesting hybrids where you can respect data privacy, keep the data plane inside the customer's VPC, but put the model somewhere else. So you can do that if you have a decoupling between the retrieval part and the generative part, which is what we are building.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, absolutely. That's really one of the big questions, right? And that's why what we're building has this very clean separation between the data plane and the model plane, because then you have much more control over where the data goes.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“It will be gradual, I think. And a lot of work is right now actually going into finding the right use cases for this technology because people are now starting to think that they can use GPT-4 for anything. That's just not true. People are trying to experiment with the right use cases for different types of models.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“I think it's already happening. I was at an exec event at Google earlier this week, and there were all of these C-level folks from all companies across the world, and they were all talking about how they're using AI and everybody's experimenting with it, and it's starting to make it into production already in various places. I don't think we have to wait for a year. It's already happening, but there are just some big hurdles that need to be overcome and they will be overcome very quickly.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Amanda is really great at speaking in quotes, by the way. That's another great quote. Yeah, I think the tidal wave is coming. There are just big problems that we have to overcome. And these are the things I just talked about, right? So hallucination, attribution, compliance, up-to-dateness, data privacy, latency. And I think the whole field is moving in this direction of just making everything ready for enterprise usage. This is going to happen.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“The European Union just has a tendency of really killing innovation because they think they can lead through regulation. And that's really the only thing that they're really good at.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Accept the incumbents, and those are the big tech companies already lobbying for regulation anyway and spreading fears of AI existential risk and things like that.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“We have to invest a lot in educating regulators. The AI community has been terrible at this. And the broader populace just needs to understand much better what AI is and what it can do and what it can't do. It's been slightly self-interest driven, I think, in that a lot of folks in AI have just wanted to keep the technology for themselves. And that's why they haven't really invested in educating the rest of society. That's really a huge issue. A bit of a side point there, but I think the people who tend to write the regulation, they generally don't really understand technology all that much anyway. I think the main risk, so speaking of Europeans, what Europe is going to try to do is over-regulate everything and just completely destroy innovation. I'm very worried about that. I think the US is traditionally much better at not over-regulating markets to let innovation thrive. I hope it stays that way. Nobody really benefits from over-regulation here.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Who wants to have either the market regulated because then they benefit because they can deal with the regulation, but small companies like mine can't. Or they are the folks who benefit from kind of fear mongering in the broader public where people start having a lot of respect for AI's capabilities and want to use it everywhere because AI is so smart it might even kill us all. There's a lot of dubious motives behind the scene there.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“There is just a non zero but very, very, very small chance there will be some sort of paperclip maximizer scenario. Have you heard of this paperclip maxim So, if you give a very intelligent system, an instruction like you need to make as many paper clips as possible, then it's going to turn everything into paperclips. And it's going to basically destroy the planet and turn everything into paperclips because that's its objective function in the process of maximizing paperclips. It will destroy everything else and turn the whole universe into paperclips. You can see why I'm saying that these are very, very small probabilities. And so probably the chance of me getting hit by lightning like right now is much higher than that happening. I think one of the issues I have with the whole debate around existential risk is that it's really a tiny probability, but we're pretending like it's a massive issue. And I think there are much bigger risks, right? Like nuclear war and pandemics and climate change. And those are things we should be focusing on much more. So the people who are pushing this narrative are really the people who are benefiting from this being the narrative. So these are the incumbent AI companies.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“This is the petition where he asks everybody to stop working on AI so that he can catch up. A lot of the narrative in the media right now is really driven by self-interest from a bunch of folks in the field. So the whole kind of existential risk debate, I think it actually comes from a very good place and a lot of people are worried about this. And I think there is a non-zero probability of AI extinction risk. So it's something we need to think about. And a lot of smart people are thinking about this, like Benjo and Hinton and all of these folks.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So, the way I think about the language model space is kind of as a pyramid. So at the top of the pyramid, we have these frontier models. So these are GPD4 and entropic models and things like that that are much better than everything else, but also much more expensive and much bigger.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“I think they're aware of it. They're starting to invest more in evaluation. They probably should have done that sooner. They have an army of annotators now, human annotators who are checking their models. So they probably have a pretty good sense of how good their model actually is. But obviously, they're not going to share that with the world.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Are lots of startups looking at this opportunity? Standard security companies are also looking at this right now. So it's very obviously a very big new threat surface. But I don't think that the actual foundation model builders like OpenAI and contextual are going to build that technology in-house. It's probably going to be an external audit.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Oh, absolutely, yeah. That's completely going to change everything. So these models can also get contaminated with data itself, right? So we need the security layer on the generations of the models. You can do all kinds of prompt injection attacks. These models, right now they're still mostly just producing language, right? But they're starting to produce code and instructions and actions. And when that happens, then you can mess with the model to get it to produce actions that you really don't want it to produce, like removing your entire database and things like that. So that definitely is something that you want to think about.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Oders' Stanford Helm project, the holistic evaluation of language models and these benchmarks where you look on static test sets, how good language models or how good they are on this static test set. But I've been arguing for a long time that that's just completely wrong anyway. And we need to do something that's much more dynamic. So ideally what you want is to see how easy is it for an adversarial person to mess with your model. The harder that is, the better your model is. And so it used to be very easy to come up with adversarial attacks where these models would just completely mess up. And it's getting harder and harder over time. So success rate of an adversarial attacker, that's something that can keep evaluating over time. So we need humans to evaluate these models by trying to break them.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Language model evaluation is a whole separate topic. It's a super interesting question, actually. I've been fascinated by AI evaluation for a really long time. And the answer is we don't really know how to evaluate the quality of these models anymore. So what we've seen people do in the field now is they're using GPD4 to evaluate the quality of other language models. That just feels wrong. So I think there's a giant opportunity in the market, actually, for a startup or several startups becoming like the moody or the S&P, the folks who evaluate the quality of AI for specific use cases because nobody really knows. It's really the Wild West out there. And one of the big problems, for example, is data contamination where a bunch of these language models are trained on the things that they are being evaluated on. So GPD4 looks like it's an amazing coder, but it might also just be trained on the data that is evaluated on, which means that it's not actually that great of a coder.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“These places have a giant mode because, as I said, it's really all about data and open AI has this very deep understanding of how people want to use language models. Basically, nobody else has. And they have this giant economy of scale where they can serve up language models very cheaply because they get so many requests coming in at the same time. So they have a giant mode. So I'm a big fan of open source, right? I would like it to be true that with open source we could just keep up with all of that, but I think that's just incredibly naive”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“OpenAI. They haven't even really trained as far as I know on the data that comes out of ChatGPT going viral. And so they had ChatGPT, it went viral. This led to the giant, giant data mode that they haven't even really used yet. So in terms of data modes, and maybe you've seen this come by actually, there was this Google memo from an internal Google employee who had written that OpenAI and Google have no mode. For me as an AI researcher, when I read that memo, I was like, this person has no idea what they're talking about.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Maybe it's useful to kind of go through the steps if you want to build your own chat GPT. Like what do you need? So the first thing you need is a core pre-trained model. And this tends to be just trained on the web, the task you're training it on is just next word prediction. Then once you have that core model, then you want to do supervised fine-tuning. So essentially, you want to fix the user interface to that model because the model doesn't really know how to follow instructions, for example. So you want the model to listen to you, but it has only been trained on predicting the next words. So it doesn't really know how to do that. So that supervised fine-tuning, that's also proprietary data. You can get a much better model out of that. And then the final step is RLHF reinforcement learning from human feedback, where you get this feedback loop to make the model even better for your specific use case, even if you don't have signal at the word level, you just have signal at the sequence level. So you can tell it like, okay, that was a good response or that wasn't a good response, but you can tell it like.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“You want to build a deep tech AI startup, then you really want to get a big data flywheel going. You want to start with a lot of data and then have a way to generate lots more data. And that data is going to be your mode. But I think one of the interesting things about these large language models is that they're incredibly sample efficient or data efficient. So you can do.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“It depends on where the data comes from. I think income is definitely an advantage there, but only some of them. And a lot of the data is just freely available on the internet. So the Lama model was not trained on any proprietary data. It was just trained on open data on the web. And there's a lot more data to be had there. And as a society, we're generating a ton of data every day to add to that big pile of data. So you can really train very high quality language models just on public data on the web. But I think if you look at the secret sauce to a lot of these other models, like why is GPT-4 so awesome? A part of that is that they went through enormous lengths to get special data that nobody else has. So allegedly they did this Whisper project where they're very good at transcribing audio because that would allow them to transcribe like all of the podcasts in the world, which gives you very high quality language if you can train on that language, but nobody else has it.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“It depends. So it's a trade off here, right? But these big models also need a lot of data. So it really is a function of the number of GPUs that you have available. Let's say you have 1,000 GPUs, you can choose to train a huge model on relatively little data, and it will be okay, but it will be under trained. So you have some sort of optimal point where you can train the model to perfection in the field. We were underestimating where that optimal point is. And it seems that data is much more important than model size when it comes to what's optimal.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, great question. I think Sam Aldman at this interesting quote where he was saying that he thought models would stop growing in size. GPT-4 kind of hit this ceiling. I think that's probably right, but not really because size doesn't matter. It's just that data size matters even more than model size. And I think the llama paper out of meta really brilliantly showed this. If you train a smaller model on more data for longer, then you get a better model. So you get more bang for your buck if you train it on more data rather than having more parameters. But in an ideal world, if you had infinite compute budget and infinite data, then you would train the biggest possible model because that's the most likely to give you emergent capabilities, as we call them in the field.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“This technology is really going to change the world and every aspect of it. So there are definitely a couple of incumbents, but they are also focused on very specific parts of the market. If you look at Anthropic and OpenAI, I think they're really chasing for this idea of AGI and they're relatively consumer facing. If you focus less on this idea of artificial general intelligence and you want to have something a bit more like artificial specialized intelligence where you just need the model to do what it needs to do, you don't need it to know about Shakespeare or quantum mechanics or things like that. You just needed to solve your business's problem. In that case, I think there's still a lot of room for innovation. And that's where we are trying to innovate.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“At this particular point in time, probably yes, but just because the field is moving so incredibly quickly. And so I think that in the next year, we're going to see lots of other models coming out. And if you can have a language model agnostic AI company that relies on language models, then that would give you a competitive advantage. But at the same time, it also runs the risk of you relying on other people's language models. It's a bit of a tricky situation.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“It's a great quote, but as always, it's a bit more nuanced than that. I think in some cases it is a feature if you want to use a language model for creative writing and if you want it to be really, really creative, then you probably want it to hallucinate. So in a way, it's a spectrum of groundedness and hallucination. If you really care about the language model doing the right thing and you want to deploy it in an enterprise critical situation, then you really don't want it to be creative. You don't want it to hallucinate you. You just want it to do what it has to do. But if you want to use it for a creative writing exercise, then sure you can have it hallucinate because you're going to revise whatever it gives you anyway.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So, we're not going to be able to know why a neural net does what it does at the scale that neural networks operate. So this is kind of like your own brain, right? Like I think your behavior is relatively predictable. So that goes for every human, right? We all can predict each other's actions, but I have no idea what's going on in your brain. And I will never know. There's no way I can know. The only way I can kind of find out is by asking you. But if you train the architecture the way we are training it right now, then at least you make sure that the model has learned to rely on the information that it finds. And that gives you much stronger attribution than if it's just predicting the next word based on what it has seen before, because it doesn't have this ability from birth, basically, to find relevant information and ground its generation on that thing it found.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So there are a couple of just really big issues hallucination. These models make things up with very high confidence, attribution. We don't know why they're saying what they're saying. We can't really trace it back to anything. There's compliance issues, so we can't really remove information from them, tricky from a GDPR perspective, for example. We can't revise information. We can't keep it up to date. There's massive data privacy issues where you have to send your very valuable company data if you're an enterprise. You have to send that to somebody else's servers. These models are also quite inefficient still. So you can make them much faster. What we are building at contextual is a different kind of language model. We're really thinking about this as the next generation of language models where we think about it from first principles for enterprise use cases. And what that means is that we want to solve all of these problems by being a bit smarter about the architecture. And the architecture, we're specifically basing it on this retrieval augmented generation.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so Disrudy started at the beginning of the year. We just saw this great need after ChatGPT had gone viral. We saw this great excitement in the world, but at the same time, a lot of disappointment about it not being quite ready yet for real world adaption in enterprises, where you actually want to use this technology. So we decided that now really is the right time to build a company to try to tackle that. And we think it's still very early innings in the game. So I think a lot of people sometimes think that the game has been played, but it's just getting started.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“I learned so much there mostly around how to focus your research direction. So I think initially I was doing all kinds of weird stuff. And it took me a while to figure out that a very clear real world application for the research that you're doing makes it much more valuable than going off on the tangent and maybe being a bit too far ahead of the rest of the field. A special place and I think actually they don't get enough credit for the impact that they've had on the world, Facebook or Meta in general actually. So almost every web app in the world runs on React, which is an open source project coming out of Meta.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Researcher called Mayon Botu, who is Jan Lecun's kind of, I wouldn't say sidekick because that doesn't really do justice to what he's done. So one of the godfathers of deep learning. And I had the opportunity to work with him. That was really an amazing time. So afterwards, when Jan and Leon started FAIR, Facebook research, I joined that out of my PhD. And that really kicked off my career.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“My journey has been a little bit unusual actually. So when I was in high school in the Netherlands, I wanted to be a cool kid during the day, but at night I was secretly fascinated by computers. So I taught myself to code. So then by the time I had to go to college and go study something, I thought I already knew everything about computer science. So I decided to study philosophy instead. Radical departure from what I had been interested in at the time, but it was fascinating. I use it. Still every day, I think. But then at some point in my career, it became clear that I had to start making money. So I needed a real job and philosophy is not really a real job. And I did some logic in between foundations of math, which is also not really a real job. So I decided to study computer science after all. So I went to Cambridge in the UK. And so that's really where I started doing NLP natural language processing. One of my internships was done at Microsoft Research in New York with a very famous research.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Data size matters even more than muddle size. If you train a smaller model on more data for longer, You get a better model. There was this Google memo from an internal Google employee who had written that OpenAI and Google have no mode. For me as an AI researcher, when I read that memo, I was like, This person has no idea what they're talking about.”
2023-06-30 · The Twenty Minute VC · 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI · IDENTIFIED FROM THE TRANSCRIPT · source