YouSaid · the spoken record

Richard Socher

lines on the record: 43
first: 2023-08-18
most recent: 2023-08-18
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“Well, I think and hope it will be the default in many hundreds of millions, if not billions, of people's computers so they can find better answers.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“I did not sign it. I don't think it makes sense to pause the training of models, just like I wouldn't want TESA to stop updating the AI in my Tesa car because I would assume it gets better and better. And I don't assume at some point the car will just take off and watch the sunset by itself because it got too intelligent.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“Have more folks talk to each other about real risks and let a small set of folks continue to think about existential risks but not scare people so much with very interesting sci-fi general fiction scenarios that would make fun action movies but are not the real issues at hand right now.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“Knowledge similar to the future is already here, but not equally distributed. How important retrieval augmentation is for LLMs? A lot of people underestimate that.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“And then you predict the right token 100% of the time. But that wouldn't be very interesting fulfilling intelligent existence. And so because companies need to make money and governments want to have a productive economy, no one is working on AI just doing whatever it wants to do because that doesn't make any money. So no one's working on AI setting its own goals. And until that happens, it's going to be hard to think about general intelligence of almost life form that it has some intelligence, even if it is not biological.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“We do have a lot of research breakthroughs we still need to make in order to achieve AGI. Just like it's very hard to know when someone would invent prompt engineering or when someone would have the idea to scale up language models and transformer networks just massively and not have other cute little models and modify those, but just scale up existing models massively like OpenAI and Ilya Satska and others ideas. We don't know when those breakthroughs will happen. I find it hard to call something artificially super intelligent or generally intelligent if all it does is predict the next tokens and you say, oh, predict this next token, it'll predict that next token. I think an intelligent system probably needs to have some of its own goals and like some of its own mind of what it might want to do. Maybe it doesn't want to generate tokens. If you just want to maximize the right token prediction, you just make sure everyone says AAA only as.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“100%. I do think past industrial revolutions, they usually happen for an individual at a surprising rate. When you're a weaver and you got some big machine now and it's just like totally automates making clothes, you will hate that machine, right? And the Ludites tried to destroy those machines. And there are people who want to slow down and destroy certain kinds of AI that change their jobs. And I feel empathy with those people. And it would be nice to have social systems that catch some of those people and help them learn new kinds of skills, help them incorporate those new technologies into their workflows so that, you know, if you're an illustrator and you used to be able to charge a thousand dollars for an illustration, because it took you three days, now it takes you three minutes. If you don't use it and you still expect to get paid for three days, but now there are thousands of people who can do it in three minutes, that if you're not adapting.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“Getting more and more expensive, and they're going to become the new bottlenecks, right? So your carpenter, people who built your house, all of those kinds of jobs are going to be more and more expensive. And then they're going to slow down overall progress so that the GDP can like 100x because of AI, it can only increase less than we think it might when we're deep in the bubble.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, that's the interesting thing, whereas, like, I'm so bullish and excited about AI. I have been for over a decade. And I think it will meaningfully change so many jobs. The more digital they are, the more data there is about a job, the more that job is likely to going to be automated and improved massively in its efficiency. But at the same time, there are still a ton of jobs where no one collects data. No one collects data about how my mate cleans and what are exactly all the inputs and the outputs and the actions and so on. Every house is slightly different. It's hard to constrain the environment. Full self-driving on off-road, dirt road, like small roads, night time, fog, like all of these things won't work for a long time. But the more constrained the environments are, the more I can do it, the more data we can collect, the more the AI can help us automate and make things more efficient. But what that means is that the task that are physical.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“You're 100% right, and I'll just jam on that. But there's a famous saying, which is the future is already here. It's just not equally distributed.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“Can do that. No normal human can translate that many different languages, but it's also hard to imagine what superhuman language is because language is a human construct. And if it was super human, it would just AIs be talking to each other with like 50 parallel streams or thousands of parallel streams. But humans can only register and understand language sequentially, right? You can't have five conversations at the same time. You can read five books at the same time. So language being a human construct and being how humans communicate thought and ideas has a limit on how superhuman it can be. It doesn't make sense to have super human language because humans couldn't understand it anymore to some degree, right? Again, you can produce it more quickly. You can generate a ton of stuff, but humans can only consume it at the rate that they have over the last several years.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“It's all of the above, right? But maybe in terms of AI and the progress, right, you think like we have this, keep having exponential growth. But I'll give you an example where we already see the flattening out of that exponential growth into an S-curve. And that is an image generation. It's photorealistic now. Look at some of these mid-journey images. You can just say, I want this exact thing and it'll just give you a photorealistic image of the scene you just described. Where do you go from there? You can't be like hyper, super duper photorealistic. Like people only can see an image, right? And that's how good it is. Now, you can, of course, still make a ton of progress in video generation and like make them moving. But image generation itself is basically maxed out now. It's hard to go. And then in some ways people think, oh, the AI has this super human capability. And you could argue, yes, like one AI algorithm can translate decently well into 50 different languages. No human.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“Airplanes and so on, the way they're structured right now, and we're slowing down. And like there hasn't been that much improvement. Like if you think about the space shuttle, it used to be that you could land coming out of space. You land like a plane in the space shuttle. Now we just drop people in a bucket with a parachute and they fall somewhere in the ocean. It doesn't look like we've made that much progress in terms of returning from space, but progress is not as linear or as exponential as a lot of people think.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“It's very natural for people to look at a type of progress and then extrapolate it further and further. And I think there's a little bit of overly strong optimism because we have again made a ton of progress, right? Like it's undeniable how much better AI has gotten. But just like with flight, for instance, human flight, we went from the first motorized human flight and then literally 30, 40 years later, we could fly loopings with machine guns and full metal airplanes high up in the speed of sound. And you're just like, wow, I mean, at this rate of progress, we're going to have vacations on the moon and we're going to have flying cars. And like everyone will just fly everywhere all the time. And so on. And then in the 50s, the whole thing just stopped and we're flying slower often now than we did before. People realize all kinds of issues. It burned a lot of gas.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“A little bit biased because I still love Salesforce in my time there. I think they have done a phenomenal job. When I was chief scientist there. And so they've been at the forefront for a while thanks to the research. And so they've been very, very quick also in incorporating that technology into its product. So that's been really great to see. And then, of course, in MySpace and Search, you can't deny the”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“That are not as good as the ads, so people click more on the ads. Like, you don't just replace all of that with a chat, right? You just lose hundreds of millions of dollars a day. And so there is still some innovator's dilemma that will not make them change their main experience overnight. It's been very carefully tuned every shade of blue has been A-B tested to death. It's very hard to make a massive change and improve that entire revenue overnight. So it'll slowly move carefully in their default experience. And that's sort of one opening that we have. But it's not going to be easy. You have to keep innovating and you have to keep working on partnerships too and try to stay clever and a little bit paranoid as a startup CEO in a space where you have these huge incumbents trillion dollar something companies that would love to crush you.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“I think it'll be a mix. I think there are companies that have been fully activated. I see like Salesforce having launched a bunch of really incredible features and made incredible announcements in their AI day that will make it harder for AI for service automation, for instance, because it's so core to their business. We've also seen Bing and Google copy what we have launched late last year with you chat. They've copied us in the sense that we've launched it earlier and then they launched something very similar three to four months after. At the same time, we don't see Google change and become a chat first search engine. They have some features somewhere else, but the main Google experience is the same. That kind of big change will be hard for Google too because they make 500 million dollars a day with privacy invading advertisements on that page. And so you don't just really nilly change most of that page and you get rid of the five, six ads that are on top of that page followed by a bunch of SEOs.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, it is the main question. The truth is, distribution won't be ever fully solved, and it's a constant uphill battle. You know, we just got de-ranked in all our sites from Google and saw like a drop right away. So we have seen a drop from getting delisted out of Google. We're working on a lot of different partnerships and yeah, constantly thinking of clever ways to do it that because we have some competition, small and large that have been copying many of our features in fairly quick succession, we have to be a little more careful there and not share all the details.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“Platform. But I think that is philosophically the right solution. And I hope people will start collaborating with us because if we don't win and others win, then there'll be actually fully left out.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, it's a great question. We need to have that search engine that we've built be an open platform where you can actually contribute your apps to and your content to. And then if it actually brings up the context from your app and we make money with it, then you can also participate in making that money or you have some subscription model, right? And you have some data that you show publicly, but then you have to subscribe. You can have that subscription happening right within your search engine by virtue of this being a much more open platform than Google. Now, we've launched this open platform last year, but to be honest, we haven't had a ton of really amazing apps being added to this platform because we just don't have hundreds of millions of users. And so if your app lets you book a kayaking trip in Indonesia or something, by the end, you only get like five people. And so been slow in an uptake on that.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, obviously it's a massive market, right? We have a trillion dollar plus company value in that, but more importantly, intuitively, we ask search engines questions every day to learn something. They ask in their phone search engines a lot. And so it's an incredibly impactful technology to help people learn. That's sort of another really big foundational block for the internet. Some people even think like Google is just sort of a yellow pages, right? I mean, there's so many examples and use cases where you can do so much better than what we had in the last 15 years.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“Think governments can help. The problem is it's very hard for governments to fund uncertain research and science projects. Because when it doesn't work out, some taxpayers will for sure complain. They're like, you guys wasted my taxpayer money. It's hard to build that. It will then inevitably have to become even more bureaucratic than we're actually has some necessity for and so on. But I do think governments should fund clients and especially open science.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“And hurdles you have to jump through to make an edit to that article. You have to build trust with folks and whatnot. But it has happened in the past that when there is enough interest and excitement and the strong structure that people can collaborate on open models”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, I certainly see that there's a lot of complexity in open models, but you also have already seen a lot of open models, right? Like Lama Tudis was probably like a week or two ago, Lama 2 came out, right? And Lama 2 is incredibly good and it made even more progress than the past open model. And so we might not see for a while until some really strong, smart engineering project manager comes along that can herd all these incredibly smart and ambitious cats to come together and build a single model. But that would be my dream is that we actually have a single model and just like Wikipedia, the whole world collaborates and adds to it. And then we get ultimately an AI that anyone on earth can talk to, just like anyone on earth can ask facts on Wikipedia. And, you know, like Wikipedia takes a lot of coordination. These articles have lots of links.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“And improve and publish papers on. So they need to have a model to exist. And those are all very, very smart people now that don't have as many resources usually. They can train a single model for like 20 or 50 million dollars because they're in universities, but they're finding ways. They're collaborating, and they're probably going to work on foundational models that are fully open source. And we now see this with surprisingly Facebook being at the very forefront of it. People will layer on top of that, make it better. And then there will be open source versions. You can run on your phone. And those will get better and better over time. And so I'm quite bullish on the LMs in particular getting more and more commoditized. And yes, there will be a few foundational companies also behind OpenAI, working very hard to catch up with them. And they did raise a lot of money too. And it's good to have some competition in that space.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, certainly you can't deny it at OpenAI is a head by a lot. I predicted that we'll have a GBD4 equivalent model before the end of the year that's open source. Of course, GBD4 keeps getting better and better. So my prediction was for the version we had like a few months ago. But I actually think that with models like Lama 2 from Facebook and everyone there, I do think open source will take over a lot of use cases, right? It's already getting close to GPD 3.5 when there's this much excitement in so many careers depending on understanding these models. Imagine all the researchers in all these universities, right? They're all of a sudden kind of out of a job unless they have an LM that works really, really well and they can do useful things with it. They're not going to just say, oh, let's just from now on run our entire research agenda on some closed API that we cannot analyze and understand.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“Be able to incorporate multiple different models because you may have a predictive forecasting model and that is important to include into an LM, like an LM won't be as good in doing a financial forecast because that's not what they're trained on. Like LMs help a ton for all things natural language because they understand so much about natural language and have so much world knowledge, but that doesn't help you if you just have a huge sequence of numbers and you need to build a very, very accurate statistical predictive model for like forecasting or something.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“Interesting, you can almost start to in the future think about the LMs as akin in some ways to a database. You don't really care about which database people are using nowadays, right? If you want to go really big, you might use an Oracle database. If you just want to build a smaller thing, some MySQL open source database and there's a bunch of others in between. And I think it won't matter that much which database you use. Just like it won't matter that much which LM you use, but it matters what you do with it, how you tune it, what kind of train data you add on to it to fine-tune it, how you retrieve facts into it so it can reason well over all of those things, how you maybe prompt it to run multiple chain of thought steps to get to a conclusion, all of those things I think will matter more and more. And so I would argue that it's even more important than switching between models is to”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“That's a great one that's sort of right in the sense that we're going to update all these models. Like we update our model every other week. And so it's not that exact model with those exact weights. But my hunch is, we will still have a lot of large language models that are running in production that are that general model architecture. I mean, there'll probably still be some transformers, even though transformers are mostly amazing because they can be trained on GPUs. There are lots of other models that we're currently not exploring because they're not trainable on GPUs.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“What we can and cannot say or hallucinate about. But hallucinations aren't a feature and are a problem for search engine, right? We do want to be most of the time when someone asks us about a fact. We need to bring in the right fact and then be very accurate.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah. You know, it depends again on the context. Your questions are perfect in that they help people understand it's neither black or white, but it's just some complexity in between. And the complexity here is that indeed the LM doesn't know your intentions and your background. And so it takes some time for them to adjust to understand and for us to train them and then to balance out like what it can and cannot say. And I think some cases we even overshot a little bit. You know, there are some folks who say, oh, you cannot tell any dirty jokes anymore or like a murder mystery is like unethical to write, so we should not ever write about a murder mystery. But no one says even king or a Gothic Christie are like unethical people for writing murder mysteries. People just think it's great entertainment, but they're scared when an AI slash large language model predicts a tokens given that prompt to write about that kind of stuff. And so in some cases, we almost overshot.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“Areas where the default large language model will not do as well. So, for instance, if you wanted to be more factual, more up-to-date, and have citations for the facts that it tells you, you need to have a search backet. And so at U.C.Con, for instance, we've had to build this very complex search back.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“You're asking very good questions that often have a more subtle answer than what would fit in a tweet. I think there are some very thin wrapper companies out there that probably have very little moat, but there are also companies that people don't realize the complexity that make it really work. And they underestimate all of a sudden all the other stuff that companies need to get right to build a viable business. So concretely, you know, you could think about Instagram, right? Instagram is like, what's the mode of Instagram? It's certainly not their AI and their backend and the brilliance of engineering. It's just like a fairly simple photo sharing app with some fun filters back in the day. None of that was rocket science. Turns out you can have modes other than your back end AI model, right? It's distribution, it's partnerships, your sales funnel processes, and so on. So there's a lot. And then there are also.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“Same time, the state before large language models, and we call them foundational models because you can often build on top of them very easily. Before that, it was even harder. Like it would have been impossible for us as a small company to build a search engine that understands all of these different things in many different languages. It would have been unthinkable, but now we can actually, because of these large language models and foundational models, we can have a general sense of understanding natural language. And you can as a small startup nowadays and build in like 80% solution very quickly. You layer more and more specific data for your task on top as after you have an MVP, a minimum viable product, and then you make it really, really good. Now, the big incumbents, they can make it really good much more quickly, but you could get to at least an 80% solution thanks to large LLMs, like more quickly also.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“So funny because it's another question where they're actually both right. So it's complicated in the sense that unsupervised data, just raw internet text is easily accessible, but there's still a lot of data sets out there that are not out there. They're actually stored in private databases. And indeed, if you want to answer customer emails automatically, it's very helpful if you're Salesforce and you already have all those emails. You already have labels that people, this knowledge-based article answered this email or answered this question, right? If you have that data, you can then train that particular AI for that company to answer its questions from its customers automatically. You can do that much more easily than if you're a small startup and you need to just get access to that data and to get permissions and so on. And so that part is true.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“Across the internet on all the chemistry articles and biology articles and so on, you learn world knowledge, you infuse world knowledge into that large language model. But only if you have enough parameters to learn it all.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“A model was to also use language modeling as one of those tasks. It was actually on our to do list, but we didn't get to before others did. And the idea of language modeling is you just predict the next word, which is very easy to get a lot of data for because you can use anything on the internet and so on. But it's actually incredibly hard. And to do it really, really well, you have to learn so much about the world, right? If I'm just, I have this sentence, like, I'm in New York City and I'm driving north too, right? And now you want to predict what's the next word. Maybe it's Boston, maybe it's Montreal, maybe it's Yale, but you probably want to put more probability on the word being Boston than Yale because there are more people probably driving to Boston because it's a bigger city. And so not only do you learn about geography, but you also learn just to predict that one word really, really accurately from that one sentence, you need to learn everything about the geography of the northeastern United States. And now if you do this billions of times.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“Totally not wrong. It's just not mutually exclusive. You need a large model and you need a lot of training data for that model. Either of them in isolation, like, you know, just like imagine the simplest neural network will just predict a single one-dimensional output line, right? It's like a regression analysis. You have some input x, some output y, and you try to model where it goes. You can model that with a handful of neurons. And the simplest one is just like a line, right? A linear regression. And that model has even fewer parameters. Long story short, if you now give this linear regression model billions and billions of training data, it's not going to learn magically anything but a simple linear line. But if you give the model billions and billions of parameters, it can learn all kinds of very complex predictive functions and abilities. So concretely, the big breakthrough on top of this idea of pronounced engineering, of being able to have a scene.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“It is super important. You just cannot train a single model for all of these different tasks with a small model. That's exactly why and how it would have always failed in the past.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“To incorporate world knowledge through a large language model to really make that happen. And you needed attention mechanisms and fast GPUs and hardware. There's a lot of things to make it work to then eventually show that it was better. And when you had very small models 10, 20 years ago, it would have been impossible to make it work.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“Oh man, yeah, I had some funny rejections on that. Actually, the paper when we submitted this prompt engineering paper, it got rejected. And the reviewer said, oh, this is such a misguided effort. There's not a single model in the world that could do any of these things together, to which I thought the brain, like we don't replace our brain, we use the same brain. So there is an existence proof for a single model for all of NLP. But it was just a very different way of thinking about the world. There's just debate and people done a lot of research and were very stuck in one way of thinking about it. There's sort of academic preconceived notions about how things have been and how certain people have worked on certain problems for a long time. And the notion that you should have a single model for each specific task was prevailing for a long time because it's also really, really hard. You needed the ability to train massively large models. You needed the ability.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“It then you have That just translates German into English. You have a question. That's mentioned. And then you just give that.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source
“The future is already here, it's just not equally distributed. And I think it will meaningfully change so many jobs. The more data there is about a job, the more that job is likely to be automated. But at the same time, there are still a ton of jobs where no one collects data. What that means is that the tasks that are physical are getting more and more expensive. And they're going to become the new bottlenecks. And then they're going to slow down overall progress so that the GDP can like 100x because of AI. It can only increase less than we think it might when we're deep in the bubble.”
2023-08-18 · The Twenty Minute VC · 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher · IDENTIFIED FROM THE TRANSCRIPT · source