YouSaid · the spoken record

Alex Wang

lines on the record
73
first
2024-06-12
most recent
2024-06-12
sittings or episodes
1
sources
podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

  1. “Or chemistry data. These are the things that are really needed to push the boundaries of the models. I think this is a global kind of infrastructure level effort that needs to happen. Like, I think we need to think about it as how do we get the world's experts to collaborate with the models to help produce AI systems that are going to be the world's best scientists or the world's best coders or mathematicians.”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  2. “Yeah, exactly. Quite east of that. Yeah, yeah, yeah, yeah, yeah, yeah. So process mining, which is one of the terms in SaaS. But basically the continued collection of existing enterprise processes, then there's the consumer version of that, which looks kind of what you're referencing, or maybe it's with the meta-ray band collaboration or whatever device ultimately does it, but sort of something that collects the longitudinal view of your own life. And then there has to be a real investment towards human experts collaborating with models to produce frontier data. So both of the things I referred to before, both enterprise process mining and for lack of a better term, consumer data collection, those are all going to produce valuable data sets, but they're not going to produce the data that's actually going to push the models forward. Because to push the models forward, we need really highly complex data that's going to be able to push the frontiers of what the models can do. So this is where you need the agent behavior. This is where you need the complex reasoning chains. This is where you need advanced code data or maybe advanced physics or biology.”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  3. “So, there's probably two main pieces. One is like this effort from Limitless or other efforts, which is basically much more longitudinal data collection. Collecting more of what's naturally happening in the world. There's a bunch of forms of this. So one is like in a workplace, I think you're going to want, you know, as creepy as it sounds, some kind of constant data collection of what apps are using, what order apps you're using, you know, where do you copy paste one thing to another thing?”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  4. “So there's data mining, and then there's forward data production. These are the two core directions for where we need this data to come from. And I think kind of taking a broader step back, I think that a lot of AI progress at this point is fundamentally more data bottleneck. If we were able to produce compute and data in lockstep with one another, so as NVIDIA continued to manufacture hundreds of billions dollars worth more of chips, if we were able to produce a proportional amount of data as we got more and more chips and we were able to produce these two together, then we would get astronomically more capable.”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  5. “I don't think everyone will, but certainly the most sophisticated companies will. And then we'll be at a point where we still need to make the models better. At the end of the day, it'll all boil down to data production. What are the means of forward production in the same way that you need the means of forward production for chips and all the other things that you care about?”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  6. “So again, I think this is a case where there's two parallel efforts one is mining existing data, which by all means is going to be a one-time hit. There's going to be a one-time benefit that you get from mining already existing data, and it could be really meaningful.”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  7. “society wide impact by producing data to help improve these models. What we see is for scientists, mathematicians, doctors, human experts in the world, it's an incredibly exciting proposition to be able to, I can transmit my capabilities, intelligence training, all of that into a model that's going to be able to have society-wide impact. I mean, it's an incredibly exciting proposition.”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  8. “Yeah, yeah. Trainers is one term, AI trainers or contributors is another term. For what it's worth, I think this process of contributing data to AI is actually one of the highest leverage jobs that humans can have. And the reason for that is let's say I'm a mathematician. I can either go into my hole and do pure math and try to do pure math research. That's one trajectory for my life. The other trajectory is I use all my skills and talents and intelligence to help make these AI models smarter. Let's say I make GPD-4 just like a little bit smarter on math. If I take that little bit of improvement of the model and I sum that up across all the times that GP4 is going to be called and used across every math student who's going to use GPT-4, every company that's going to use GPT-4, every developer that's going to use GPT-4, that's a huge amount of impact. And so as a human expert, you have the ability to have”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  9. “But I think we don't think about this with data. And I think we need to do something very similar. And this process of producing data, it's sort of a hybrid human synthetic process. And that's really how we think about it, which is you need algorithms that can do a lot of the heavy lifting in producing synthetic data, but you need human experts who are going to be able to guide the AI systems and basically help provide input as to, you know, when the AI system gets stuck or when they have a factuality issue or when it's in a situation where it hasn't encountered before. A lot of autonomous vehicle scale up has been through these safety drivers. You know, you have safety drivers inside the car, and when the car starts screwing up, you have the safety driver disengage and sort of take over. And you need that kind of setup for these AI systems. You need AI models to be generating large amounts of data and then humans who can kind of take over and nudge the models when necessary to make sure that you get really high quality data.”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  10. “Yeah, so I think the second part is to your point new data that has to be produced. We need the means of production of new frontier data to get us from GPD-4 to GPT-10. I think when you think about chips, this is very natural, which is, oh yeah, we need to build more and more fabs. We need to build bigger fabs. We need to like increase the resolution and lower, lower nanometer fabs. Like for compute, it's very natural for us to think about increasing the means of production.”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  11. “Reasoning capability, which would definitely be a big breakthrough. The other one is just it's a data problem. It's like you need data for every scenario where you want these models to reason well in. You just need to overwhelm them with data in all those scenarios and you're going to get models that can reason really well.”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  12. “Yeah, I actually think that if you look at what these models can do, they're very good at reasoning in situations where they've seen a lot of data before. You know, I think we like to think about these AI as if they're like little human intelligences, but they're very different. Human intelligence and machine intelligence are very different. Humans have a very general form of intelligence. If a kid is raised in a very small neighborhood, they can live their whole lives in that small neighborhood and they can go to an entirely different part of the world and they can navigate, understand what's going on. No AI system today would be able to do that level of sort of drag and drop in one situation to another situation and figure out what's going on. I think we have to be cognizant that that's a limitation. But what that means is that for any situation that we want these models to perform well in, we need to have data of that situation or that scenario. And actually the model will perform really well. So there's kind of two ways to think about resolving the reasoning gap that exists in these current models. One is obviously you build some sort of general”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  13. “Exactly. This is to be a process of like every enterprise I have a set of very important problems for my enterprise, then I need to go through the process of like basically mining all of my existing data and refining all of that existing data for use for AI systems to solve my own problems.”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  14. “Basically, three pillars. So, first is there's a lot of this data that's locked up in the world's enterprises today. And none of that gets on the internet for very good reasons. But just to give a sense of scale, you know, JP Morgan's proprietary internal data set is 150 petabytes. GPT-4 was trained on an internet data set that was less than one petabyte. So the amount of data that exists inside large enterprises is just absolutely astronomical. So there's one process of just sort of mining all this existing enterprise data for all the goodness that exists within it.”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  15. “What I really believe is what we need from now forward is frontier data. We need to basically have data abundance of frontier data. We're right now we're in a sort of data scarcity mindset or we're hitting a data wall. And this frontier data is exactly what we're talking about. Frontier data in my mind is complex reasoning chains, complex discussion, agent chains of models going and looking up a piece of data, doing some reasoning, looking up another piece of data, maybe correcting if it has an error, tool use. All of the key components that we would think of an agent being able to do, that all needs be encapsulated into the frontier data to power capabilities of these models.”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  16. “The simple answer is like a lot of the thought process and a lot of the thinking that humans go through when they are doing more complex tasks that doesn't get written down on the internet. So for example, if I'm a fraud analyst inside a large bank, my job is understanding based on a set of transactions that seems suspicious whether or not it's a fraudulent transaction. And I need to analyze all sorts of different pieces of data and use my deduction and use all my human intelligence to make that decision. That process that I go through, it's not like I'm writing down step by step like, oh, I looked at this piece of data and I looked at this piece of data. And then based on that, I deduce this. And I'm not writing all that down on the internet to later be crawled by these models. One way to think about it is like all of the reasoning and thinking that is powering the economy today, none of that gets written down on the internet. And so if you just train on the internet, the model has no ability to learn from all of that.”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  17. “Anything that's easy and free to crawl or stuff that can be torrented, there's a lot of reports that there's a lot of torrented data in some of these models. Basically anything that is sort of like already written down and easy to get from the open internet. And then the first stage of a lot of this AI improvement has been these advances in pre-training, which is basically training these models to be really, really good at emulating the internet. And right now, we're at a point where these models are exceptionally good at emulating the internet. They're better than any human at emulating the internet. But the problem is when we think of AGI, when we think of powerful AI systems, we want much more than just emulating the internet. You want AI systems that can do tasks. You want AI systems that can solve difficult problems. You want AI systems that humans can collaborate with to solve all their daily problems. This process of building agents and AI models that are capable of all these things, you know, we're not going to get there from internet data. And we've already used up all the internet data.”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  18. “Yeah, so at a super high level, we've used up all the easy data. We've used up all of the internet data, the common crawl and newer versions of the common crawl.”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  19. “A combination of more algorithmic improvement, but in particular, we need to ensure that there's more data to support it.”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  20. “It's kind of this interesting thing. So there's three ingredients that go into these AI models or three pillars. So there's compute, of course, there's data, and there's the algorithms. The history of AI is that progress comes from sort of all three of these pillars sort of being built together. You certainly need a lot of computational capability, but you need the algorithmic advances like the transformer originally or RLHF or whatever future algorithmic advances come. And then you need the data pillar to support it as well. And I think a lot of the plateau that we've recently seen can almost be explained at a very high level from hitting kind of a data wall. The GPD4 was a model basically trained on nearly all of the internet and using a huge amount of computational capabilities. And a lot of, I think, what the industry has been doing over the past few years is scaling the computation dramatically, but not necessarily by building up the other two pillars in tandem. So there needs to be, I think.”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  21. “All in the same timeframe, we haven't yet seen the big breakthrough since GPT-4, which actually that model came out before this huge inflection in NVIDIA expenditure. So overall, it's this interesting thing where we're seeing investment into compute go up dramatically, go up exponentially right now, but we're still, I think as a community, as an industry kind of waiting for the next great model.”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  22. “I think it's pretty fascinating. I mean, I think there's been this especially coming up now where OpenAI has had GBP4 since fall of 2022. And since that time frame, we haven't yet seen a new base model or a new model that's jaw droppingly better than GBD4. You know, we haven't seen the GPD 4.5 or the GPD5 or the other labs haven't yet come out with models that are leagues and leagues better than GPT-4 despite way, way more compute expenditure. Since when ChatGPT came out, you know, you can look at the graph of NVIDIA's revenue and it just inflects. It's just like it just goes straight up after GPT. Four came out and it goes from, I think the NVIDIA's data center revenue, they were doing roughly about $5 billion a quarter, and then it shoots up to now it's north of $20 billion a quarter. So there's been tens of billions going to more than $100 billion of spend on high-end NVIDIA GPUs.”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source

  23. “At its core, this AI technology has the potential to be one of the greatest military assets that humanity has ever seen. Potentially even more of a military asset than nukes. Let's say China or Russia had AGI today in the United States didn't, I would imagine they would use that to conquer. The CCP's system is incredibly good at taking very aggressive centralized action and centralized industrial policy to drive forward critical industries. They have a clear shot at racing forward.”

    2024-06-12 · The Twenty Minute VC · 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's · IDENTIFIED FROM THE TRANSCRIPT · source