YouSaid · the spoken record

Nathan Labenz

lines on the record: 112
first: 2025-10-14
most recent: 2025-10-14
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“Class of models got about a 50% on that benchmark. And GPT 4.5 popped up to like 65%. So in other words, it basically, of the things that were not known to the previous generation of models, it picked up a third of them. Now, there's obviously still two-thirds more to go, but I would say that's a pretty significant leap, right? These are super long tail questions. I would say most people would get like close to a zero. You'd be like the person sitting there at the trivia night who like maybe gets one a night is kind of what I would expect most people to do on simple QA. And that checks out, right? Like obviously the models know a lot more than we do in terms of facts and just general information about the world. At a minimum, you can say that GPT 4.5 knows a lot more, a bigger model is able to absorb a lot more facts. Qualitatively, people also said in some ways, maybe it's better for creative writing. It was never really true.”
2025-10-14 · a16z Podcast · Is AI Slowing Down? Nathan Labenz Says We're Asking the Wrong Question · IDENTIFIED FROM THE TRANSCRIPT · source
“And there are a number of interesting, of course, there's a million benchmarks, whatever. The one that I zero in on the most in terms of understanding how GPT 4.5 relates to both O3 and GPT-5 and OpenAI obviously famously terrible at naming. We can all agree on that. I think a decent amount of this confusion and sort of disagreement actually does stem from unsuccessful naming decisions. 4.5 on this one benchmark called simple QA, which is really just a super long tail trivia benchmark. It really just measures, do you know a ton of esoteric facts? And they're not things you can really reason about. You either just have to know or don't know these particular facts.”
2025-10-14 · a16z Podcast · Is AI Slowing Down? Nathan Labenz Says We're Asking the Wrong Question · IDENTIFIED FROM THE TRANSCRIPT · source
“It's really not clear yet to me whether or not the scaling laws have petered out or whether we have just found a steeper gradient of improvement that is giving us better ROI on another front that we can push on. So they did train a much bigger model, which was GPT 4.5.”
2025-10-14 · a16z Podcast · Is AI Slowing Down? Nathan Labenz Says We're Asking the Wrong Question · IDENTIFIED FROM THE TRANSCRIPT · source
“And then also he presents sort of his theory of why we shouldn't worry about the future because it's slowing down. And why don't we just share how we interpreted kind of his history, which as I interpret it was this idea of, hey, the simplistic version is we've figured out this way such that if you throw a bunch of data into the model, it gets better in sort of order of magnitude. And so the difference between GB2 and GPT-3 and the GPT-3 and GPT-4. But then that sort of significant, the difference. But then it achieved sort of a diminishing returns significantly. And we're not seeing it into GPT-5 and thus we don't have to worry anymore. How would you edit the characterization of his view of the history and then we can get into the differences between four and five? The scaling law idea, which is definitely worth agreeing, taking a moment to note that it is not a law of nature. We do not have a principled reason to believe that scaling is some law that will go indefinitely. All we really know is that it has held through quite a few orders of magnitude so far. I think that”
2025-10-14 · a16z Podcast · Is AI Slowing Down? Nathan Labenz Says We're Asking the Wrong Question · IDENTIFIED FROM THE TRANSCRIPT · source
“Come to take for granted and kind of forget that GPT-4 didn't have a lot of the things that now were sort of expected in the GPT-5 release because we'd seen them in 4.0 and 01 and 03 and all those things sort of maybe boiled the frog a little bit when it comes to how much progress people perceived in this last release. Yeah, a couple reactions. So one is, and even to complicate your two by two even further in the sense of is it bad now versus it bad later? Cal is not really who we both admire, by the way, a lot. Cal is a great guy and a valuable contributor to the thought space, but he's not as concerned about sort of this sort of future AI concerns that sort of the AI safety folks and many others are concerned about. He's more concerned about what it means to life for cognitive performance and development now in the same way that he's worried about social media's impact. And you think that's a concern, but nowhere near as big a concern as what to expect in the future.”
2025-10-14 · a16z Podcast · Is AI Slowing Down? Nathan Labenz Says We're Asking the Wrong Question · IDENTIFIED FROM THE TRANSCRIPT · source
“Maybe one reason that's happened a little bit is that there were just a lot more releases between GPT four and five. So what people are comparing to is, you know, something that just came out a few months ago, 03, right, that only came out a few months before GPT-5. Whereas with GPT-4, it was shortly after ChatGPT and it was all kind of this moment of, whoa, this thing is like exploding onto the scene. A lot of people were seeing it for the first time. If you look back to GPT-3, there's a huge leap. I would contend that the leap is similar from GPT-4 to 5. These things are hard to score. There's no single number that you could put on it. Well, there's loss, but of course one of the big challenges is that what exactly does a loss number translate into in terms of capabilities? So it's very hard to describe what exactly has changed, but we could go through some of the dimensions of change if you want to and enumerate some of the things that I think people maybe are starting to or have.”
2025-10-14 · a16z Podcast · Is AI Slowing Down? Nathan Labenz Says We're Asking the Wrong Question · IDENTIFIED FROM THE TRANSCRIPT · source
“Want to have to sit here and read this code and figure out what's going on. It's not even about typing the code anymore. I'm way too lazy for that, but it's even about figuring out how the code is working. Try again and just try again. I do find myself at times falling into those traps. But I would say big part of the reason I can fall into those traps is because the AIs are getting better and better and increasingly it's not crazy for me to think that they might be able to figure it out. So that's my kind of first slice at the takes that I'm hearing. There's almost like a two by two matrix maybe that one could draw up whereas do you think AI is good or bad now and in the future and do you think it's like not a big deal or a big deal? And I think it's both on the good and bad side. I definitely think it's a big deal. The thing that I struggle to understand the most is the people who kind of don't see the big deal that it seems pretty obvious to me and especially when it comes again to the leap from GBD4 GB5.”
2025-10-14 · a16z Podcast · Is AI Slowing Down? Nathan Labenz Says We're Asking the Wrong Question · IDENTIFIED FROM THE TRANSCRIPT · source
“Really right to watch out for that sort of stuff. And then, as we've covered it many conversations in the past, I've got a lot of questions about what the ultimate impact of AI is going to be. And I think he probably does too. But then when it comes to, it's a strange move from my perspective to go from, there's all these sort of problems today and maybe in the big picture to, but don't worry, it's flatlining, like kind of worry, but don't worry because it's not really going anywhere further than this, or it's scaling has kind of petered out or we're not going to get better AI than we have right now. Or even maybe the most easily refutable claim from my perspective is GPT-5 wasn't that much better than GPT-4. And that, I think, is where I really was like, what, wait a second. I was with you on a lot of things and some of the behaviors that he observes in the students, I would cop to having exhibited myself when I'm trying to code something these days. A lot of times I'm like, oh man, can't the AI just figure it out? I really don't.”
2025-10-14 · a16z Podcast · Is AI Slowing Down? Nathan Labenz Says We're Asking the Wrong Question · IDENTIFIED FROM THE TRANSCRIPT · source
“When it comes to some of the worries about the impact that AI might be having, even already on people, you know, he looks over students' shoulders and watches how they're working and finds that basically he thinks that they are using AI to be lazy, which is no big revelation. I think a lot of teachers would tell you that. Shocker puts that in maybe more dressed up terms that people are not even necessarily moving faster, but they're able to reduce the strain that the work that they're doing places on their own brains by kind of trying to get AI to do it. And, you know, if that continues, and I think he's been, I think a very valuable commenter on the impact of social media, certainly I think we all should be mindful of how is my attention span evolving over time and am I getting weak or averse to hard work? Those are not good trends if they are showing up in oneself. So I think he's”
2025-10-14 · a16z Podcast · Is AI Slowing Down? Nathan Labenz Says We're Asking the Wrong Question · IDENTIFIED FROM THE TRANSCRIPT · source
“Nathan, I'm stoked to have you on the AC podcast for the first time. I've been podcast partners for a long time with you leading cognitive revolution. Welcome. It's great to be here. Thank you. So we were talking about Cal Newport's podcast appearance on lost debates. We thought it was a good opportunity to just have this broad conversation and really entertain this sort of question of is AI slowing down? So why don't you sort of steal man some of the arguments that you've heard on that side from him more and more broadly than we could sort of have this broader conversation? Yeah, I mean, I think for one thing, it's really important to separate a couple of different questions, I think, with respect to AI. One would be, is it good for us right now even? And is it going to be good for us in the big picture? And then I think that is a very distinct question from are the capabilities that we're seeing continuing to advance and at a pretty healthy clip. So I actually found a lot of agreement with the Cal Newport podcast that you shared with me.”
2025-10-14 · a16z Podcast · Is AI Slowing Down? Nathan Labenz Says We're Asking the Wrong Question · IDENTIFIED FROM THE TRANSCRIPT · source
“A growing debate about whether AI progress has plateaued or if our expectations have simply caught up to the pace or change. On this episode, I'm joined by Nathan LeBanz, host of the Cognitive Revolution to unpack whether AI innovation is actually slowing. We break down the case for slowdown from Cal Newport's argument that students are using AI to get lazier to the claims that GPT-5 wasn't a leap over GPT-4. Nathan and I look at what's really happening under the hood of AI, from new reasoning and map capabilities to real scientific discoveries and multimodal systems that go far beyond chatbots. We also discuss agents, automation, and how quickly work itself is starting to change. And we'll end on the big question. If progress isn't slowing down, how should we shape it toward a future we actually want? Let's get started.”
2025-10-14 · a16z Podcast · Is AI Slowing Down? Nathan Labenz Says We're Asking the Wrong Question · IDENTIFIED FROM THE TRANSCRIPT · source
“AI is not synonymous with language models. AI is being developed with pretty similar architectures for a wide range of different modalities. And there's a lot more data there. Feedback is starting to come from reality. Maybe we're running out of problems we've already solved when we start to give the next generation of the model these power tools and they start to solve previously unsolved engineering problems. I think you start to have something that looks kind of like superintelligence.”
2025-10-14 · a16z Podcast · Is AI Slowing Down? Nathan Labenz Says We're Asking the Wrong Question · IDENTIFIED FROM THE TRANSCRIPT · source