YouSaid · the spoken record

Mihir Bellare

lines on the record: 62
first: 2025-10-13
most recent: 2025-10-13
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“So, in context learning, you create your new DSL and you give it to the prompt and you can see the confidence rising with each new example, the entropy reducing. And that sort of is a validation of the model.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, yeah, token probe is still up, and you can see actually the token probe is software that we built, and thanks to Martin and A16Z's generosity, it's running on your servers, and anyone can go and test. And what we have done there is we actually show the entropy.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, you know, even with LLMs, in the paper, we say that you can improve the inference by following this low or minimum entropy path. So that's a very sort of small step that we are building and training models that will do inference based on the entropic path.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“So, I have one quick question. What's next for you? I mean, you've tackled in context learning, you've got a model for LOMs, and I've got a generalized model for their solution space. What are you thinking about tackling next?”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“There are several mini-instead examples. And yeah, it's creating this new manifold. I didn't want to use that definitional answer. I thought it might sound too. Work to mathematical But essentially, if LLMs really created this new manifold, Then I would be convinced. But so far, they have just gotten better at navigating the existing manifold, the existing training set.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“So considering the existing traded data, if it ever does that, if it does something that's outside of that distribution, then clearly we're on a path to learning new things. And if not, then everything is just a computational step from what's already known”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“I think that you can almost take a definitional approach to answer this question, Vishal. Like, the problem with these types of questions is if you have billions of dollars and you can collect whatever data you want, you can make a model do anything you want, right? And so I'm saying like at some level, you've got this entire capital Structure machinery behind these models, so you're like, oh, it can be good at science. Well, sure, you put a billion dollars of solving materials science and collect all this data, you'll be good at material science or whatever it is. And so, but there is a definitional answer, which is And I'm going to draw from your work, which is there is a manifold that's in there based on the data that's been training on. And then the question is, if it ever produces something that's off, like a new manifold.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“If the very real world tasks. Good question You know which LLMs or these models Domain where you have the most training data is probably cod And coding is where. You can also have the most structure. And yet anyone who has used these tools, whether it's cursor or whatever, Claude Cood, LLMs continue to hallucinate, continue to generate unreasonable code. You have to. You have to constantly Babysit these models. So the day an LLM can create a large software project without any. Babysitting is that the day I'll be a little bit convinced that it's towards Asia. But again, I don't think it'll be able to create new science. If it does, that's when I'll begin with.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“I don't know about this bit. Let me ask one more LLM question, which is. Are there any benchmarks or real world tasks that if they occurred, you'd sort of reevaluate and say, hey, maybe LLMs are closer to the path to AGI than I thought.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, exactly. So it was so easy in some sense to build these artifacts and then just measure them that people have been going around trying to do that. One time I really dislike is prompt engineering. Engineering used to mean sending a man to the most broad or providing five nights reliability. Prompt engineering is prompt twiddling. You fiddle with a prompt and the bottle changes, and the inference, the output changes. And, you know, you have hundreds of papers Just doing one ex winner on the other, changing a prompt this way, that way, and writing their observations. And as a result, you know, lots of these papers are being written, are being submitted for review. Reviews get busy looking at all this kind of empirical work. And my personal taste is First, try to understand, model it And then you can do the other thing”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“Listen, I honestly, I mean, I find there's so much empiricism in the current AI community exactly because we don't understand the system. You know, it kind of reminds me. I feel like systems went the other way, right? It's like we had all of these models, but then we didn't understand how the systems worked. And then we just actually did measurement. It feels like ML or the AI stuff is the opposite, which is like, we know we don't understand them. And so we just measure them, but now we're trying to come up with the models.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“Some of them are receptive to it, definitely. But You know These large conferences and their reviewing process is so random. And the kind of questions they ask, you know, I'm a modeling person. I like to model things. And I submitted one version of this work to one very famous. Machine learning or AI conference, and the reviewer said, okay, this is a model, so what? So there is.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“What's interesting is you have these anecdotal examples of humans developing languages de novo that have been recorded, right? Like it's either the Guatemalan or Nicaraguan sign language, right? Where there are these students. That developed their own language without being taught. And so that would suggest that language as follows intelligence. The problem is they're all anecdotal, right? Like who knows if somebody didn't teach them sign language? Like nobody really knows there is no controls. So this is all these observational studies. And there's so few of them you have to wonder if it's just kind of sloppy observation. And so I think that the question is still outstanding.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“So, one of the new architectures, architectural things is how do we get these models to do approximate simulations? Test out that idea and whether to proceed or not. So, yeah, we have, you know, another thing that I've always wondered about is did we develop as humans, did we develop language because we were intelligent or because we develop language, we accelerated our intelligence? So I don't know which side of the camp you follow on that”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“You know, Jan has been pushing at this JPA architecture. Energy based architectures seem promising. The way I have been sort of thinking about it is There's this set of benchmarks or the arc prize That Mike Kanuro and Francois Chalet have. And if you understand why the LLMs are failing on this test, maybe you can sort of reverse engineer a new architecture that will help you succeed in them, right? And I agree with a lot of what several people say that, you know, language is great, but language is not the answer. When I'm looking at catching a ball that is coming to me, I'm mentally doing that simulation in my head. I'm not translating it to language to figure out where it'll land. I do that simulation in my head.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“But I think we need a new architecture to sit on top of LLM Reach AJI. You know, a very basic thing what Martin just said, you give them eyes and you give them ears, you make them multimodal, of course they'll become more powerful. But you need a little bit more than that. The way human brains learn with very few examples, that's not the way transformers learn. And, you know, I'm not saying that we need to create an Einstein or a gale, but there has to be an architectural leap that is able to create these manifolds. And just throwing new data will not do it. It'll just smoothen out the already existing manifold”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“So, I mean, again, I love LLMs. They are fantastic. And they are going to increase productivity like nobody's business. But I don't think they are the answer. So, you know, Yad Lichon famously says that LLMs are a distraction on the road to AG.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“I completely agree. There has to be a new sort of architectural leap that is needed. Go from the current, you know, just throwing more data and more compute, it's going to plateau. It's the iPhone 15, 16, 17.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, because I mean, there's this view that people have. They're like, well, This is all good and well, but I could just take an LLM and I can give it eyes and I can give it ears and I could put it in the world and it'll gain information and based on that it'll improve itself and therefore it can learn new things. But the counterpoint that I've always just intuitively thought to that is the amount of data used to train these things is so large how much can you actually evolve that manifold given an incremental? I mean almost none at all, right? There has to be some other way to generate new manifolds that aren't evolving the existing one”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“Can you think that based on the work you've done, can you bound the amount of data, computer, or Data or compute that would be needed in order for it to evolve. So, one of the problems. If you just take LLMs as they exist, there was so much data used to create them to create a new manifold will need a lot more data just because of the basic mechanisms, right? Otherwise, it'll just kind of like, you know, get kind of consumed into the existing set of data. Like, have you found any bounds of what would be needed to actually evolve the manifold in a useful way? Or do you think we just need a new architecture?”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“Okay, so AGI. So, how do I define AGI? So the way I would say that LLMs currently navigate through this known Bayesian manifold. AJI will create new manifolds. So, right now these models navigate, they do not create. AGI will be when we are able to create Science, new results, new math. When an AGI comes up with a theory of relativity, I mean, it's an extremely high bar, but you get what I'm saying. It has to go beyond what it has been trained on to come up with. New paradigms, neoscience. That's by definition of AGI.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“The way I think about it, the way we try to formulate it in our papers, it's beyond a sarcastic parrot, but it's not AGI. It's doing Bayesian reasoning over what it has been trained on. So it's a lot more sophisticated than just a stochastic parrot.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“Martine was talking earlier about how the discourse was either stochastic parrots or AGI recursive room. How are you How do you conceive of sort of the AGI discourse or even the concept? What does it mean to the extent that it's useful? How do you think about that?”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“They are not inventing new kinds of math. They are able to connect known results in a sequence of steps to come up with the answer. So, even the LLMs, what they are doing is they are exploring all sorts of solutions In some of these solutions, they start going on this path where the next token entropy is low. So that's where I say they are in that Bayesian manifold. Where you have this entropy collapse. And by doing those steps, you arrive at the answer. But you're not inventing new math. You're not inventing new axioms or new branches of mathematics. You're sort of using what you've been trained on to arrive at that answer So, those things LLMs can do, they'll get better at it of connecting the known dots. But creating new dots, I think we need an architectural advance.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“And for that, you have to go outside your training set. Similarly, any LLM that was trained audit would not have come up with quantum mechanics That's where particle duality or this whole probabilistic notion or that energy is not continuous, but it is quantized. You had to reject Newtonian physics. Or ghettos incompleteness serum. He had to go outside the axioms to say that, okay, it is incomplete. So those are examples where you're creating Neoscience are fundamentally new results, that kind of self improvement is not possible with these architectures. They can refine these, they can fill out these roles. Where the answer already exists. Another example, you know, which has received a lot of press these days is these IMO results, international math will appeared. Know whether it's a human solving it or the LLM solving it”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“So, any model, any LLM that was trained on 1915 physics. Would never have come up with a theory of relativity. Einstein had to sort of reject the Newtonian physics and come up with this space time continuum. He completely rewrote the rules. So that is an example of AGI. Where you are generating or generating new knowledge. It's not simply unrolling your work.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so you can represent the sort of information contained in these models. And let's go back to that matrix analogy that I have, the matrix abstraction. So like I said, these models represent a subset of the rows. So, a subset of the rows are represented, but some of these rows Are able to help fill out some of the missing rows. But since the model knows how to do multiplication doing this step by step, then every row that is corresponding to, let's say, 769 times 125 or whatever, all those multiplications. Can fill out the answer because it has those algorithms sort of embedded in them. You just need to unroll them. So it can sort of self improve up to a point. But beyond a point, these models can only sort of generate what they have been trained on. So let me give you I'll give you three examples.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, actually, you know what's kind of interesting is like often most people agree that if you have one LLM and you just feed the output into the input, like it's not going to do anything. But then often people will say, Well, what if you have two LM? You have no external information, but you have two LLMs talking to each other. Maybe they can improve each other and then you can have like, you know, a takeoff scenario. But again, you even address this, even in the case of like n number of LLMs using kind of the matrix model to show that like you just aren't gaining any information entropy.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“So, you know, another phrase that We've been using recently is the output of the LLM is the inductive closure of what it has been trained on. So when you say that it can recursively self improve, It could mean one of two things. So let's get back to the”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“What I've found very impressive is you've used this basic model to show a number of things, right? To describe in context learning and to map to Bayesian learning, but you did it for another one where you kind of, you've sketched out this almost glib argument on Twitter, on X, where you made this rough argument for why recursive self-improvement. Can't happen without additional information. And so maybe just walk through very quickly how the same model, you can just very quickly show that a model can never recursively self-improve.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“This is an example of fewer short learning or in context learning, right? But when I give that prompt along with these examples to this LLM, I'm not saying to the LLM, okay, this is an example of few short learning. So learn from these examples. You just pass this to the LLM as a prompt and it processes it exactly the way it would process any other prompt, which is not an example of in context learning. So that really means that the underlying mechanism is the same Whether you give a set of examples and then ask it to complete a talk, a task like an incontext learning, or just give it some prompt for continuation that I'm going out for dinner with Martin tonight. There's no in-context learning there. But”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“The weights are standard time. Yeah, yeah. Yeah, yeah. This was happening in October 2020. I had no access to internals of OpenAI. I could just access their API. OpenAI had no access to internal structure of Statsguru or the DSL that I cooked up in my head. Yet after showing it only a few examples, it learned it right away. So that's an example where it has seen DSLs or structures in the past. And now using this evidence ratio, okay, this is what my DSL looks like. Now a new natural language query, it is able to create the right posterior distribution for the tokens. That map to the example that I've seen. Now, the other beautiful thing about this is”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“So I created my own DS Which natural language query in cricket to this DSL, which then I can translate into a SQL query or a REST API, whatever. But getting the DSL is important. Now, these LLMs, I have never seen that DSL. I designed it. But yet, after showing a few examples, it learned it. How did it learn that”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“And given that I'm going out with Martin, it can produce a Bayesian posterior. It uses that evidence that Martin is the one that I'm going for dinner with, and it'll produce a next token distribution that will focus on the likely places that we are going. So this matrix, because it's represented in a compressed way, yet the models respond to everything, every prompt. How do they do it? Well, they go back to what they've been trained on, interpolate there, and use the prompt as sort of some evidence to compute a new distribution.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“Represent, even with a trillion parameters. So, what in an abstract sense what is happening is Models get trained on certain data from the training set and certain subset, a small subset of these rows, you have reasonable values. The next token distribut Whenever you give the prompt something new, then it'll try to interpolate with what it has learnt and what's there in the new prompt and come up with a new distribution. But it's basically, so it's more than a stochastic parrot Is sort of Bayesian on this subset of the matrix that it has been trained on. So, when I say I'm going out for dinner with Martin. I'm reasonably sure that it has never encountered that phrase. It's trading data, right? But it has encountered variants of this phrase.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“Next tokens or 50,000 tokens, then the size of it, the number of rows in this matrix is more than the number of atoms across all galaxies that we know of. So clearly we cannot represent it exactly. Now fortunately, a lot of these rows do not appear in real life and arbitrary collection of tokens, you are not going to use that as a prompt. Similarly, you saw a lot of these rows are absent and a lot of the column values are also zero. When you say the cat sat on the, it's unlikely to be followed by the token corresponding to, let's say, numbers. Or an arbitrary collection of because there are only a very small subset of tokens that can follow a particular prompt. So this matrix is very, very sparse. But even after that sparsity and even after removing the sort of gibberish prompts, the size of this matrix is too much for these models too.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“Okay, so yeah, let's start with that matrix abstraction. So, the idea behind the matrix is you have this gigantic matrix where every row corresponds to a prompt. And then the number of columns of this matrix is the vocabulary of the LLM, the number of tokens it has that it can emit. So for every prompt, this matrix contains the distribution over this vocabulary. So, when you say the cat sat on the, you know, the column that corresponds to mat will have a high probability. Most of them will be zero. But, you know, reasonable continuations will have a non-zero probability. And so you can imagine that there's this gigantic matrix. Now, the size of this matrix is, you know, if we just take just the old first generation GPT-3 model, which had a context window of 2000 tokens and a vocabulary of 50,000”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“So, this is something that I really appreciate about your work. And so the thing that really struck me is as soon as these things showed up, you actually got busy trying to have a formal model of what they're capable of, which was in stark contrast to what everybody else was doing. Everybody else was like, AGI, these things are going to recursively self-improve. Or they'll say, oh, these are just stochastic parrots, which doesn't mean anything. So everybody had rhetoric. And sometimes this rhetoric was fanciful. And sometimes this rhetoric was almost reductionist, like, oh, it's just a database, which is clearly not true. And the thing that really struck me about your work is you're like, no, let's figure out exactly what's going on. Let's come up with a formal model. And once we have a formal model, we can reason about what that means. And then, you know, in my reading of your work, I kind of break it up into two pieces.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“Yes, in some sense, progress is plateauing. It's like the iPhone, you know, with the iPhone came out. Wow, what is this thing? And the early iterations, constantly we were amazed by new capabilities. But the last seven, eight, nine years, it's maybe the camera got a little bit better or one thing changed here or memory is more. But there has been no fundamental advance in what it's capable of. You can sort of see a similar thing happening with these LLMs. And this is not true for just one company and one model. You look at what OpenAI is coming up with or what Anthropic Google. Or all these open source Chinese model or Mistral, the capabilities of LLMs has not fundamentally changed. They've become better, right? They've improved, but they have not crossed into a different realm.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“Just when ChatGPD was released, it was nice. It could write poems, it could write limericks, it could answer some hallucinating questions. But the capabilities that have emerged now, that pace has been very sort of surprising to me.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“What has most surprised me the pace of development. So GPD3 was a nice parlor trick and you had to jump through hoops to get it to do something useful. But starting with Chat GPD was an advance over GPT-3. And then you had all these things like chain of thought, instruction following, GPT-4 really made it polished. And the pace of development has really surprised me. Now, when I started working with GPT-3, I could sort of see what its limitations were, what I could make it do, what I couldn't make it do. But I never thought of it as What these LLMs have become for me now and what have become for millions of people around the world. We treat these models as our co-workers almost like an intern that you constantly chatting with them, brainstorming, making them do all sorts of work, which we couldn't imagine, you know.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“So then I started in this journey of developing a mathematical model trying to understand how it worked. That's been sort of my journey through this world of AI and LLMs because I was trying to solve this cricket problem.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“So, based on the new query, I would look through my set of natural language queries. I have about 1,500 examples. And I would pick the six or seven most relevant ones. And then that and the structured query I would send as a prefix and the new query, then GPD3 magically completed it. And the accuracy will be very high. So that had been running in production since September 2021, you know, about 15 months before ChatGPD came. And the whole revolution some sense started, and Iraq became very popular. But this is something sort of I accidentally did in trying to solve that problem for QuickInfo. Once Once I built it, I was thrilled that this work, but I had no idea why it worked I stared at that transformer architecture diagram. I read those papers, but I couldn't understand how or why it worked.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“So I got early access to GP3, you know, getting access those days was difficult, but somehow I got it. But soon I realized that no, I cannot really do it. Because Statsguru, the backend databases were so complex. And if you remember GPT-3 had only a 2048 token context window, there was no way in hell I could fit the complexities of that database in that context window. And GPT3 also did not do instruction following at that time But then in trying to solve this problem, I accidentally invented what's now called rack. Where, based on the natural language query, I created a database of natural language queries. I created a DSL, which then translated into a REST call to statscrew.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“I appreciate So, I'm still friendly with the people who run ES Ventrick and further, the editor in chief, whenever he comes to New York, we meet up, we go out for a drink. And so he was here in 2000. So now the story shifts to how LLMs and me sort of met. So January 2000, right before the pandemic, he was here. And I again said, why did you do something about Statsguru? And he looks at me and says, why don't you do something about Statsguru? He was kind of joking, but he thought maybe, you know, I had some. Ways to fix the interface. Anyway, then the pandemic hit, the world stopped. But in July of 2020, the first version of GPT-3 was released. And I saw someone use GPD Write a SQL query for their own database using natural language And I thought, can I use this to fix Stats Guru?”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“And so, you know, we built cricket at the very start rich sport. You'll think baseball multiplied by a thousand. And we had built this free searchable stats database on Cricket called Stats Guru. And this has been available on trick and for since 2000. But because you can search for anything, everything was made available on Statsguru. And, you know, you can't expect people to write SQL queries to query everything. So how did we do it? Was a web Where you could formulate your query using that form. And in the back end, that was translated into SQL query, got the results, and got it back. But as a result, that because you could do everything, everything was made available, the web form had like 25 different checkboxes, 15 text fields, 18 different drop downs. The interface was a mess. It was very daunting.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“But in the 90s, I was one of the people who started this portal called CrickInfo. And Krakenfo, at one point, it was the most popular website in the world. It had more hits than Yahoo. That was before India came on.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“Okay. So, yeah, as Martin said, my background is very similar to his. We come from doing networking. So my PhD thesis, my sort of early work at Columbia has all been in networking. But there's another side of me, another hat that I wear, which is both an entrepreneur and a cricket fan. I was going to say, don't you?”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“Then your prediction and entropy reduces. And you can arrive at an answer which you're confident of and which is correct. And the algorithms are pretty much the same way. That's why chain of art works. What happens with your thought is you ask the LLM to do something chain of thought. It starts breaking the problem into small steps. These steps, it has seen in the past. It has been trained on. Maybe with some different numbers, but the concept it has been trained on. And once it breaks it down, then it's confident. Okay, now I need to do A, B, C, D, D, and then I arrive at this answer, whatever”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source
“You have no idea. You can have some vague idea given the two numbers, right? And so in your mind, the next token distribution of the answer is going to be diffuse, right? You don't know, you have maybe a vague guess if you are mathematically very good. Maybe your guess is more precise, but it's just going to be diffused. And it's not going to be the correct answer. But if you say, can I write it down and do it the way we have learned multiplication tables, now you know exactly what to do next step, you write 769 and then 1025, and then you know exactly. So at each stage of that process, your prediction entropy is very low. You know exactly what to do because you have been taught this algorithm. And by invoking this algorithm, saying, okay, I'm not going to just guess the answer, but I'm going to do it step by step.”
2025-10-13 · a16z Podcast · Columbia CS Professor: Why LLMs Can’t Discover New Science · IDENTIFIED FROM THE TRANSCRIPT · source