YouSaid · the spoken record

Beyang Liu

lines on the record: 82
first: 2026-01-20
most recent: 2026-01-20
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“We had a coding tool that would do coding autocomplete and the kind of like ban our top line metric there was completion acceptance rate you know like given that I suggest this change to the user what is the likelihood they're going to accept that seems like you know bulletproof right but actually like I think in building that we ended up over optimizing that to a certain extent because there's and like any metric you choose there's going to be a way to game it”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“Any eval set, like what are you trying to capture? If you're building an end user product, at the end of the day, what you care about is the product experience. And so you construct the eval set to kind of proxy the vibes of the user using the product. And by definition, that means your eval set is always lagging a little bit from the frontier because it takes time to like distill what is a good product experience into a set of evals. And we've had multiple times in our past where we've picked a number. Just to take an example, like with back in the kind of like code completion days of 2023 or whatnot.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so my take on EVALs is evals are definitely effective as a sort of like unit test or a smoke test. Because if you push a change to your agent and it breaks something, you want to know, right? Like if there's an important workflow that you're like, hey, this should work reliably well because if this doesn't work, then probably a lot of other things break. And that's a great instance where you want an eval that will alert you when it goes from green to red. I think where it gets hairier is treating evals as a kind of like optimization target because.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“You might have different answers, but I think if you construct the agent right, they're not going to be wildly different. So, for instance, we have a subagent that's designed to search for things, like uncover relevant context. And it is a bit of dice roll every time, right? Like it takes a slightly different trajectory. It might search for different things. But it's to the point now where if I want to find something in the code base, I have like 99% confidence that this thing will eventually be able to kind of like stochastically iterate to the right answer. And so in that... In that way of thinking, it's like, yeah, how it gets there might vary. But if I wanted to do a specific thing, it's reliable enough that I can invoke it.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“Can I just push on this? Because I mean, listen, call me a traditionalist But for me, computer infrastructure is compute network and storage, right? And like databases. And these are resources that are abstracted. So give me storage, give me network. The semantics like what actually happens I write, right? That's like my code right here where like. Figure it out for me. It's like we're advocating actual logic and correctness. It just feels like in a way like a little bit. You know, like in your case, for example, if you pick up, you know, let's say you're using model the 2.1 and then you go to model v2.2. You have wildly different answers, right? It's almost like a new instruction set or something.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“The non determinism is something that people struggle with a lot. But so for me, I actually do. Historically, pre AI. Like when you think about computer systems, the basic unit of composability is the function call in programming, right? So it's like when you think about your system, it's like this function calls out to these other functions and those other functions delegate to these other functions. I do think there's still an analog to that in the agent world. Like the agent is really the analog of the function, but just updated or generalized to AI.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“Sounds so hard to me. Like, this is the first time in computer science I can think of where we've actually abdicated correctness and logic to, right? Like in the past, it was a resource, right? Yeah. Like whatever. It's not logic. It's like, okay, so maybe the performance is different. Maybe the availability is different. But like whatever I put in, I'm going to get back out, whether it's a database or compute or whatever. But now we're like. Figure out this problem for me. So you're kind of abdicating core logic and correctness. Your unit test comes back with works 45% of the cases.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“And so what we view as the kind of atomic composable unit is not the model. It's this thing called the agent, which is essentially this contract of like user puts text in and gets certain behaviors out. And that agent is really a product of both the model, plus all these other things that I just listed. And so when it comes to figuring out what models we want to use, it's not so much like, hey, we want to use the latest quote-unquote frontier model from XYZ lab. It's really about, hey, what behavior do we want the agent to take, or in some cases the sub-agent, and how do we find the right model that enables that agent to do its job?”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“Oh, for sure, for sure. It's like if you have an agent harness, like a set of tool descriptions, and you swap out the model, then there's no guarantee that that thing is going to work well with the model that you swapped in.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“And with the same model with wildly different tool descriptions and system prompts, you actually get completely different behaviors out of that model.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“Okay, so let me explain. So, like, when you're interacting with an agent, at the end of the day, you care about how that agent is going to respond to your inputs, what tools it's going to use, what sort of trajectories it's going to take, what sort of thinking it does. A lot of that goes back to the model, but it's not solely dependent on the model. There's a lot of other things that can influence how an agent behaves. There's the system prompt, there's a set of tools that you give it, there's a tooling environment, there's a tool descriptions, there's a sort of instructions that you give it for connecting to feedback loops.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so I would say our philosophy is not model centric. It's more agent centric. So we view the model as an implementation detail”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“I see. Interesting. That's a great way to put it How do you think about the difference between using somebody else's model, like one of the soda labs versus, yeah. Building your own model versus, you know, using an open source model. How does that fit in your philosophy?”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“Big prompt agent go at it. But then there's other types of development where it's like I want to build a brand new feature. We just shipped this code review panel in our editor extension. And that was kind of like a situation where I was like, I don't actually know what this review experience should look like because it's not me reviewing other people's codes, me reviewing agents' code, which is like a new workflow. And for that, I kind of did want like a more interactive back and forth interaction between me and the agent. I don't think it's necessarily like these two things don't have to be completely separate products, but they are distinct working modalities.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“Figure out what the software looks like as you go along. And sometimes it's the same person saying both things, right? Like when I go, there's some features where it's like implement billing where I'm like, okay, I know exactly what protocols we need to support and stripe integration. I know what feedback loops we need to hit. Then it's like, okay.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“So there's definitely different working styles, right? Depending on the task or maybe depending on the person, you talked to people using coding agents and some of them are like, I just want to write a paragraph long prompt and then have the agent go figure it out. I want to come back to something that's like mostly working. And then there's other people who say, like, actually, I don't want to do that because half the time I myself don't have a clear idea of what I want yet. The creative process is sort of one where you kind of like,”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“Can I dig philosophically into this just a little bit? So I had a conversation with somebody that works on cloud code, which is a very successful CLI tool. And this person's like, you know what we've done over time is we've literally just removed. Stuff between the user and the model. Like, that's it. Like, that's kind of like the way that we improve things. Are we just like do less? And let the model do more. And so I guess that makes a kind of intellectually or intuitively interesting kind of makes sense. But on the other hand, it seems expensive. You're like, here is. Is this state of the art model that costs a billion dollars to train? And like now it's just the user in the model. And so it's almost like that statement is almost contrary. Like that”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“But then it just kind of kept coming back up. And at one point, we're like, all right, let's just try it and see how it works. And we launched it. And it's been growing very quickly since then.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“But as we build more and more, we kind of realized there's sort of this efficient frontier that you can draw, this two by two grid. And one axis is intelligence, but the other axis is latency. And there's multiple interest points along this trade off curve. It's not just that having the smartest model makes your experience the best. The smartest model often tends to be a significant amount slower than other models on the market. And so we felt that there was like an opportunity for us to create like a faster top-level agent that couldn't do as complex of coding tasks, but it could do these targeted edits. And when we started to play around with these small fast models, we realized that, hey, actually, the inference costs are significantly lower. And that got us thinking, like going back to folks like my dad, right? Like he's just doing this stuff on the side. He doesn't want to spend hundreds of dollars per month to create these kind of like simple games. We're like, hmm, maybe there's like a model here. I think it started as a joke. Someone's like, we should just do ads and see how that works. And everyone was like, nah, that'll never work.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“It's really funny because I think we had this sort of reputation for being like the primo agent, super intelligent one, but we never had like a flat rate pricing model. We did pure usage-based pricing, and that also meant that there was never any incentive to switch to a cheaper model for users. So our tack was like the most intelligence, and you just pay for the inference cost.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“And so, my dad, who's never written a single line of code in his life, is able to just make a simple game that has him count the numbers. And then if he gets it right, the little rocket ship blasts off. So it's kind of interesting. It's a really interesting time to be building because even if you're building for professional developers as we are, a lot of the technology ends up being just kind of widely accessible.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“And original tack was like, okay, let's build this into some of the existing things that we created. But the more we started playing around with technology, the more we sort of came to the conclusion that this was actually truly disruptive and we should actually start from first principles to see build the agent from the ground up and see what tools we really need. So what we've arrived at is the coding agent, which works, I think, very well in large code bases, because again, we push this to a lot of our customers, but it's also great for hobby coding. Like I spun my data up recently on it, and he's been using it to create these like iPad games for our kid. Because, you know, typical Asian dad try to teach him math, right?”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“So I think your focus is really on large code bases. How is that structurally different from me coding my little home room to Yeah, so it's funny you mentioned that historically, the company's focus has really been on large code bases, but with AMP, we decided to build it almost completely separate from the existing code. And the reason for that was, one, we built AMP. AMP is really at this point like seven, eight months old. So we started AMP in around like February, March this year And that was right at the wave of this new type of LM hitting the world, the agentic tool use LM worked. Yeah, finally worked, right? Like after so many demo videos, finally there was a model that could actually do robust tool calling and compose that with reasoning.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“Agents, do you share that, or is that just kind of the outside? I would say there's certain things that we're doing that I think are quite unique. It was the top recently on one of these benchmarks, right? Yeah, I think there's like some startup out there that compares pull request merge rates or something. We managed to claim the top spot. That's awesome. Yeah. Yeah, it was very gratifying to see. But again, I would say that I think we're opinionated on some parts of our philosophy of building agents. And my own take is I think a lot of these opinions will soon become widespread. But there's other elements of what we're doing, which are like people like to read in a lot to AI these days. And sometimes it's just like, look, we actually did something very simple here at Yield of Good Results, and we shipped that and it works very well.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“Enabling humans to understand code because if you've ever worked inside a large code base, you know that that probably takes anywhere from 80 to 99% of the time. And then the remainder is when you actually understand the problem well enough to actually write the code. So that's where we kind of like built up our domain expertise. And then when LM sort of matured, it was something that we were always kind of like monitoring in the back of our minds. Originally looked at LMs and embeddings as a way to enhance the ranking signals that were incorporating into our search engine. And then when things really hit their stride with ChatGPT and all that, it was fairly obvious to us that there was a big opportunity to combine LMs, which were this amazing technology with a lot of the stuff that we'd built up to that point. And then I guess to round that out, finally, our latest product is this coding agent called AMP. What's interesting about that amp is it's kind of in view as like a very sophisticated kind of opinionated view on API.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“More efficient inside large organizations and to make the practice of actually building software way more accessible, primarily to professional software engineers, but I think our eventual vision was always to expand the franchise. And we started by tackling the key problem, which is”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“Talk like a systems guy. Yeah, yeah. For me, like this whole phenomenon of LMs encoding, it's almost like a homecoming of sorts because I really, I didn't know that. That's awesome. Yeah. Definitely taught me AI as well. She's a great teacher. Yes, that's amazing. I gotta say, that was like the one class. I was so happy I comped out of because I didn't think I would do well if I did pass the comp, I thought it would defeat me. I think a family comp too many times. There were those two. I was a TA for that class, actually. 228. 228. Yeah, that's right. That's what it was. Yeah, yeah. So cool. Great. So SourceCraft started code search navigation. But then now you've been making ways with AMP, which is like, you know, an agent, what we call it. So maybe we talk through a little bit about what you've been working on, maybe pre-AI and now just to level set. Yeah. So kind of like the history of the company is we were really built to make coding a lot.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“Details? Yeah. So background, I've been working on DevTools for more than the past decade of my life. I started SourceGraph about 10 plus years ago, brought the world's first kind of like production legit code search engine to market and push that to, I think, a good portion of the Fortune 500. Prior to that, I was a developer at Palantir. And I guess now it's like the early days, right? That's where I met my co-founder, Quinn, and we're working on data analysis software. And a lot of large enterprise code bases that were kind of like drop shipped into and realized that there was a big need for better tooling for understanding massive code bases. And then before that, I guess relevant now is I actually did machine learning as a concentration when I was doing my studies. So I did some computer vision research. No, that? Yeah, under Daphne Kohler at AI guy. Yeah, yeah. OGAI. Dawson Angler, like Tyler stuff. I thought you were a system.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“Biang, thanks for coming and joining. So, the topic today is AI encoding, but I mean, I would say you're one of the world's experts on this. And so we would love to kind of do a deep dive and kind of how you view the problem, how you view the solution, of course. Your co-founder and CTO of SourceCraft. So we'll talk a bit about that as well. Of course, we've got Guido. Thanks for being here. And so maybe just to start, we can do a bit of a background on you. And then we'll just kind of dig into”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“It's funny. Exactly. That's right. That narrative, I think, is largely been dispelled within our circles. But I think that it's sort of like taking on a life of its own in other circles and it's made its way to some of the halls of policymaking in the US. This is the old adage of like, you know, do you blame it on ignorance or malice? I honestly don't know, but it is clearly like nonsensical. And I think very much in the national interests to be still telling this story.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“You talk to some devs and they're like, you know, I've never been more productive, but coding isn't fun anymore. That's one of the things that we're trying to solve for it. It's like amazing new technology. It feels like magic. Never experienced anything like this before in my life. The narrative that was fun was like, this thing will just, you know, run our lives for us, or it's going to kill us all. Total annihilation, like Terminator. And there's just like absolutely no danger that this thing's going to, you know, acquire a mind of its own and try to reach out to the community and kill you.”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source
“Is the first time in computer science I can think of where we've actually abdicated correctness and logic to a Like in the past, it was a resource, right? So maybe the performance is different, maybe the availability is different, but whatever I put in, I'm going to get back out. But now we're like, This problem”
2026-01-20 · a16z Podcast · From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu · IDENTIFIED FROM THE TRANSCRIPT · source