YouSaid · the spoken record

Garrett Lord

lines on the record: 71
first: 2025-08-24
most recent: 2025-08-24
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“I didn't get a PhD in AI or math or physics, and I haven't trained myself to be a frontier mouse, but there is a lot to each unit of data, whether it's improving. There's a ton of science and research out right now around how do you make sure that the data that you're producing is improving the model. And it's very hard for Modable to understand, they can really care about, to zoom out, they care about three things. They care about quality first. You have to have high quality data. And if you imagine you're training a model, like teach them a student and you're giving it the wrong data, it's extremely challenging to overcome that. So quality is first and foremost. And then the other huge problems you have is like volume. Like how do you generate thousands of pieces of data in the most advanced domains of chemistry and mathematics and physics?”
2025-08-24 · Lenny's Podcast · Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO) · IDENTIFIED FROM THE TRANSCRIPT · source
“What is kind of like think back on your school days? Like, how do you get an A on your 5,000 word paper? Well, there's like a great introductory statement and there's scientific proof, you know, like, so you can build a rubric that allows a model to sit in the middle and actually auto evaluate responses. We're seeing a lot of rubrics work as well.”
2025-08-24 · Lenny's Podcast · Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO) · IDENTIFIED FROM THE TRANSCRIPT · source
“Okay, and then you also have like multimodal work, like audio, like classifying music and understanding. We're engaging like thousands or not thousands, like probably hundreds of top music students at, you know, the Weed Muses Schools in the country who are improving models understanding of music. And you also have the thing called, which we haven't talked about here, like a rubrics and a rubric like models are you can put a model in as a judge. Like you can, if you, what is a good educational design? Or what's a good MRI result? And instead of having some of these, in some of these domains, you actually don't have a guaranteed correct right answer. And so models can sit in the middle as a judge and actually understand”
2025-08-24 · Lenny's Podcast · Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO) · IDENTIFIED FROM THE TRANSCRIPT · source
“A trajectory is basically just like the entire environment that is collecting what you're doing. So it's your screen, it's your mouse Yep.”
2025-08-24 · Lenny's Podcast · Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO) · IDENTIFIED FROM THE TRANSCRIPT · source
“You know, seeing the latest models fail on. So you have, yeah, I think that gives you the gamut. You also have, you know, we talk about professional domains, like these reinforcement learning environments. There's a bunch of papers out there that basically speak to people narrating over their step-by-step tool use. So as they go to solve a problem from start to finish, interact with multiple different service areas, interact with multiple different tools, you know, they're like, you know, there's papers that talk about this, but talking over what they're doing, actually following and screen recording where their mouse is going, how they're problem solving. When they run into a roadblock, what do they do? They really want to understand how humans think.”
2025-08-24 · Lenny's Podcast · Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO) · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, like, a great example, let's take a nonverifiable domain like education. So there's like a PhD student Rachel on the network. She got her PhD from the University of Miami, spent two decades as a teacher teaching students in the eighth grade. And she was an adjunct professor at a local community college in the field of education. And so she is interacting with the state of the art models in educational design. So actually trying to understand what is the best way to teach people and like how do you frame the how do you spot incorrect issues in a model and the way that they're like training people and help the models understand the forefront of educational design with the hands-on experience of being an eighth grade teacher for 10 plus years and having a PhD in education? So that's an example of like, you know, you can have that all the way down to like a verifiable engineering problem that you're seeing the latest, you know.”
2025-08-24 · Lenny's Podcast · Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO) · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so War for AI talent is very expensive, super privileged and proud to be working with him. And so, you know, each unit of data, we have to build on an environment for them to actually create the data. Then we have to understand at a unit level we're trying to approximate the actual gain from that piece of data and whether it can improve in a particular capability area. And then we're also focused on evolving the use cases to also follow what the model builders want, which is they want more real world tools.”
2025-08-24 · Lenny's Podcast · Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO) · IDENTIFIED FROM THE TRANSCRIPT · source
“Brandonly experience and treating people like experts, like PhD students expect to be treated different than lower cost international labor with a different work expectation. And so these PhDs come into a community. We have an instructional design team and an assessments team that's going through and basically iteratively helping them understand how to use the tools that we built and how to interact with the latest models. Then they go in and start actually creating data and that process is On our side, the model builders they want to know that the data we're producing is high quality. So we have our own research team, our own post training team, hired a gentleman from Meta that a lot of the post training over there.”
2025-08-24 · Lenny's Podcast · Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO) · IDENTIFIED FROM THE TRANSCRIPT · source
“Great example is a public paper called GPQA. So for the engineers out there that want to read about it, like essentially the crux of the paper is you break the model, you provide a ground truth the right answer to the question, you provide the step-by-step reasoning steps. So you might imagine because models are non-deterministic, like the model can get the answer right once, but it might not get the answer right, you know, three out of five times. So you actually prove where the model's failing. You actually break down into like, where is it failing? You know, maybe it can get the, it knows the question, but it can get the right answer, but the actual steps to get there are wrong and they're really focused on like the steps to get there. So there's like 10 steps in a math problem, right? Like step six through 10 is wrong. And so like, how do you fix the actual steps? And what are they doing? So they're going in. We put them, you know, we really focused on calling.”
2025-08-24 · Lenny's Podcast · Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO) · IDENTIFIED FROM THE TRANSCRIPT · source
“And I often say we help point out where the models are weak So, in order to break a model, it's pretty tough for the average person to break a model and get an incorrect response. But if you're a PhD in physics, like you can go in in multiple kind of subdomains of physics and prove where the model is actually breaking, either breaking in its reasoning steps or it's where it's broken and its ground truth right answer, or we start throwing tools in there or needing to follow some step-by-step process. And it's, I wouldn't say it's easy for them, but the average person cannot break the models. And that's where we really come in.”
2025-08-24 · Lenny's Podcast · Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO) · IDENTIFIED FROM THE TRANSCRIPT · source
“Capability areas in the economy, right? And so that generally speaking right now is focused on advanced STEM domains, advanced science and math domains, and then the kind of derivative functions of accounting, law, medicine, finance, where they want to make the models more capable. And then the work that we're doing, I think, to come full circle to your question, like we're doing work across so many domains. I mean, we have millions of bachelor students that are being used for work in like audio, work in customizing a model depending on the voice and tone where you are geographically in the country. What do women versus men prefer, all the way to the most advanced PhDs domains out there?”
2025-08-24 · Lenny's Podcast · Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO) · IDENTIFIED FROM THE TRANSCRIPT · source
“Market something to make it more abstract is like it used to be generalist work. Like a lot of the market before the model started to get better was leveraging talented international lower cost labor to do basic generalist tasks. But really what's happened is the models have gotten so good that the generalists are no longer needed. Like what they really need is experts. Experts across every area that the models are focused on. And really you could think about these model builders as they're focused on like the most economically valuable”
2025-08-24 · Lenny's Podcast · Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO) · IDENTIFIED FROM THE TRANSCRIPT · source
“Are like real unique Proposition as a business is the fact that we like have an engaged audience. We have 18 million professionals across, you know, we have 500,000 PhDs. We have 3 million master's students. We're a global platform. And so depending on kind of what you're looking for across any area academic knowledge, what is the definition of a PhD? It's essentially to like be at the, how do you get your PhD? You defend your thesis. Defending your thesis means generally speaking, like you have proven that you have extended the world's knowledge in a particular domain. And so the ability to hyper-target this audience into chemistry, math, physics, biology, coding, and really touch parts of human knowledge that have never before made it to the internet is really where we excel. And I would say that when you talk about the label,”
2025-08-24 · Lenny's Podcast · Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO) · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, yes and no, because take, for example, trajectories or you want to be able to do people use flight search or like an accounting end-to-end process, or you want to be able to conduct biological experiments. You need actual trajectory data. You need to, there's still very much a lot of the labs are still, they have points of view on what data collects. It's evolving very quickly. But I think reinforcement learning is really like preference ranking, right? Like which question do you like more questionnaire or question B? SFT data is like a prompt and a response. And obviously the labs are very focused on these like thinking or reasoning models. So in order to improve a reasoning model, you need to actually have the step-by-step instructions of which when you interact with a lot of these frontier models, they struggle in very advanced domains. And so, you know, I think there's a variety of data.”
2025-08-24 · Lenny's Podcast · Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO) · IDENTIFIED FROM THE TRANSCRIPT · source
“And so that's the state of art of models. Most of the gains that are happening from models right now are coming from the push trading side of the house. And there's just an incredible amount of demand to stay at the absolute frontier of where models are going.”
2025-08-24 · Lenny's Podcast · Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO) · IDENTIFIED FROM THE TRANSCRIPT · source
“Improves the state of our capabilities in our models. And you can see a lot of these popular benchmarks on what are called model cards, you know, when Lama Force released, you'll see like the benchmarks across various domains. And each one of the research teams inside of the labs have different. Use cases basically they're running experiments. Think like the scientific process they have like a hypothesis around how to improve the model, they're trying to collect small pieces of data to see if that hypothesis works out, if that hypothesis is proving true, then they expand the overall collection of the data in that effort. And it can look like reinforcement learning environments. It could look like trajectories. Audio and multimodal, it can be text based, like prompt response pairs It can also be like”
2025-08-24 · Lenny's Podcast · Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO) · IDENTIFIED FROM THE TRANSCRIPT · source
“Pursuit of sucking up everything that was on the internet. That was the pre training side. And there was a lot of gains from pre-training. Like models continue to get better. And about 18 months ago, 24 months ago, we started to really see an asymptoting of gains coming from because they had essentially sucked up all the knowledge on the internet. And so labs really shifted towards most of the gains now coming from the post-training side of the house. And what post training is. It's augmenting and improving the data they have across every discipline or capability area that they care about. So take coding or mathematics or law or finance. They are focused on collecting high quality data.”
2025-08-24 · Lenny's Podcast · Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO) · IDENTIFIED FROM THE TRANSCRIPT · source
“So I think it's helpful to take, I guess, a step back of what is training a model look like. So there's really two primary functions. There's a pre-training and a post-training process in training a model. And for a long time, these AI providers or LLMs or frontier labs were focused on basically sucking up more and more information on the pre-training side of the house. And that's basically the entire corpus of like a written human knowledge. That's not just written, but like every YouTube video, every book. Basically,”
2025-08-24 · Lenny's Podcast · Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO) · IDENTIFIED FROM THE TRANSCRIPT · source
“Not what we're hearing from our employers. This is just enabling human beings to be even more productive. Used to put a Google search on a skill on your resume because you like grew up with Google being like AI native. Young people are at a huge advantage.”
2025-08-24 · Lenny's Podcast · Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO) · IDENTIFIED FROM THE TRANSCRIPT · source
“Of the largest expert network in the world. We have this massive strategic advantage, which is like no customer acquisition costs. The only mode in human data is access to an audience.”
2025-08-24 · Lenny's Podcast · Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO) · IDENTIFIED FROM THE TRANSCRIPT · source
“Will never be a time like this. I've never seen anything like it. I doubt I'll ever feel anything like this in business again, where there's unlimited demand. How do you make sure that three months or not, six months right now, you have like no regrets? Get on the plane to go talk to a customer, make delay night push, check the data six times over again”
2025-08-24 · Lenny's Podcast · Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO) · IDENTIFIED FROM THE TRANSCRIPT · source