YouSaid · the spoken record

Harrison Chase

lines on the record: 27
first: 2024-03-28
most recent: 2024-03-28
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“Mine is really, really exciting. And I think a similar thing could maybe, I think you can think of like personalization also as like what this would look like to end users and maybe like consumer facing apps. So you start with like a generic application that does the same thing for everyone, but then it maybe learns to search the web differently for me and ELAD or something like that. And so I think that's concretely how it could manifest.”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“So, and I think that actually tackles like I actually see a lot of similarities between that and Lang Chain Lang Smith in some way. And I think it's all towards this idea of like, so DSPY or DSP or whatever, it's basically this idea of like optimization. You have kind of like inputs, outputs. You then have your application, which they similarly think as multiple steps. And you basically optimize your application through a variety of different ways. The main one of which I would say is probably FewShot examples, although we'll probably do a webinar with Omar and he can correct me if I'm wrong. And I think the idea of continual learning is basically doing that optimization, but in an online manner where you don't have ground truth necessarily, but you get feedback from the environment, thumbs up, thumbs down if things are good. And so I think, yeah, that kind of like optimization loop, whether offline or online.”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“Yeah, absolutely. I think at a high level, it would basically allow the application to automatically get better over time. And it could get better in the sense that it's just more accurate. So, you know, it first does a mistake. You then tell it that it made a mistake and it automatically kind of incorporates that either as a few shot example or update to a prompt, but it starts learning from its mistakes and its successes as well, right? There's a really cool project called DSPY or DSPI. I don't know how to pronounce it, but it's out of Stanford.”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“Memory is a big interest of ours. And so I hope that will have some kind of breakthroughs there. I think a lot of the, specifically around learning from interactions, incorporating that back in at a user level, in a similar vein, also this type of more like system level memory, I think, is really interesting and building up building towards this idea of almost like continual learning. So there's like, can you learn from your interactions? And you can do that in a variety of different ways. This may just be where we sit in the ecosystem, but one exciting and probably under talked about ways is just idea of building up few shot example data sets and really using those. I think it's much faster and cheaper than fine-tuning models. It's easier to do than trying to like programmatically change the prompt in some way. That's still kind of like a bit of a art. And so yeah, continual towards continual learning.”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“Think the most exciting stuff is that the application and UX layer right now. I think that's where the most exciting stuff is there. One of the, I don't know if this is maybe more the capabilities side-ish, but like memory, I think, is super interesting, especially like personalized long-term memory. I don't know if it's necessarily tooling so much that needs to be built there as it's just like an application in a UX that's really focused on that. And if I wasn't doing Ling chain, if I was starting a company right now, I'd probably start something at the application layer and it would probably be something that really takes advantage of like long-term memory.”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“Fun story about that, the original models that we launched with OpenAI actually deprecated like a month ago. So the actual original link chain, you can't run because the models don't exist anymore. I think we see increasingly interest in open source, but the reasoning abilities are still just like lagging behind Clod 3 or GPT-4. And I think for a lot of the applications that it probably depends on the types of applications that you're building, but a lot of the applications that Linkchain is focused on with this kind of like reasoning aspect, those are just so crucial. And I don't think we see super compelling. I still don't think we see super compelling reasoning abilities in the open source models. And maybe that's one of my hot takes, but I think for a lot of the lane chain apps, the open source”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“We see people experimenting with it. I think the only real place where they're doing it is when they've reached like really critical scale, which I still don't think is that many applications to date. I think there's a lot of difficulties with it. One's like gathering the data set for it. And so I think a lot of the things we have in Ling Smith tackle a lot of these issues, but like gathering the data set for it. So like having that data visibility and starting to curate that data set, evaluating the fine-tuned model. So like evaluation and testing is a huge pain point there that we're trying to tackle in a few ways. The third is just like, yeah, back to this point of people are still just like experimenting so rapidly. It's much harder to change a fine-tuned model than it is to change a prompt or even changed few shot examples. And so I think we're seeing more and more people use few shot examples, but not a ton graduating to the fine-tuning just because, yeah, I think.”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“Text splitting and chunking and all of that. And I don't know exactly how that will change. Like, will you still do that? But you now just retrieve the whole document. Like we have a concept in Lang chain of like a parent document retriever, which basically creates multiple vectors for each document. So maybe you just do that. Maybe you still maybe you chunk it up into larger chunks and just retrieve those larger chunks. Maybe use a traditional search engine like Elasticsearch or something. I'm not sure. That's probably the place I have the least confidence in.”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“Did this kill Rag? I think there's a few things actually just today. One of our team members, Lance Martin, there's that like everyone's doing the needle in the haystack thing and now all these models are like green across the board for whatever reason. They've all figured it out. But I think that actually really doesn't reflect a lot of rag use cases in my opinion because the needle in the haystack is like, okay, given this long context, can I find a single information point? But oftentimes rag is about seeing multiple information points and then reasoning over them. And so I think with a benchmark he released is exactly that. Like as you increase the number of needles, performance goes down, as you might expect. And then also when you ask it to reason rather than just retrieve the performance drops as well. And so I think there's more work to be done there. And then I think another thing is just around the ingestion for rag in the indexing process. Like a lot of the attention has been paid to.”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“Very good question that a lot of people are thinking about who are a lot smarter than me. I think, I mean, a few thoughts. I think like longer context windows definitely make single shot things much more realistic, like extraction of elements in a long PDF. You can do that one shot. Rag over a single long PDF or like five long PDFs. Okay, cool. You can do that. You can do that one shot. There are definitely things at scale that don't fit into a single context window. There are also things where it requires iterations. You need to decide what to do, interact with the environment, get that back. So this whole idea of chaining and agents, I don't like that's less around context windows and more around interacting with the environment and getting feedback. And so I don't think that's going anywhere. I think with respect to rag in particular, because I think that's where it often comes up.”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“Yeah, I think that's interesting. I think that's a real enterprise concern. I think a lot of the agents are still just figuring out how to make agents work. This is a different axis almost, but to the point around switching models, I do think we see a desire for this, especially when you start going to scale. So I think it's like make something work with GPT-4, but then, okay, you're rolling it out. Is that like, you know, are you really going to eat that much cost with GPT-4? Can you use GPT 3.5? Do you want to fine-tune? And so I think that transition is where we really start to see people thinking about switching models. There's definitely some switching models at the beginning. Like if you just want to play around with different models and see their capabilities. But I think the most pressing need to switch models happens when you go from prototype to scale. Cost and latency would be differentiators there as well.”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“Context windows one as well. So I think this gets to like, yeah, what's the right context that you can be passing if it's longer, you know, if that changes, then that changes, that doesn't like that changes the whole architecture of your application modalities one.”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“It's not as easy as it seems like it should be. And I think the main thing is that the prompts still need to be different for each model. I do think the prompts will probably start to converge in the sense that if you think the models are getting more and more intelligent, then like hopefully these small idiosyncratics don't matter as much. And as more and more model providers start supporting the same things, then that will make it easier. And what I mean by that is, you know, so many prompts for OpenAI, which is the leading and most used one, use function calling.”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“Support, you're maybe trying to explicitly kind of like resolve a ticket or something like that. And the recommendation bit is a bit more focused on a user's preferences and what they like. And I think we've seen a few, I think we've seen a few things emerge there. But I'd say customer support encoding are the two. Klarna as well, you know, they came out and had a pretty good release.”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“I think on the customer support side, that's a pretty obvious use case. I think Sierra has emerged there and is doing quite well there. I think, yeah, the cognition demo was very impressive. I think they did a lot of things right. I think they really nailed a really interesting UX, and that was maybe one of the things that I was most excited about. And then obviously it seems to work very well. And so I don't know exactly what they're doing under the hood. But those type, like coding problems in general, we see a lot of people working on. I think there's a really nice feedback loop that you can get by just like executing the code and seeing if it works. And, you know, as well as the fact that people building it are developers. And so they can test it. Coding, customer support. There's some interesting stuff around recommendation chatbots almost. So I draw a distinction between that and customer support where with”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“And then so now retrieval is like a multi-step process and more there and it's explicitly around query analysis. Few shop prompting and that whole data flywheel, I think we're starting to see come into play more on the agent side. I kind of alluded to this earlier, but I think the way that we've kind of thought about things is there's kind of like chains, which are sequential steps. You're going to do this and then you're going to do this and then you're going to do this. And you're always going to do those in this exact sequence. And then, you know, last March or April or whenever auto GPT came out and it was like we're literally just going to run this in a for loop and it's going to be this autonomous agent. And I think the things that we see making it into production and informed a lot of the development of Lang Graf is something in the middle where it's like this controlled state machine type thing. And so we've seen a lot of that come out recently. And so I'd maybe call out that as like one thing that we've really updated.”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“That does generally match, I think Langchain from the beginning has always been focused on those types of applications and not only the open source, but also Lang Smith, the platform. So I think a lot of the emphasis that we put into the testing and the observability is really focused on these multi-step things. We've always been focused on those. Probably it's generally true in the market that there's been more of a trend towards those. But from our perspective, we've always been focused on those. And so I think that hasn't been as dramatic. I think there has been interesting things within that that have emerged just calling out like a few things within RAG. I think we've seen really interesting and advanced query analysis start to come into play. So you're not just passing the user question directly to an embedding model. You're maybe doing some analysis on it to figure out which retriever should I send it to or like what is the bit that I should search? Is there kind of like explicit metadata filter?”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“And I think there's a few different ways to do this. Like, I think open AI implemented it in their chat GPT, where in the way I think it does it under the hood is it basically has functions that it can call to say like, remember this fact or delete this fact. And so that's a really interesting active loop that the agent is engaging in where it explicitly decides what it wants to remember and what it doesn't want to remember. I also think one thing that I'm bullish on is a more kind of like passive background process that kind of looks at conversations and almost like extracts insights. And then you can use those insights in kind of like future conversations. And I think there's pros and cons to each. And I think it speaks to the memory in general. I feel is like a field that's just like super, super nascent. I actually am underwhelmed that the amount of like really interesting stuff that's going on there. And so I think a bunch of different approaches. No, no.”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“Also, I think it's super interesting. I have a few thoughts here. So I think there's maybe two types of memory and they're related, but I'll draw some distinction between kind of like system level procedural memory and then like personalization type memory. So system level memory, I mean more like what's the right way to use a tool? What's the right way to accomplish this objective?”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“Know to do that, and then it can kind of like run in a cycle. But we see a lot of shortcomings there. And so I think planning the ability of LLMs is a big one. And that'll get better over time. The last one is maybe a little bit more vague, but I think even just as builders, we're still figuring out the right ways how to make all these things work. What's the right information flow between all the different nodes in order to get those nodes, which are typically an LLM call, to work? Do you want to do few shot prompting? Do you want to fine-tune models? Do you want to just work on improving the instructions and the prompt? And so I think there's a lot of how do you test those nodes? That's a big thing as well. How do you get confidence in your LLM systems and LLM agents? And so I think there's a lot of workflow around that to kind of like be discovered and figured out.”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“Of like imparting their domain knowledge about how information should flow through these components. Because I think the LLMs by themselves still aren't able to reason fully about how that should happen. And I think we see a few kind of like a lot of research is actually around this, I would say, in the academic space, specifically I think there are two different types of research papers around agents that we see. We see some around like planning for agents. So there's a bunch of papers that do kind of like an explicit planning step up front. And then there are other research papers that do a bunch around reflection. So like after an agent does something, is this actually right? How can I kind of improve upon that? And I think both of those are basically trying to get around the shortcomings of LLMs in that in theory they should do that automatically, right? Like you shouldn't have to ask an LLM to plan or to think about whether what it's done is correct.”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“Yeah, that's a really good question. I think there's a few things. One, I think figuring out the right UX for a lot of these things is still an open question in my mind. And that's not necessarily something we can help with. I think there's a lot of exploration that applications need to do to figure out how to communicate what these agents are good at and bad at to end users and expose ways to maybe let them course correct and see what's going on. And so, you know, I think we try to emphasize a lot of this observability of intermediate steps and even correcting intermediate steps, but there's a lot of experimentation around UX that I think needs to happen. Another big part, I think, is basically the planning ability of the underlying LLMs. I think that's probably the biggest, I think when we see people building agents that work right now, it's often breaking it down into a bunch of smaller components and”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“Structured outputs and the importance of those has remained relatively stable. But then the way that you put things together and the kind of like patterns for building things has definitely evolved over time from simple chains to complex chains to then these kind of like autonomous agents to now something maybe in the middle of like complex state machines or graphs or something. And so it's really that upper layer, which is like the common ways to put things together that I think we've seen the most rapid kind of like churn.”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“Type to production, they wanted to customize a lot of these bits. And so we've invested a lot in a lower level kind of like chaining protocol. So lane chain expression language and then in different protocol lane graph, which is something we're really excited about. And that's more aimed at basically graphs that are not DAGs. So, you know, all these agents are basically running an LLM in a loop. You need cycles. And so Lane Graph helps with that. And so I think what we've kind of seen is the underlying bits of There's all these different integrations. And like, you know, there's LLMs, vector stores, and sometimes they change, right? When chat models came out, like that was a very big change in the API interface. And so we had to add a new abstraction for that. But those have, especially over the past few months, remained relatively stable. We've invested a lot in this underlying runtime, which emphasizes a few things, streaming.”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“That's been a fun exercise. So, I mean, if we go back to the original version of Langchain, what it was when it came out was essentially three kind of like high-level implementations. Two were based on research papers and then one was based on Nat Friedman's like Natbot type of agent web crawler thing. And so there was some high level kind of like abstractions. And then there was a few like integrations. So we had integrations with, I think like OpenAI, coherent hugging face to start or something like that. And those two layers have kind of like remained. So we have 700 different integrations. We have a bunch of kind of like higher level chains and agents for doing particular things. I think the thing that we've put a lot of emphasis in to your point around kind of like what's remained constant and what remains and what's changed is like a lower level kind of like abstraction and runtime for joining these things together. One of the things that we pretty quickly saw was that as people wanted to improve the performance go from prototype”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“Evaluation, monitoring, and all of those types of things. And so, you know, what lang chain is has evolved over time as the company's grown.”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT
“Yeah, absolutely. So how I would answer the question what ling chain is has kind of evolved over time, as has the entire landscape? Lang chain, the open source package, started, yeah, as a side project. So my background's in ML and MLOps. I was at my previous company. I knew I was going to leave. I didn't know what I was going to do. This was in September, October of 2022. And so I went to a bunch of hackathons, a bunch of meetups, chatted with folks that were playing around with LLMs and saw some common abstractions, put it in a Python project as just fun side project. Turned out to strike a chord, be fantastic timing, you know, chat GPT came out like a month later. And it's kind of evolved from there. So right now, Ling chain, the company, there's really two main products that we have. One is the Lang Chain open source packages and happy to dive into that more. And then the other is Lang Smith platform for testing.”
2024-03-28 · No Priors · Open sourcing AI app development with Harrison Chase from LangChain · IDENTIFIED FROM THE TRANSCRIPT