YouSaid · the spoken record

Arthur Mensch

lines on the record
62
first
2024-04-29
most recent
2024-04-29
sittings or episodes
1
sources
podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

  1. “Of scale matters in the sense that if you spend more training compute, you can make the models more compressed. So you do need to have some compute to compress models. Now, scale isn't the only ingredient to the recipe you need to scale, but you also need to have proper data, otherwise you reach some data quality limit. You need to have proper techniques for training. I mean, people call it compute multiplier, I guess. How do you actually make some efficiency gain that are not costing you compute? Because compute is expensive. And so one of the things that we do at Mistral is to try and harvest this compute multiplayers.”

    2024-04-29 · The Twenty Minute VC · 20VC: Mistral's Arthur Mensch: Are Foundation Models Commoditising | How Do We Solve the Problem of Compute | Is There Value in the Application Layer | Open vs Closed: Who Wins and Mistral's Position · IDENTIFIED FROM THE TRANSCRIPT · source

  2. “I guess it taught us that there was a lot of interest for efficiency rather than scale. And so that's why we continued targeting very efficient models with the Mixtra 867B and more recently Mixtral 862B, ensuring that for certain cost and for a certain size we were reaching the top performance of the market. That has been a major motivation for us to target efficiency while simultaneously scaling toward the larger and larger.”

    2024-04-29 · The Twenty Minute VC · 20VC: Mistral's Arthur Mensch: Are Foundation Models Commoditising | How Do We Solve the Problem of Compute | Is There Value in the Application Layer | Open vs Closed: Who Wins and Mistral's Position · IDENTIFIED FROM THE TRANSCRIPT · source

  3. “it serves two purposes so the first was to show that there was a lot of slack in compressing models and so from a scientific perspective it was a good finding and and a good learning from the community it also filled a gap in the efficiency to performance 2d space of models where there was definitely something missing 7b is the size that allows to run efficiently a model on your mac book or on your smartphone and we made it sufficiently smart so that it was still useful so there was already 7p models before but they weren't good enough to do interesting applications and so by by targeting this specific space we talked to the developers immediately because developers like the casual developers running on a gaming gpu or on its macbook so it created a lot of curiosity and adoption because it was a missing spot in the”

    2024-04-29 · The Twenty Minute VC · 20VC: Mistral's Arthur Mensch: Are Foundation Models Commoditising | How Do We Solve the Problem of Compute | Is There Value in the Application Layer | Open vs Closed: Who Wins and Mistral's Position · IDENTIFIED FROM THE TRANSCRIPT · source

  4. “That point for me was probably around March, end of March last year where I decided to leave on Friday and I resigned on Monday. You can't stay if you have decided to resign, otherwise it's not very fair.”

    2024-04-29 · The Twenty Minute VC · 20VC: Mistral's Arthur Mensch: Are Foundation Models Commoditising | How Do We Solve the Problem of Compute | Is There Value in the Application Layer | Open vs Closed: Who Wins and Mistral's Position · IDENTIFIED FROM THE TRANSCRIPT · source

  5. “So it's not a zero to one decision. It's not a binary decision. You start to think like I'm 10% leaning on living and then it grows and at some point you cross the threshold and you say, okay, well, I guess now that I'm sufficiently decided there's no way in which I stay more than a few days. Otherwise, I wouldn't be candid with my colleagues. And so that's how you get started. You say there's no turning back anymore.”

    2024-04-29 · The Twenty Minute VC · 20VC: Mistral's Arthur Mensch: Are Foundation Models Commoditising | How Do We Solve the Problem of Compute | Is There Value in the Application Layer | Open vs Closed: Who Wins and Mistral's Position · IDENTIFIED FROM THE TRANSCRIPT · source

  6. “You have to share some things so you share the infrastructure, you share the code base, you share findings. We're doing general purpose models. And general purpose models, you need to evolve them in different directions. So you need to make them speak different languages. You need to make them be able to code, be able to do mathematics, be able to reason, you need to add multimodality to them. All of these things are loosely coupled. It's useful if you use the same framework for optimization, for data, for training. But you don't want to have your team spend their entire day in meetings for coordination. And it's actually pretty hard to figure out. I think so far we've managed to scale it relatively well. The team is only 25 people, so that's actually not super challenging. It will become more and more of a challenge. That's what I remember from DeepMind. It works very well at the beginning. Jimmy was a bit too slow, and I think they recovered sufficiently well since. We have optimized the team to be as fast as possible and to ship as fast as possible.”

    2024-04-29 · The Twenty Minute VC · 20VC: Mistral's Arthur Mensch: Are Foundation Models Commoditising | How Do We Solve the Problem of Compute | Is There Value in the Application Layer | Open vs Closed: Who Wins and Mistral's Position · IDENTIFIED FROM THE TRANSCRIPT · source

  7. “A team of five is faster than a team of 50, except if you organize the team of 50 to be 10 teams of five that are sufficiently uncoupled. One finding that I learned the hard way at DeepMine and the reason why we created the company in a slightly different way in terms of organization of the science team. And also the reason why we knew we had a chance to do interesting things with a smaller team.”

    2024-04-29 · The Twenty Minute VC · 20VC: Mistral's Arthur Mensch: Are Foundation Models Commoditising | How Do We Solve the Problem of Compute | Is There Value in the Application Layer | Open vs Closed: Who Wins and Mistral's Position · IDENTIFIED FROM THE TRANSCRIPT · source

  8. “That was Andrew Eng flying a helicopter backward. It's a control problem, which is not easy to solve. And I'm not sure if it was really AI related. I think he was saying that he was using a neural network to control all of this. But that's the first memory of me being shown what you could do with machine learning at the time. That was in 2013, I think.”

    2024-04-29 · The Twenty Minute VC · 20VC: Mistral's Arthur Mensch: Are Foundation Models Commoditising | How Do We Solve the Problem of Compute | Is There Value in the Application Layer | Open vs Closed: Who Wins and Mistral's Position · IDENTIFIED FROM THE TRANSCRIPT · source

  9. “Yes, I was a bit curious and a bit stubborn. Not very nice to my brothers. I was the eldest of them also. I don't know. You should ask them. I think they have good memories, hopefully.”

    2024-04-29 · The Twenty Minute VC · 20VC: Mistral's Arthur Mensch: Are Foundation Models Commoditising | How Do We Solve the Problem of Compute | Is There Value in the Application Layer | Open vs Closed: Who Wins and Mistral's Position · IDENTIFIED FROM THE TRANSCRIPT · source

  10. “You can't really scale that much quicker. You can't raise like $2 billion on the seed round. I mean, at least you couldn't in 2023.”

    2024-04-29 · The Twenty Minute VC · 20VC: Mistral's Arthur Mensch: Are Foundation Models Commoditising | How Do We Solve the Problem of Compute | Is There Value in the Application Layer | Open vs Closed: Who Wins and Mistral's Position · IDENTIFIED FROM THE TRANSCRIPT · source

  11. “We are still bottlenecked by compute for Schef, but that's because we don't have many of it. We have 1.5K 800, which is a few percent of our competitors.”

    2024-04-29 · The Twenty Minute VC · 20VC: Mistral's Arthur Mensch: Are Foundation Models Commoditising | How Do We Solve the Problem of Compute | Is There Value in the Application Layer | Open vs Closed: Who Wins and Mistral's Position · IDENTIFIED FROM THE TRANSCRIPT · source

  12. “The most technical saving enterprises are definitely ready for it. In order to widen the adoption, there's definitely some tooling to be brought to the market.”

    2024-04-29 · The Twenty Minute VC · 20VC: Mistral's Arthur Mensch: Are Foundation Models Commoditising | How Do We Solve the Problem of Compute | Is There Value in the Application Layer | Open vs Closed: Who Wins and Mistral's Position · IDENTIFIED FROM THE TRANSCRIPT · source