Skip to content

Podcast

#381 – Chris Lattner- Future of Programming and AI

Lex Fridman Podcast

Source ↗ ← All highlights
  • Mojo’s Vision
    • Mojo programming language improves AI accessibility and usability.
    • It addresses the challenge of programming diverse AI hardware, enabling broader innovation. Transcript: Chris Lattner Mojo? Well, so I mean, I think you have to zoom out. So I’ve been working on a lot of related technologies for many, many years. So I’ve worked on LLVM and a lot of things and mobile and servers and things like this, but the world’s changing. And what’s happened with AI is we have new GPUs and new machine learning accelerators and other ASICs and things like that, that make AI go real fast. At Google, I worked on TPUs. That’s one of the biggest, larger scale deployed systems that exist for AI. And really what you see is if you look across all of the things that are happening in the industry, there’s this new compute platform coming. And it’s not just about CPUs or GPUs or TPUs or NPUs or IPUs or whatever, all the PUs. It’s about how do we program these things. And? And so for software folks like us, right, it doesn’t do us any good if there’s this amazing hardware that we can’t use. And one of the things you find out really quick is that having the theoretical capability of programming something and then having the world’s power and the innovation of all the smart People in the world get unleashed on something can be quite different. And so really where Mojo came from was starting from a problem of we need to be able to take machine learning, take the infrastructure underneath it and make it way more accessible, way More usable, way more understandable by normal people and researchers and other folks that are not themselves like experts in GPUs and things like this. (Time 0:06:55)
  • AI Hardware Accessibility
    • The increasing diversity of AI hardware (GPUs, TPUs, etc.) creates a programming challenge.
    • Making this hardware accessible to a wider range of users is crucial for unlocking its full potential. Transcript: Chris Lattner It’s about how do we program these things. And? And so for software folks like us, right, it doesn’t do us any good if there’s this amazing hardware that we can’t use. And one of the things you find out really quick is that having the theoretical capability of programming something and then having the world’s power and the innovation of all the smart People in the world get unleashed on something can be quite different. And so really where Mojo came from was starting from a problem of we need to be able to take machine learning, take the infrastructure underneath it and make it way more accessible, way More usable, way more understandable by normal people and researchers and other folks that are not themselves like experts in GPUs and things like this. And then through that journey, we realized, hey, we need syntax for this. Lex Fridman We need to do programming language so one of the the main features of the language i say so fully in jest is that uh it allows you to have the file extension to be uh an emoji or the fire emoji Which is one of the first emojis used as a file extension I’ve ever seen in my (Time 0:07:37)
  • Python’s Strengths and Weaknesses
    • Python’s slowness is a common complaint, but its ecosystem and usability are strengths.
    • Mojo aims to combine these strengths with C-like performance by adding missing features. Transcript: Chris Lattner Well, I guess you could say several things, but probably that it’s slow. I think that’s usually what people complain about. And so, I mean, other people complain about tabs and spaces versus curly braces or whatever, But, I mean, those people are just wrong because it is actually just better to use indentation. Wow. Strong words. Lex Fridman So, actually, I have a small tangent. Let’s actually take that. Let’s take all kinds of tangents. Oh, come on, Lex. You can push me on it. I can take it. Design. Listen, I’ve recently left Emacs for VS Code and The kind of hate mail I had to receive. Because on the way to doing that, I also said I’ve considered Vim. Yep. And chose not to and went with VS Code. You’re touching on deep religions, right? Anyway, Tabs is an interesting design decision. And so you’ve really written a new programming language here. Yes, it is a superset of Python. But you can make a bunch of different interesting decisions here. Totally. Yeah. And you chose actually to stick with Python in terms of some of the syntax. Chris Lattner Well, so let me explain why, right? So, I mean, you can explain this in many rational ways. I think that the annotation is beautiful, but that’s not a rational explanation, right? So, but I can defend it rationally, right? So first of all, Python 1 has millions of programmers. It is huge. It’s everywhere. It owns machine learning, right? And so, factually, it is the thing, right? Second of all, if you look at it, C code, C++ code, Java, whatever, Swift, curly brace languages also run through formatting tools and get indented. And so if they’re not indented correctly, first of all, we’ll twist your brain around. It can lead to bugs. There’s notorious bugs that have happened across time where the annotation was wrong or misleading and it wasn’t formatted right. And so it turned into an issue, right? And so what ends up happening in modern large-scale code bases is people run automatic formaters. So now what you end up with is indentation and curly braces. Well, if you’re going to have the notion of grouping, why not have one thing? And get rid of all the clutter and have a more beautiful thing. Also, you look at many of these languages, it’s like, okay, well, you can have curly braces, or you can omit them if there’s one statement, or you just like enter this entire world of complicated Design space that objectively, you don’t need if you have Python style indentation. Lex Fridman So yeah, I would love to actually see statistics on errors made because of indentation. Chris Lattner Like how many errors are made in Python versus in C++ that have to do with basic formatting, all that kind of stuff. I would love to see. I think it’s probably pretty minor because once you get, like you use VS Code, I do too. So if you get VS Code set up, it does the indentation for you, generally. And so you don’t, it’s actually really nice to not have to fight it. And then what you can see is the editor is telling you how your code will work by indenting it, which I think is pretty cool. Lex Fridman I honestly don’t think I’ve ever, I don’t remember having an error in Python because I indented stuff wrong. Chris Lattner So, I mean, I think that there’s, again, this is a religious thing. And so I can joke about it and I love to kind of, you know, I realize that this is such a polarizing thing and everybody wants to argue about it. And so I like poking at the bear a little bit, right? But frankly, right, come back to the first point, Python 1. Like it’s huge, it’s an AI, it’s the right thing. For us, like we see Mojo as being an incredible part of the Python ecosystem. We’re not looking to break Python or change it or quote unquote fix it. We love Python for what it is. Our view is that Python is just not done yet. And so if you look at, you know, you mentioned Python being slow. Well, there’s a couple of different things that go into that, which we can talk about if you want. But one of them is it just doesn’t have those features that you would use to do C-like programming. And so if you say, okay, well, I’m forced out of Python into C for certain use cases. Well, then what we’re doing is we’re saying, okay, well, why is that? Can we just add those features that are missing from Python back up to Mojo? And then you can have everything that’s great about Python, all the things you’re talking about that you love, plus not be forced out of it when you do something a little bit more computationally Intense or weird or hardware-y or whatever it is that you’re doing. Well, (Time 0:16:10)
  • Mojo’s Compilation
    • Mojo supports interpreted, JIT-compiled, and statically compiled modes.
    • This enables dynamic metaprogramming at compile time, combining Python’s expressiveness with efficiency. Transcript: Chris Lattner Mojo, the complicated answer, does all the things. So it’s interpreted, it’s just compiled, and it’s statically compiled. And so this is for a variety of reasons. So one of the things that makes Python beautiful is that it’s very dynamic. And because it’s dynamic, one of the things they added is that it has this powerful metaprogramming feature. Use case like you define a class that has the plus method right you can overload the dunder methods like dunder add for example and then the plus method works on your class and so it has Very nice and very expressive dynamic metaprogramming features in mojo we want all those features come in like we don’t want to break python we want it all to work but the problem is you Can’t run those super dynamic features on an embedded processor or on a GPU. Or if you could, you probably don’t want to just because of the performance. And so we entered this question of saying, okay, how do you get the power of this dynamic metaprogramming into a language that has to be super efficient in specific cases? And so what we did was we said, okay, we’ll take that interpreter. Python has an interpreter in it, right? Take that interpreter and allow it to run at compile time. And so now what you get is you get compile time metaprogramming. And so this is super interesting and super powerful because one of the big advantages you get is you get Python style expressive APIs. You get the ability to have overloaded operators. And if you look at what happens inside of like PyTorch, for example, with automatic differentiation and eager mode, like all these things, they’re using these really dynamic and powerful Features at runtime, but we can take those features and lift them so that they run at compile time. (Time 0:20:31)
  • Autotuning’s Power
    • Autotuning in Mojo addresses the complexity of hardware and algorithms.
    • It empirically determines optimal parameters, like tile sizes, for specific hardware. Transcript: Lex Fridman Like a really, like, you know, we mentioned list comprehensions. To me, from a quick glance at Mojo, which, by the way, I have to absolutely, like, dive in. As I realize how amazing this is, I absolutely must dive in. That looks like just an incredible feature for machine learning people. Chris Lattner Yeah, well, so what (Time 0:26:41)
  • Mojo’s Typing System
    • Mojo uses optional and progressive typing.
    • This gives performance benefits of types while maintaining Python compatibility. Transcript: Lex Fridman Sexy and beautiful like Python, as we mentioned. Is it a typed language? So what’s the role of types? Yeah, good question. Chris Lattner So Python has types. It has strings, it has integers, it has dictionaries and all that stuff. But they all live at runtime. And so because all those types live at runtime in Python, you don’t have to spell them. Python also has this whole typing thing going on now, and a lot of people use it. I’m not talking about that. That’s kind of a different thing. We can go back to that if you want. But typically, you just say, I have a def, and my def takes two parameters. I’m going to call them A and B, and I don’t have to write a type. So that is great, but what that does is that forces what’s called a consistent representation. So these things have to be a pointer to an object with the object header, and they all have to look the same. And then when you dispatch a method, you go through all the same different paths, no matter what the receiver, whatever that type is. So what Mojo does is it allows you to have more than one kind of type. And so what it does is allows you to say, okay, cool. I have an object and objects behave like Python does. And so it’s fully dynamic and that’s all great. And for many things, classes, like that’s all very powerful and very important. But if you want to say, Hey, it’s an integer and it’s 32 bits or 64 bits or whatever it is, or it’s a floating point value and it’s 64 bits. Well, then the compiler can take that and it can use that to do way better optimization. And it turns out again, getting rid of the interactions, that’s huge means you can get better code completion because you have, because compiler knows what the type is. And so it knows what operations work on it. And so that’s actually pretty huge. And so what Mojo does is it allows you to progressively adopt types into your program. And so you can start again it’s compatible with Python. And so then you can add however many types you want, wherever you want them. And if you don’t want to deal with it, you don’t have to deal with it. And so one of our opinions on this is that it’s not that types are the right thing or the wrong thing. It’s that they’re a useful thing. Well, (Time 0:35:22)
  • Python vs. Mojo Typing
    • Python’s type annotations are ignored by the interpreter, acting as hints for external tools.
    • Mojo enforces declared types at compile time for better performance and error checking. Transcript: Lex Fridman Superset part is it’s just i mean there’s so much brilliant stuff here that definitely is incredible. We’ll talk about that. First of all, how’s the typing implemented differently in Python versus Mojo? So this heterogeneous flexibility, you said it’s definitely implemented. Chris Lattner Yeah, so I’m not a full expert in the whole backstory on types in Python. So I’ll give you that. I can give you my understanding. My understanding is basically, like many dynamic languages, the ecosystem went through a phase where people went from writing scripts to writing a large scale, huge code bases in Python, and at scale, it kind of helps to have types. People want to be able to reason about interfaces, what do you expect, a string or an int? These basic things, right? And so what the Python community started doing is it started saying, okay, let’s have tools on the side, checker tools, right? That go and enforce invariance, check for bugs, try to identify things. These are called static analysis tools generally. And so these tools run over your code and try to look for bugs. What ended up happening is there’s so many of these things, so many different weird patterns and different approaches on specifying the types and different things going on that the Python community realized and recognized, hey, hey, hey, there’s a thing here. And so what they started to do is they started to standardize the syntax for adding types to Python. Now, one of the challenges that they had is that they’re coming from kind of this fragmented world where there’s lots of different tools, they have different trade-offs and interpretations, And the types mean different things, and so if you look at types in Python, according to the Python spec, the types are ignored. Right, so according to the Python spec, you can write pretty much anything in a type position, okay? And technically, you can write any expression, okay? Now, that’s beautiful, because you can extend it, you can do cool things, you can write, build your own tools, you can build your own house linter or something like that, right? But it’s also a problem, because any existing Python program may be using different tools, and they have different interpretations. And so if you adopt somebody’s package into your ecosystem, try around the tool you prefer, it may throw out tons of weird errors and warnings and problems just because it’s incompatible With how these things work. Also because they’re added late and they’re not checked by the Python interpreter, it’s always kind of more of a hint than it is a requirement. Also, the CPython implementation can’t use them for performance. And so it’s really that’s the big one, right? Lex Fridman So you can’t utilize the for the compilation for the just in time compilation. Chris Lattner Okay, exactly. And this this all comes back to the design principle of it’s, it’s kind of the kind of hints, they’re kind of the definitions a little bit murky, it’s unclear exactly the interpretation In a bunch of cases. And so because of that, you can’t actually, even if you want to, it’s really difficult to use them to say like it is going to be an int and if it’s not, it’s a problem, right? A lot of code would break if you did that. So in Mojo, right? So you can still use those kind of type annotations. It’s fine. But in Mojo, if you declare a type and you use it, then it means it is going to be that type. And the compiler helps you check that and force it and it’s safe. And And it’s not, it’s not a like best effort hint kind of a thing. So (Time 0:40:38)
  • Immutability and Value Semantics
    • Immutability (using let) improves code reasoning and prevents bugs.
    • Mojo uses value semantics, providing logical copies of data like arrays and tensors, reducing bugs. Transcript: Lex Fridman The other performance improvement comes from immutability. So what’s this var and this let thing that we’ve got going on? And what’s immutability? Chris Lattner Yeah. So one of the things that is useful, and it’s not always required, but it’s useful, is knowing whether something can change out from underneath you. Right and so in python you have a pointer to an array right and so you pass that pointer to an array around to things if you pass into a function they may take that and scroll away in some other Data structure and so you get your array back and you go to use it now somebody else is like putting stuff in your array how do you reason about that? It gets to be very complicated and leads to lots of bugs. And so one of the things that, again, this is not something Mojo forces on you, but something that Mojo enables is a thing called value semantics. And what value semantics do is they take collections like arrays, like dictionaries, also tensors and strings and things like this that are much higher level and make them behave like Proper values. And so it makes it look like if you pass these things around, you get a logical copy of all the data. And so if I pass you an array, it’s your array. You can go do what you want to it. You’re not going to hurt my array. Now, that is an interesting and very powerful design principle. It defines away a ton of bugs. You have to be careful to implement it in an efficient way. Lex Fridman Is there a performance hit that’s significant? Chris Lattner Generally not, if you implement it the right way. But it requires a lot of very low-level getting the language right bits. I (Time 0:52:03)
  • Complexity as the Enemy
    • The AI industry’s biggest enemy is complexity, spanning hardware, software, and models.
    • Modular aims to reduce this complexity, making AI more accessible and manageable. Transcript: Chris Lattner We’re all struggling with, and it’s like fish can’t see water, is complexity. Sure, yes. Complexity. That was very philosophical of you. It was very sad. And so if you look at it, yes, it is on the hardware side. Yes. All these accelerators, all these software stacks that go with the accelerator, all these, like there’s massive complexity over there. You look at what’s happening on the modeling side, massive amount of complexity. Like things are changing all the time. People are inventing, turns out the research is not done, right? And so people want to be able to move fast. Transformers are amazing, but there’s a ton of diversity even within transformers. And what’s the next transformer, right? And you look into serving. Also, huge amounts of complexity it turns out that all the cloud providers right have all their very weird but very cool hardware for networking and all this kind of stuff and it’s all Very complicated people aren’t using that you look at classical serving right there there’s this whole world of people who know how to write high performance servers with zero copy Networking and like all this fancy asynchronous I.O. And like all these fancy things in the serving community. Very little that has pervaded into the machine learning world. Right? And why is that? Well, it’s because, again, these systems have been built up over many years. They haven’t been rethought. There hasn’t been a first principles approach to this. And so what modular is doing is we’re saying okay we’ve built many of these things right so i’ve worked on tensorflow and tpus and things like that other folks on our team have like worked On pytorch core we’ve worked on onyx runtime we’ve worked on many of these other systems and so built systems like the apple accelerators and all that kind of stuff like our team is quite Amazing and so one of the things that roughly everybody at modular is grumpy about is that when you’re working on one of these projects you have a first order goal get the hardware to work Get the system to enable one more model get this product out the door enable the specific workload or make it solve this problem for this this product team right and nobody’s been given A chance to actually do that step back. And so we as an industry, we didn’t take two steps forward. We took like 18 steps forward in terms of all this really cool technology across compilers and systems and runtimes and heterogeneous compute and like all this kind of stuff. And like all this technology has been, you know, I wouldn’t say beautifully designed, but it’s been proven in different quadrants. Like, you know, you look at Google with TPUs, massive, huge exaflops of compute strapped together into machines that researchers are programming in Python in a notebook. That’s huge. That’s amazing. That’s incredible. It’s incredible. And so you look at the technology that goes into that, and the algorithms are actually quite general. So lots of other hardware out there and lots of other teams out there don’t have the sophistication or maybe the the years working on it or the the budget or whatever that google does right And so they should be getting access to the same algorithms but they just don’t have that right and so what modular is doing is we’re saying cool this is not research anymore like we’ve We’ve built auto tuning in many systems. We’ve built programming languages, right? And so like have, have, you know, implemented C++, have implemented Swift, have implemented many of these things. And so, you know, this, it’s hard, but it’s not research. And you look at accelerators. Well, we know there’s a bunch of different weird kinds of accelerators, but they actually cluster together, right? And you look at GPUs. Well, there’s a couple of major vendors of GPUs, and they maybe don’t always get along, but their architectures are very similar. You look at CPUs. CPUs are still super important for the deployment side of things. And you see new architectures coming out from all the cloud providers and things like this, and they’re all super important to the world, right? But they don’t have the 30 years of development that the entrenched people do, right? And so what modular can do is we’re saying, okay, all this complexity, like it’s not, it’s not bad complexity. It’s actually innovation, right? And so it’s innovation that’s happening, and it’s for good reasons, but I have sympathy for the poor software people, right? I mean, again, I’m generally a software person too. I love hardware, but software people want to build applications and products and solutions that scale over many years. They don’t want to build a solution for one generation of hardware with one vendor’s tools, right? And because of this, they need something that scales with them. They need something that works on cloud and mobile, right? Because, you know, their product manager said, hey, I want it to be, have lower latency and it’s better for personalization or whatever they decide, right? Products evolve. And so, the challenge with the machine learning technology and the infrastructure we have today in the industry is that it’s all these point solutions. And because there are all these point solutions, it means that as your product evolves, you have to like switch different technology stacks or switch to a different vendor. And what that does is that slows down progress. Lex Fridman So (Time 1:09:22)
  • Mojo’s Superset Status
    • Mojo is designed to be a superset of Python, enabling gradual migration and compatibility.
    • It allows importing existing Python packages via CPython integration while native Mojo offers speedups. Transcript: Lex Fridman Yes, yes. Chris Lattner So, and this has two sides of it. So Mojo’s not done yet. So I’ll give you a disclaimer, Mojo’s not done yet. But already we see people that take small pieces of Python code, move it over, they don’t change it, and you can get 12x speedups. Like somebody was just tweeting about that yesterday, which is pretty cool, right? And again, interpreters, compilers, right? And so without changing any code, without, also this is not with, this is not jit compiling or doing anything fancy. This is just basic stuff, move it straight over. Now, Mojo will continue to grow out, and as it grows out, it will have more and more and more features, and our North Star is to be a full superset of Python, and so you can bring over basically Arbitrary Python code and have it just work. And it may not always be 12x faster, but it should be at least as fast and way faster in many cases is the goal. Now it’ll take time to do that, and Python is a complicated language. There’s not just the obvious things, but there’s also non-obvious things that are complicated, like we have to be able to talk to CPython packages, to talk to the C API, and there’s a Lex Fridman Bunch of pieces to this. So you have to, I mean, just to make explicit, the obvious, it may not be so obvious until you think about it. So, you know, to run Python code, that means you have to run all the Python packages and libraries. Yeah. So that means what? What’s the relationship between Mojo and CPython, the interpreter that’s, presumably would be tasked with getting those packages to work. Chris Lattner Yep, so in the fullness of time, Mojo will solve for all the problems, and you’ll be able to move Python packages over and run them in Mojo. Without the CPython. Without CPython, someday. Yeah. Right, it’s not today, but someday. And that’ll be a beautiful day, because then you’ll get a whole bunch of advantages, you’ll get massive speedups and things like this. Lex Fridman But you can do that one at a time, right? You can move packages one at a time. Exactly. Chris Lattner But we’re not willing to wait for that. Python is too important. The ecosystem is too broad. We want to both be able to build Mojo out. We also want to do it the right way without intense time pressure. We’re obviously moving fast. And so what we do is we say, okay, well, let’s make it so you can import an arbitrary existing package. Arbitrary, including like you write your own on your local disk or whatever. It’s not like a standard package, like an arbitrary package. And import that using CPython. Because CPython already runs all the packages, right? And so what we do is we built an integration layer where we can actually use CPython. Again, I’m practical. And to actually just load and use all the existing packages as they are. The downside of that is you don’t get the benefits of Mojo for those packages. And so they’ll run as fast as they do in the traditional CPython way. But what that does is that gives you an incremental migration path. And so if you say, hey, cool, well, here’s a, you know, the Python ecosystem is vast. I want all of it to just work, but there’s certain things that are really important. And so if I, if I’m doing weather forecasting or something, well, I want to be able to load all the data. I want to be able to work with it. And then I have my own crazy algorithm inside of it. Well, normally I’d write that in C++. If I can write in Mojo and have one system that scales, well, that’s way easier to work with. Lex Fridman Is it hard (Time 1:38:53)
  • Swift for TensorFlow’s Lessons
    • The Swift for TensorFlow project faced challenges due to Python’s dominance in ML.
    • Mojo addresses this by being Python-compatible and designed for eager mode execution. Transcript: Chris Lattner Life choices I make. Did you go to the desert and did you meditate on it? Lex Fridman Okay. All right. No, it was bold. It was bold and needed. And I think, I mean, it’s just bold and sometimes to take those leaps is a difficult leap to take. Chris Lattner Yeah. Well, so, okay. I mean, I think there’s a couple of different things. So actually I left Apple back in 2017, like January, 2017. So it’s been a number of years that I left Apple. And the reason I left Apple was to do AI. Okay. So, and again, I won’t come out on Apple and AI, but the, uh, uh, at the time, right. I wanted to get into and understand and understand the technology, understand the applications, the workloads. And so I was like, okay, I’m going to go dive deep into applied and AI and then the technology underneath it, right? I found myself at Google. And that was like when TPUs were waking up. Exactly. And so I found myself at Google and Jeff Dean, who’s a rock star, as you know, right. And the, and in 2017 TensorFlow is like really taking off and doing incredible things. And I was attracted to Google to help them with the TPUs, right. And TPUs are an innovative hardware accelerator platform, have now, I mean, I think proven massive scale and like done incredible things. Right. And so one of the things that this led into is a bunch of different projects, which I’ll skip over. Right. One of which was this Swift for TensorFlow project. Right. And so that project was a research project. And so the idea of that is say, okay, well, let’s look at innovative new programming models where we can get a fast programming language. We can get automatic differentiation into the language. Let’s push the boundaries of these things in a research setting. Right. Now that project, I think lasted two, three years. There’s some really cool outcomes of that. So one of the things that’s really interesting is I published a talk at an LLVM conference in 2018. Again, this seems like so long ago, about graph program abstraction, which is basically the thing that’s in PyTorch 2. And so PyTorch 2, with all this Dynamo Rio thing, it’s all about this graph program abstraction thing from Python bytecodes. And so a lot of the research that was done ended up pursuing and going out through the industry and influencing things. And I think it’s super exciting and awesome to see that. But the Swift for TensorFlow project itself did not work out super well. And so there’s a couple of different problems with that. One of which is that you may have noticed Swift is not Python. There’s a few people that write Python code. Yes. Lex Fridman And so it turns out that all of ML is pretty happy with Python. It’s actually a problem that other programming languages have as well, that they’re not Python. We’ll probably maybe briefly talk about Julia, who’s a very interesting, beautiful programming language, but it’s not Python. Chris Lattner Exactly. Well, and so if, and so like, if you’re saying, I’m going to solve a machine learning problem where all the programmers are Python programmers. And you say, the first thing you have to do is switch to a different language. Well, your new thing may be good or bad or whatever, but if it’s a new thing, the adoption barrier is massive. Lex Fridman It’s still possible. Chris Lattner It’s still possible. Yeah, absolutely. The world changes and evolves and there’s definitely room for new and good ideas, but it just makes it so much harder. And so, lesson learned, Swift is not Python. And people are not always in search of learning a new thing for the sake of learning a new thing. And if you want to be compatible with all the world’s code, turns out, meet the world where it is. Second thing is that a lesson learned is that Swift has a very fast and efficient language, kind of like Mojo, but a different take on it still, really worked well with eager mode. And so eager mode is something that PyTorch does, and it proved out really well. And it enables really expressive and dynamic and easy to debug programming. TensorFlow at the time was not set up for that, let’s say. That was not. The timing is also important in this world. Yeah, yeah, and TensorFlow is a good thing, and it has many, many strengths, but you could say Swift for TensorFlow is a good idea, except for the Swift and except for the TensorFlow part. Swift because it’s not Python and TensorFlow because it’s not… It wasn’t set up for eager mode at the time. Yeah. It was 1.0. Exactly. And so one of the things about that is in the context of it being a research project, I’m very happy with the fact that we built a lot of really cool technology. We learned a lot of things. I think the ideas went on to have influence in other systems like PyTorch. A few people use that, I hear. And so I think that’s super cool. And for me personally, I learned so much from it. And I think a lot of the engineers that worked on it also learned a tremendous amount. And so I think that that’s just really exciting to see. And I’m sorry that the project didn’t work out. I wish it did, of’s it’s it’s a research project and so you’re there to learn from it well (Time 2:05:24)
  • Jeremy Howard’s Influence
    • Jeremy Howard’s use of progressive image sizing outperformed Google on ImageNet training.
    • This highlights his deep understanding of algorithms and optimization, inspiring Mojo. Transcript: Chris Lattner He’s been pushing for this kind of thing. He’s been pushing for more. Lex Fridman He’s wanted this for years. Yeah, he’s wanted this for a long, long time. He’s wanted this for years. For people who don’t know Jeremy Howard, he’s like one of the most legit people in the machine learning community. He’s a grassroots. He really teaches. He’s an incredible educator. He’s an incredible teacher, but also legit in terms of a machine learning engineer himself. He’s been running the fast.ai and looking I think, for exactly what you’ve done with Mojo. Chris Lattner Exactly. And so, I mean, the first time, so I met Jeremy pretty early on, but the first time I sat up and I’m like, this guy is ridiculous, is when I was at Google and we were bringing up TPUs and we had A whole team of people and there was this competition called DonBench of who can train ImageNet fastest, right? Yes. And Jeremy and one of his researchers crushed Google. Not through sheer force of the amazing amount of compute and the number of TPUs and stuff like that, that he just decided that progressive imagery sizing was the right way to train the Model and fewer epochs faster and make the whole thing go vroom, right? And I’m like, this guy is incredible. And so you can say, anyways, come back to where’s Mojo coming from. Chris finally listened to Jeremy. It’s all his fault. (Time 2:32:27)
  • Strict Mode in Mojo
    • Mojo provides fn, a strict mode for function definitions that enforce variable declarations.
    • This offers stricter error checking and predictability, appealing to systems programmers. Transcript: Chris Lattner That the right way to say it it’s either optionally or progressively or progressively i think so so so people have very strong opinions on the right word to use yeah i don’t know i look Lex Fridman Forward to your letters uh so there’s the the var versus let but let is for constants yeah uh var is an optional uh yeah var makes it mutable, so you can reassign. Chris Lattner Okay. Lex Fridman Then there’s function overloading. Oh, okay. Chris Lattner Yeah. Lex Fridman I mean, there’s a lot of source of happiness for me, but function overloading, I guess, is that for performance? Why does Python not have function overloading? So I can speculate. Chris Lattner So Python is a dynamic language. The way it works is that Python and Objective-C are actually very similar worlds if you ignore syntax. And so Objective-C is straight line derived from small talk. A really venerable, interesting language that much of the world has forgotten about, but the people that remember it love it generally. And the way that Smalltalk works is that every object has a dictionary in it, and the dictionary maps from the name of a function or the name of a value within an object to its implementation. And so the way you call a method in an Objective-C is you say, go look up, the way I call foo is I go look up foo, I get a pointer to the function back and then I call it. That’s how Python works. And so now the problem with that is that the dictionary within a Python object, all the keys are strings, and it’s a dictionary, so you can only have one entry per name you think it’s as Simple as that i think it’s as simple as that and so now why do they never fix this like why do they not change it to not be a dictionary why do they not change like do other things um well you Don’t really have to in python because it’s dynamic and so you can say i get into the function now if i got past an integer do some dynamic test for it if it’s a string go do another thing um There’s another additional challenge which is even if you did support overloading you’re saying okay well here’s a version of a function for integers and a function for strings well You’d have even if you could put it in that dictionary you’d have to have the caller do the dispatch and so every time you call the function you’d have to say like is an integer is a string And so you’d have to figure out where to do that test. And so in a dynamic language, overloading is something you don’t have to have. But now you get into a type language. And in Python, if you subscript with an integer, then you get typically one element out of a collection. If you subscript with a range, you get a different thing out. And so often in type languages, you’ll want to be able to express the fact that, cool, I have different behavior depending on what I actually pass into this thing. If you can model that, it can make it safer and more predictable and faster and like all these things. Lex Fridman It somehow feels safer, yes, but also feels empowering. Like in terms of clarity, like you don’t have to design hold different functions. Chris Lattner Yeah, well this is also one of the challenges with the existing Python typing systems, is that in practice, like you take subscript, like in practice a lot of these functions, they don’t Have one signature. They actually have different behavior in different cases. And so this is why it’s difficult to retrofit this into existing Python code and make it play well with typing. You kind of have to design for that. Lex Fridman The program Python might be interested in is def versus fn. So it’s two different ways to define a function. And fn is a stricter version of def. What’s the coolness that comes from the strictness? Chris Lattner So here you get into what is the trade-off with the superset? Lex Fridman Yes. Okay. Chris Lattner So a superset, you have to, or you really want to be compatible. Like if you’re doing a superset, you’ve decided compatibility with existing code is the important thing, even if some of the decisions they made were maybe not what you’d choose. Yeah. Okay. So that means you put a lot of time in compatibility, and it means that you get locked into decisions of the past, even if they may not have been a good thing, right? Now, systems programmers typically like to control things, right? And they want to make sure that, you know, not in all cases, of course, and even systems programmers are not one thing, right? But often you want predictability. And so one of the things that Python has, for example, as you know, is that if you define a variable, you just say x equals four. I have a variable named x. Now I say some long name equals 17. Print out some long name. Oops, but I typoed it. Well, the compiler, the Python compiler doesn’t know, in all cases, what you’re defining and what you’re using, and did you typo the use of it or the definition? And so for people coming from type languages, again, I’m not saying they’re right or wrong, but that drives them crazy because they want the compiler to tell them, you typoed the name Of this thing. And so what FN does is it turns on, as you say, it’s a strict mode. And so it says, okay, well, you have to actually intentionally declare your variables before you use them. That gives you more predictability, more error checking and things like this, but you don’t have to use it. And this is a way that Mojo is both compatible, because defs work the same way that defs have always worked, but it provides a new alternative that gives you more control and allows certain Lex Fridman Kinds of people that have a different philosophy to be able to express that and get that. But usually, if you’re writing Mojo code from scratch, you’ll be using FN. Chris Lattner It depends. Again, it depends on your mentality, right? It’s not that def is Python and FN is Mojo. Mojo has both, and it loves both, right? It really depends on… FN is just strict. Yeah, exactly. Are you playing around and scripting something out? Is it a one-off throwaway script? Cool. Python is great at that. I will still be using FN, but yeah. I love strictness. Control. Lex Fridman Power. Chris Lattner You also like suffering, right? Yes. Go hand in hand. How many pull-ups? Lex Fridman I’ve lost count this point. Chris Lattner And that’s cool. I love you for that. And I love other people who like strict things, right? But I don’t want to say that that’s the right thing because Python’s also very beautiful for hacking around and doing stuff and research and these other cases where you may not want that. Lex Fridman You see, I just feel like, maybe I’m wrong on that, but it feels like strictness leads to faster debugging. So in terms of going from, even on a small project from zero to completion, it’s just, I guess it depends how many bugs you generate usually. Chris Lattner Well, so, I mean, it’s again, lessons learned in looking at the ecosystem. It’s really, I mean, I think it’s, if you study some of these languages over time, like the Ruby community, for example, (Time 2:40:43)
  • Error Handling in Mojo
    • Unlike C++‘s zero-cost exceptions, Mojo’s error handling is as fast as returning a value.
    • This allows consistent error handling on GPUs and accelerators, simplifying API design. Transcript: Lex Fridman Therapy session. Yes, I definitely will. Speaking of asserts, exceptions are called errors. Why is it called errors? Chris Lattner So we use the same, we’re the same as Python, right? But we implement in a very different way, right? And so if you look at other languages, like we’ll pick on C++, our favorite, right? C++ has a thing called zero cost exception handling. Okay, and this is, in my opinion, something to learn lessons from. Lex Fridman It’s a nice, polite way of saying it. Chris Lattner And so zero cost exception handling, the way it works is that it’s called zero cost because if you don’t throw an exception, there’s supposed to be no overhead for the non-error code. And so it takes the error path out of the common path. It does this by making throwing an error extremely expensive. And so if you actually throw an error with a C++ compiler using exceptions, it has to go look up in tables on the side and do all this stuff. And so throwing an error can be like 10,000 times more expensive than returning from a function, right? Also, it’s called zero-cost exceptions, but it’s not zero-cost. By any stretch of the imagination, because it massively blows out your code, your binary, it also adds a whole bunch of different paths because of destructors and other things like That that exist in C++, and it reduces the number of optimizations, it has like all these effects. And so this thing that was called zero cost exceptions, it really ain’t. Okay. Now, if you fast forward to newer languages, and this includes Swift and Rust and Go and now Mojo. Well, and Python’s a little bit different because it’s interpreted. And so like, it’s got a little bit of a different thing going on. But if you look at compiled languages, many newer languages say, okay, well, let’s not do that zero-cost exception handling thing. Let’s actually treat and throwing an error the same as returning a variant, returning either the normal result or an error. Now, programmers generally don’t want to deal with all the typing machinery and like pushing around a variant. And so you use all the syntax that Python gives us, for example, try and catch, and you know, functions that raise and things like this. You can put a raises, decorate around your functions, stuff like this. And you want to control that. And then the language can provide syntax for it. But under the hood, the way the computer executes it, throwing an error is basically as fast as returning something. Lex Fridman Interesting. So it’s exactly the same way from a compiler perspective. Chris Lattner And so this is actually, I mean, it’s a fairly nerdy thing, right? Which is why I love it. But this has a huge impact on the way you design your APIs, right? So in C++, huge communities turn off exceptions because the cost is just so high, right? And so the zero cost is so high, right? And so that means you can’t actually use exceptions in many libraries, right? Even for the people that do use it, well, okay, how and when do you want to pay the cost? If I try to open a file, should I throw an error? Well, what if I’m probing around looking for something, right? I’m looking it up in many different paths. Well, if it’s really slow to do that, maybe I’ll add another function that doesn’t throw an error or turns an error code instead. And now I have two different versions of the same thing. And so it causes you to fork your APIs. And so, you know, one of the things I learned from Apple and I so love is the art of API design is actually really profound. I think this is something that Python’s also done a pretty good job at in terms of building out this large scale package ecosystem. It’s about having standards and and things like this. And so, you know, we wouldn’t want to enter a mode where, you know, there’s this theoretical feature that exists in language, but people don’t use it in practice. Now, I’ll also say one of the other really cool things about this implementation approach is that it can run on GPUs and it can run on accelerators and things like this. And that standard zero-cost exception thing would never work on an accelerator. And so this is also part of how Mojo can scale all the way down to little embedded systems and to running on GPUs and things like that. Can (Time 2:48:56)