Skip to content

Podcast

Google- The AI Company

Acquired

Source ↗ ← All highlights
  • Google’s Innovator’s Dilemma
    • Google faces a classic innovator’s dilemma: new AI products can be better but far less profitable than legacy Search.
    • The company must choose between protecting $140B annual profit from Search or disrupting itself with AI. Transcript: Ben Gilbert Here’s a dilemma. Imagine you have a profitable business. You make giant margins on every single unit you sell. And the market you compete in is also giant. One of the largest in the world, you might say. But then on top of that, lucky for you, you also are a monopoly in that giant market with 90% share and a lot of lock-in. David Rosenthal And when you say monopoly, monopoly as defined by the U.S. Government. That is correct. Ben Gilbert But then imagine this. In your research lab, your brilliant scientists come up with an invention. This particular invention, when combined with a whole bunch of your old inventions by all your other brilliant scientists, turns out to create the product that is much better most Purposes than your current product. So you launched the new product based on this new invention, right? Right. I mean, especially because out of pure benevolence, your scientists had published research papers about how awesome the new invention is and lots of the inventions before also. So now there’s new startup competitors quickly commercializing that invention. So of course, David, you change your whole product to be based on a new thing, right? Uh, this sounds like a movie. Yes, but here is the problem. You haven’t figured out how to make this new incredible product anywhere near as profitable as your old giant cash printing business. So maybe you shouldn’t launch that new product. David, this sounds like quite the dilemma to me. Of course, listeners, this is Google today. (Time 0:01:00)
  • Lunch Idea Became Phil And Ads
    • Georges Hinton and Noam Shazir started a language-model project over lunch and dropped other work to pursue it.
    • Their probabilistic model ‘Phil’ later powered Did You Mean and AdSense features at Google. Transcript: David Rosenthal Now, Georges was one of Google’s first 10 employees, incredible engineer. And just like Larry Page’s dad, he had a PhD in machine learning from the University of Michigan. And even when Georges went there, it was still a relatively rare contrarian subfield within computer science. So the three of them are having lunch, and George says offhandedly to the group that he has a theory from his time as a PhD student that compressing data is actually technically equivalent To understanding it. And the thought process is, if you can take a given piece of information and make it smaller, store it away, and then later reinstantiate it in its original form, the only way that you Could possibly do that is if whatever force is acting on the data actually understands what it means, because you’re losing information, going down to something smaller, and then Recreating the original thing. It’s like you’re a kid in school. You learn something in school, you read a long textbook, you store the information in your memory, then you take a test to see if you really understood the material. And if you can recreate the concepts, then you really understand it. Ben Gilbert Which kind of foreshadows big LLMs today are like compressing the entire world’s knowledge into some number of terabytes. That’s just like the smash down little vector set. Little, at least compared to all the information in the world. But it’s kind of that idea, right? You can store all the world’s information in an AI model in something that is like kind of incomprehensible and hard to understand. But then if you uncompress it, you can kind of bring knowledge back to its original form. David Rosenthal Yep. And these models demonstrate understanding, right? Ben Gilbert Do they? That’s the question. That’s the question. They certainly mimic understanding. David Rosenthal So this conversation is happening. You know, this is 25 years ago. And Noam, the new hire, the young buck, he sort of stops in his tracks and he’s like, wow. If that’s true, that’s really profound. Is this in one of Google’s micro kitchens? This is in one of Google’s micro kitchens. They’re having lunch. Where did you find this, by the way, a 25 year old? This is in the Plex. This is like a small little passage in Stephen Levy’s great book that’s been a source for all of our Google episodes in the Plex. There’s a small little throwaway passage in here about this because this book came out before ChatGPT and AI and all that. So Gnome kind of latches on to Georges and keeps vibing over this idea. And over the next couple of months, the two of them decide in the most googly fashion possible that they are just going to stop working on everything else and they’re going to go work on This idea, on language models and compressing data and can they generate machine understanding with data. And if they can do that, that that would be good for Google. (Time 0:10:12)
  • Engineering Scaled Research To Product
    • Jeff Dean parallelized a 12-hour translation model into a 100 millisecond production system.
    • Engineering and distributed infrastructure turned research breakthroughs into massive product wins at Google. Transcript: David Rosenthal And as Jeff knows very well, because he and Sanjay basically built it with Erz Holza, Google’s infrastructure is extremely parallelizable. Distributed, you can break up workloads into little chunks, send them all over the various data centers that Google has, reassemble the projects, return that to the user. Ben Gilbert They are the single best company in the world at parallelizing workloads across CPUs, across multiple data centers. David Rosenthal CPUs. We’re still talking CPUs here. Yep. And Jeff’s work with the team gets that average sentence translation time down from 12 hours to 100 milliseconds. (Time 0:22:11)
  • The ‘Cat Paper’ Sparked Modern AI Products
    • The 2011 ‘cat paper’ proved large unsupervised neural nets could learn meaningful features on distributed CPU clusters.
    • That result unlocked recommendation systems like YouTube’s feed and catalyzed a decade of AI-driven consumer products. Transcript: Ben Gilbert What do they do with it now? They want to do some research. So they try out, can we do cool neural network stuff? And what they do in a paper that they submitted in 2011, right at the end of the year, is, and I’ll give you the name of the paper first, building high-level features using large-scale, Unsupervised learning. But everyone just calls it the cat paper. The cat paper. You talk to anyone at Google, you talk to anyone in AI, they’re like, oh yeah, the cat paper. What they did was they trained a large nine-layer neural network to recognize cats from unlabeled frames of YouTube videos using 16,000 CPU cores on a thousand different machines. And listeners, just to like underscore how seminal this is, we actually talked with Sundar in prep for the episode, and he cited seeing the cat paper come across his desk as one of the Key moments that sticks in his brain in Google’s story. Yeah. David Rosenthal A little later on, they would do a TGIF where they would present the results of the cat paper. And you talk to people at Google, they’re like, that TGIF, oh my God, that’s when it all changed. Ben Gilbert Yeah. It proved that large neural networks could actually learn meaningful patterns without supervision and without labeled data. And not only that, it could run on a distributed system that Google built to actually make it work on their infrastructure. And that is a huge unlock of the whole thing. Google’s got this big infrastructure asset. (Time 0:41:23)
  • DNN Research Auctioned From Hotel Room
    • DNN Research (Alex, Ilya, Jeff Hinton) auctioned itself to bidders including Baidu, Microsoft, Google and DeepMind before selling to Google for $44M.
    • The acquisition brought transformative talent into Google Brain. Transcript: David Rosenthal They start a company called DNN Research, Deep Neural Network Research. This company does not have any products. This company has researchers. Who just won a big competition. And predictably, as you might imagine, it gets acquired by Google almost immediately. Oh, are you intentionally shortening this? That’s what I thought the story was. Oh, it is not immediately. Ben Gilbert Oh, okay. There’s a whole crazy thing that happens where the first bid is actually from Baidu. Oh, I did not know that. So Baidu offers $12 million. Jeff Hinton doesn’t really know how to value the company and doesn’t know if that’s fair. And so he does what any academic would do to best determine the market value of the company. He says, thank you so much. I’m going to run an auction now and I’m going to run it in a highly structured manner where every time anybody wants to bid, the clock resets and there’s another hour where anybody else Can submit another bid. David Rosenthal No way. Ben Gilbert I didn’t know this. This is crazy. He gets in touch with everyone that he knows from the research community who is now working at a big company who he thinks, hey, this would be a good place for us to do our research. That includes Baidu. That includes Google. That includes Microsoft. And there’s one other. Facebook, of course. It’s a two-year startup. Wait, so it does not include Facebook? It does not include Facebook. Think about the year. This is 2012. So Facebook’s not really in the AI game yet. They’re still trying to build their own AI lab. David Rosenthal Yeah, yeah, because Jan LeCun and Fairwood start in 2013. Ben Gilbert Is it Instagram? Nope. It is the most important part of the end of this episode. Wait, well, it can’t be Tesla because Tesla is older than that. David Rosenthal Nope. Well, OpenAI wouldn’t get founded for years. Wow. Okay. You really got me here. Ben Gilbert What company slightly predated OpenAI doing effectively the same mission? David Rosenthal Oh, of course. Of course. Hiding in plain sight. DeepMind. Wow. DeepMind, baby. Ben Gilbert They are the fourth bidder in a four-way auction for DNN Research. Now, of course, right after the bidding starts, DeepMind has to drop out. They’re a startup. They don’t actually have the cash to be able to buy. David Rosenthal Yeah. Didn’t even cross my mind because my first question was like, where the hell would they get the money? Because they had no money. Ben Gilbert But Jeff Hinton already knows and respects Demis, even though he’s just doing this at the time startup called DeepMind. David Rosenthal That’s amazing. Wait, how is DeepMind in the auction? But Facebook is not. Ben Gilbert Isn’t that wild? That’s wild. So the timing of this is concurrent with the, it was then called NIPS. Now it’s called NURPS conference. So Jeff Hinton actually runs the auction from his hotel room at the Harrah’s Casino in Lake Tahoe. David Rosenthal Oh my God. Amazing. Ben Gilbert So the bids all come in and we got to thank Cade Metz, the author of Genius Makers. Great book on the whole history of AI that we’re actually going to reference a lot in this episode. The bidding goes up and up and up. At some point, Microsoft drops out. They come back in, told you DeepMind drops out. So it’s Baidu and Google really going at the end. And finally, at some point, the researchers look at each other and they say, where do we actually want to land? We want to land at Google. And so they stop the bidding at $44 million and just say, Google, this is more than enough money. We going with you. Wow. David Rosenthal I knew it was about $40 million. I did not know that whole story. It’s almost like Google itself and the Dutch auction IPO process. Right. How fitting. Ben Gilbert That’s kind of a perfect DNA. Yes. Wow. And the three of them were supposed to split at 33 each. And Alex and Ilya go to Jeff and say, I really think you should have a bigger percent. I think you should have 40 percent and we should each have 30. And that’s how it ends up breaking down. (Time 0:52:17)