Jim Keller: Moore's Law, Microprocessors, and First Principles | Lex Fridman Podcast #70

I just wanted to say this about Intel vs AMD. Back when Intel was the "underdog" it couldn't compete with Athlon/Opeteron CPUs from AMD with its Pentium IV line, fan boys from both sides continually escalated the armchair warfare on Slashdot and Anandtech forums when Intel finally came out with the Conroe architecture that blew the competition away. I've seen this pendulum swing from one side to the other and back. Putting Intel marketing (awful) and AMD's marketing (awful again...), and their fan bases (toxic) aside, can we acknowledge the fact that working on a computer architecture is an extraordinarily complex task and that requires brilliant people all working together? I'll defer this fascinating topic, which most software engineers and tech enthusiasts are completely and utterly unaware of, to this talk by the legendary chip designer (x86 spec coauthor, A4/A5/Zen/Tesla AI ASIC architect) - Jim Keller and his interview with Lex Fridman: [this video] Designing & manufacturing computer chips is hard af, having worked in semiconductor manufacturing for over 12 years (primarily on the backend side of the Fab). Edit: Redacting some specifics

Yeah that's how RISC vs CISC is taught in class, I've heard that same thing. I think it's an outdated paradigm though, if it's not been wrong all along. A CISC chip can still officially support instructions, but deprecate them and implement them very slowly in microcode, as is done with some x86 instructions. And a RISC chip manufacturer might just have used this phrase as a marketing line because designing RISC chips is easier than starting with a monster that has tons of instructions, and they needed some catchy phrase to market their alternative. They then get into the gradual process of making their customers happier one by one by adding special, mostly cold, silicon targeted for special workflows, like in this instance. Ultimately, the instruction set isn't that relevant anyways, what's more relevant is how the microcode can handle speculative execution for common workflows. There's a great podcast/interview from the x86 specification author: [this video]

I've been watching Lex Fridman's youtube podcast and there is a recent interview with Jim Keller [1]. Keller is a chip designer famous for his involvement in multiple chips at Intel, AMD, Apple and he was co-author of the x86-64 instruction set. He also worked for Tesla. There is a point in the conversation where Lex and Jim clearly disagree about how "easy" self-driving AI should be. Lex is clearly pessimistic and Jim is clearly optimistic. I have to admit I was more swayed by Lex's points than by Jim's, but it is hard to discount someone so clearly (extraordinarily) expert and working directly in the field. 1. [this video]

If you noticed how insanely Slack freaked out and handled the service outage. The application just got stuck, didn't know what to do, I see a faint 503 error message which looks like a webpage nginx error displayed right in the app. Clearing cache didn't help and the app just quit. Electron apps are a nuisance and should be eradicated from this planet. We've come a full circle -> Bloat the web -> Create javascript frameworks -> Build SPA/PWA apps -> Invade the desktop land -> Gain traction because "its crossplatform" and now try to reimplement GUI elements from some fancy_new framework that is developed by people who are entirely unqualified for it. Throw away all the work that's done by OS developers and designers that are getting paid as professionals to develop guidelines like this: [link] The modern tech stack needs to be refactored just like they refactor silicon architecture every 10 years: [this video] Edit: People talking about would native app handle this better? Sure. I argue that it wouldn't just get rid of all UI elements and drop the entire DOM on the page, the UI just crashed. And everytime I open slack it seems like its doing something else entirely. There is a massive difference in quality of native UI libraries and Bulma for example, and its not just the library, you gotta rely on React to redraw things and it depends on 4 billion npm dependencies. The web truly sucks. And it is now eating into desktop apps.

Moore's law might be dead but the deeper law is still alive. Moore's law is technically "the number of transistors per unit area doubles every 24 months" [1]. The more important law is that the cost of transistors halves every 18-24 months. That is, Moore's law talks about how many transistors we can pack into a unit area. The deeper issue is how much it costs . If we can only pack in a certain amount transistors per area but the cost drops exponentially, we still see massive gains. There's also Wright's law that comes into play [3] that talks about dropping exponential costs just from institutional knowledge (2x in production leads to (.75-.9)x in cost). [1] [link] [2] [this video] [3] [link]

> I expect within the next 6-15 years that we'll hit this limit You are talking about fundamental physical limits? Um... No, [1] "At about 26-28.5 minutes of the Jim Keller: Jim notes that transistors could reach 10 atoms by 10 atoms by 10 atoms while avoiding quantum effects. People are also working on harnessing quantum effects." We are so far away from fundamental limits. That even if we assume we could somehow double transistor density every two years, which we dont anymore, there are still at least another 15-20 years before we are close to that limit. And that is not accounting any 3D Transistors. Realistically we already have Roadmap up to 2030 from TSMC. The only limit we will hit is that the node is too expensive and market could no longer afford the premium. Which You could expect to happen within next 10-15 years. [1] [this video]

I'm generally against rewrites as they're often shortcuts for new developers that instantly reject a code base as 'too complicated' without understanding it. However, I have had success in rewriting my own code base a number of times. This is a unique situation of a one-man-band that understands the business case, handles support, and every bit that went into every decision of the code base. With this kind of clarity, when you've spent years with the problem domain, you can uniquely rewrite a project to get at the real business case that, now with years of experience, you see what your customers actually wanted you to solve in the first place, or maybe what they evolved to want in the end. Jim Keller advocates rewrites in a similar way as the only way to progress in chip design. I think we could rewrite very basic things in exciting ways with this sort of attitude. For instance, we know a lot more about what we need in a desktop OS, an email client, a search engine, etc. Basic things that have gotten a lot of cruft over the years as they evolved. Taking a fresh look could be rewarding. I guess one could reframe this as 'first principals thinking" but with the caveat that the problem needs to be truly understood. Here's Jim Keller talking about it: [this video]

> By 2025, either these models will have hit a wall on diminishing returns and it will take a complete pivot to some other approach to continue to see notable gains, or the products will continue to have improved at compounding rates and access will have become business critical in any industry with a modicum of competition. Is there a single example in AI, or even technology as a whole, where simply continuing to apply one technique has led to compounding growth? Even Moore's Law, according to none other than Jim Keller[1], is more a consequence of thousands of individual innovations that are each quite distinct from others but that build on each other to create this compounding growth we see. There is no similar curve for AI. In this case, GPT-3 (released in 2020) uses the same architecture as GPT-2 (released in 2019), expanded to have ~100x more parameters. It's not hard to see that compounding this process will rapidly hit diminishing returns quickly in terms of time, power consumption, cost of hardware, etc. Honestly, if Google, Amazon and Microsoft didn't see increased computational cost as a financial benefit for their cloud services, people might be willing to admit that GPT-3 is a diminishing return itself: for 100x parameters, is GPT-3 over 100x better than GPT-2? It seems that the big quantum leaps in AI come from new architectures applied in just the right way. CNNs and now transformers (based on multi-head attention) are the ways we've found to scale this thing,