2 HN users recommended this tutorial2 months2024score 3
Years with an HN recommendation, 2007–2026
HN recommends it as a long but worthwhile way to demystify how language models work technically.
Andrej Karpathy builds a bigram character-level language model from scratch, introducing torch.Tensor and the framework of training, sampling and loss evaluation. It is the first step toward a GPT-style Transformer in his makemore series.
“It's long (2h), but well worth the time if anyone is curious about the technical workings of LLMs.” (HN, 2024)
I resource I found invaluable in demystifying LLM's was Andrej Karpathy's Youtube series. In particular, his "Makemore" video built a basic LLM in the same vein as OP's. It's long (2h), but well worth the time if anyone is curious about the technical workings of LLMs. [this video]
Builds a basic language model from scratch, demystifying LLMs.
You've made something interesting. Many people have already answered this here [1], so there's no point in me explaining it again. Dropping some pointers that will be helpful for you: - [link] - Highly recommend this one [this video] - [link] - [link] 1: ex: [link]
Highly recommended pointer for next-token prediction.
This is beautiful. I'd actually been going down this same rabbit hole with sqlite, I hadn't gotten far enough to bring a neural net into it. I'd been inspired by the makemore lecture series[0]. At the 1hr mark or so, he switches from counting to using a nn, which is about as far as I've gotten. Breaking it down into a relational model is actually a really great exercise. [0] [this video]
Moves from counting bigrams to a neural net at the one hour mark.
For reference, this project is from Eureka Labs, which is the new education startup from Andrej Karpathy. * [link] It appears to be a generalized version of the bi-gram project, which he links to here: * [this video] It's cool to see him putting in even more effort on the edu work. It's a real gift he has, and was visible all the way back in his grad student projects.
Karpathy's bigram project that the new course generalizes.
Video list and Hacker News comment labels from HackerTube (prompted by @karpathy), released under ODC-By 1.0, built on the open-index/hacker-news archive. Hacker News data is provided by Y Combinator; comments remain the work of their authors. Video titles, channel names and thumbnails belong to their creators and are shown via YouTube. HackerTube Watch is an unofficial edition. Data export of 2026-10-31.
Plays via YouTube's privacy-enhanced embed
4 Hacker News comments
From the HN archive; up to 20 kept per video, favoring different commenters. Recommendations first.
Citations, criticism and jokes