The spelled-out intro to neural networks and backpropagation: building micrograd

For studying? Mainly watching and re-watching Karpathy's 'Zero To Hero'[1] and Stanford's 'Introduction to Convolutional Neural Networks for Visual Recognition'[2], also a lot of transformers from scratch videos like Umar Jamali's videos[3], and I also study backwards to McCulloch and Pitts. Reading the 30 papers [link] and so on. And of course Yannic Kilcher[4], and also listening in on the paper discussions they do on discord. Practicing a lot with just doing backpropagation by hand and making toy models by hand to get intuition for the signal flow, and building all kinds of smallish systems, e.g. how far can you push whisper, small qwen3, and kokoro to control your computer with voice? People think that deepseek/mistral/meta etc are democratizing AI, but its actually Karpathy who teaches us :) so we can understand them and make our own. [1] [this video] [2] [link] [3] [link] [4] [link]

From my other comment elsewhere. These resources helped me understand the topics better. If anyone wants to understand fundamentals of machine learning, one of the superb resources I have found is, Stanford's "Probability for computer scientists"[1]. It goes into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. But, this is a primarily a probability course that discusses the fundamentals of machine learning. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familiarity with linear algebra topics) There is a course reader for CS109 [2]. You can download pdf version of this. Caltech's learning from data was really good too, if someone is looking for theoretical understanding of ML topics [3]. There is also book for excellent caltech course[4]. Also, neural networks zero to hero is for understanding how neural networks are built from ground up [5]. [1] [link] [2] [link] [3] [link] [4] [link] [5] [this video]

That concept is not the easiest to describe succinctly inside a file like this (or -- while we are completely at it, in a Hacker News post like this!), I think (especially as there are various levels of 'beginner' to take into account here). This is considered a very entry level concept (not as an insult -- simply from an information categorization/tagging perspective here :D :)), and I think there might be others who would consider it to be noise if logged in the code or described in the comments/blogpost. After all, there was a disclaimer that you might have missed up front in the blogpost! "This post assumes familiarity with Python, NumPy, and some basic experience training neural networks." So it is in there! But in all of the firehose of info we get maybe it is not that hard to miss. However, I'm here to help! Thankfully the concept is not too terribly difficult, I believe. Effectively, the loss function compresses the task we've described with our labels from our training dataset into our neural network. This includes (ideally, at least), 'all' the information the neural network needs to perform that task well, according to the data we have, at least. If you'd like to know more about the specifics of this, I'd refer you to the original Shannon-Weaver paper on information theory -- Weaver's introduction to the topic is in plain English and accessible to (I believe) nearly anyone off of the street with enough time and energy to think through and parse some of the concepts