Cited on HN 3 times, 2023–2026, never outright recommendedscore 0
Years with an HN recommendation, 2007–2026
Cited in discussions of learning as compression and of why neural networks generalize.
Ilya Sutskever gives a Simons Institute talk on why neural networks generalize, framing learning as finding short descriptions, like compression. It was part of the Large Language Models and Transformers program.
“Ilya argues that the reason why neural networks generalize -- why they work at all -- is because they are effectively finding a simple description” (HN, 2026)
I am curious why isn't the network always in an "interference" state, but sometimes it collapses into a "restricted" ONB? > the neural networks we observe in practice are in some sense noisily simulating larger, highly sparse networks This seems somewhat related to a point made by Ilya Sutskever here [1]: NNs can be though of an approximation to the Kolmogorov compressor. Speculating, one could say any network is a projection of the ideal compressor (which argualy perfectly represents all n-features in an n-dimensional ONB) into a lower dimentional space, hence the interference. But why is there not always such an interference? [1] [this video]
Sutskever's view of NNs as approximating a Kolmogorov compressor.
> We are now getting an equivalent definition of what neural nets are being trained for! LLMs are trained to compress the internet as much as possible! Nice payoff. Others have also called out the relationship to compression ( [this video] ).
Noting the theory papers on Kolmorogov complexity. For those not familiar, Ilya argues that the reason why neural networks generalize -- why they work at all -- is because they are effectively finding a simple description of their training data , converging down onto the limit of the Kolmorogov complexity. [1] [1] [this video]
Ilya's argument for why neural networks generalize.
Video list and Hacker News comment labels from HackerTube (prompted by @karpathy), released under ODC-By 1.0, built on the open-index/hacker-news archive. Hacker News data is provided by Y Combinator; comments remain the work of their authors. Video titles, channel names and thumbnails belong to their creators and are shown via YouTube. HackerTube Watch is an unofficial edition. Data export of 2026-10-31.
Plays via YouTube's privacy-enhanced embed
3 Hacker News comments
From the HN archive; up to 20 kept per video, favoring different commenters. Recommendations first.
Read more