Plays via YouTube's privacy-enhanced embed

Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP

Name: Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP
Description: Sasha Rush surveys linear RNNs and state space models as replacements for attention in transformer-style models, in a talk for MLSys. It covers the foundational preliminaries and predates Mamba.

Sasha Rush

Lectures

YouTube

3 HN users recommended this lecture3 months2024 score 3.5

Years with an HN recommendation, 2007–2026

HN commenters recommend it for building intuition about state space models and the tradeoffs between transformer and SSM layers.

Sasha Rush surveys linear RNNs and state space models as replacements for attention in transformer-style models, in a talk for MLSys. It covers the foundational preliminaries and predates Mamba.

“I highly recommend Sasha Rush's video on it” (HN, 2024)

1 strong recommendations · 2 recommendations · Lectures · advanced · partly dated

Machine learning & deep learning LLMs & generative AI Lectures AI, Machine Learning & Data

3 Hacker News comments

From the HN archive; up to 20 kept per video, favoring different commenters. Recommendations first.

@a_wild_dandan Mar 2024 Strongly recommends

on HN thread Jamba: Production-grade Mamba-based AI model

To those curious about the tradeoffs between transformer and state space model layers, I highly recommend Sasha Rush's video on it: [this video]

Explains tradeoffs between transformer and state space model layers.

View on Hacker News

@anomaly72 Jan 2024 Recommends

on HN thread MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts

Would recommend Sasha Rush's lecture [this video] , it provides the intuition for state space models via linear RNNs

Gives intuition for state space models via linear RNNs.

View on Hacker News

@srush Feb 2024 Recommends

on HN thread The Annotated S4

Hi! Blog author. This was an attempt a couple years ago to understand and write about this paper in a detailed way. Here is a video going through this topic as well: [this video] Section 2 of the blog post is no longer very relevant. A lot of advances (DSS, S4D) simplified that part of the process. Arguably also this all should be updated for Mamba (same authors).

Author points to a video walkthrough of the S4 topic.

View on Hacker News

Video list and Hacker News comment labels from HackerTube (prompted by @karpathy), released under ODC-By 1.0, built on the open-index/hacker-news archive. Hacker News data is provided by Y Combinator; comments remain the work of their authors. Video titles, channel names and thumbnails belong to their creators and are shown via YouTube. HackerTube Watch is an unofficial edition. Data export of 2026-10-31.