13 HN users recommended this talk12 months2012–2025score 14
Years with an HN recommendation, 2007–2026
HN returns to it for years as an elegant introduction to writing a lexer that is easy to follow even without a parsing background.
Rob Pike shows how the Go text/template lexer is built, using state functions, goroutines and channels. The talk was given at a Google Technology User Group in Sydney in 2011.
“It's far more interesting than it sounds, and even if you don't know much about lexing/parsing it's easy to follow.” (HN, 2014)
Personally, I have a penchant for writing my own pull parsers. Its a mind-expanding exercise. The neat thing about Go is that parsers can return functions that consume the next token. Rob Pike has an excellent video about this: [this video]
Pike on parsers returning functions that consume the next token.
Rob Pike did a really good talk entitled 'Lexical Scanning In Go'[0], which covers how the lexing for text/template works. It's far more interesting than it sounds, and even if you don't know much about lexing/parsing it's easy to follow. [0] [this video]
Explains lexing in text/template; easy to follow even without parsing background.
Lexical Scanning in Go by Rob Pike [this video] I love everything about this talk. It walks you through building a lexer from scratch in a simple and elegant way, through a very interesting use of coroutines. I appreciate the bits of humor in the talk as well.
Builds a lexer from scratch simply and elegantly using coroutines, with humor.
As an aside: watching Pike's talk on lexing and parsing in Go really hits this point home. It shows how language support for cheap threads (goroutines) and channels really change the way you come up with an algorithm. [this video]
Shows how goroutines and channels change how you design algorithms.
> Building a lexer out of a recursive function type actually sounds like a really cool thing to try Rob Pike did just that for Go's text/template package ( [link] ). See the source lexer source code at [link] and note that stateFn is defined recursively. There's a video by Pike describing how he built the parser at [this video] .
Pike describes building the text/template lexer using recursive state functions.
> What most people don't realize is that for concurrency to be useful, there must be parallelism. No, as far as I understand from this and other talks, Rob Pike argues that concurrency is useful even without parallelism: it's a better way to structure software. A nice example for this is shown in his talk about lexical scanning in Go: [this video]
Example of concurrency as a way to structure software, without parallelism.
I'm not sure I like the idea of parsing templates with regular expressions. There is a better way, as demonstrated in Robe Pike's video[1] detailing how the Go template language is implemented. Edit: which isn't meant to detract from the fact that this is a great article. [1] [this video]
Shows a better alternative to regex for parsing templates.
For those who don't know who Rob Pike is, he's pretty awesome, helped create UTF-8 among other things, and now works on Go. [link] He has given some really great Go talks which are available on youtube, about an hour each: Origins of Go Concurrency Style - [link] Lexical Scanning in Go - [this video] Go Concurrency Patterns (One of my favorites) - [link]
One of several great Go talks by Pike, about an hour.
Wow, perfect timing! I'm currently writing my first lexer/parser in D as well ( [link] ) for a Scheme-like language used in Halo 3. I found Rob Pike's talk on lexical scanning to be pretty useful ( [this video] ) and I've modeled mine pretty heavily after the text/template lexer: [link] .
Useful when writing a first lexer; the commenter modeled theirs on it.
I would like to mention that most, if not all, of the Go compiler and related utilities is written in Go itself, with high quality, open source code that can be found in the "go" package of the standard library: [link] I would recommend using that as a template for one's own compiler, because it's written in idiomatic Go (originally by the language creators) and is cleanly split across abstraction boundaries (packages "scanner", "parser", "ast", and so on.) There's also a smaller, simpler parser in the "text/template/parse" package, that is used to parse the templating languages in "text/template" and "html/template". As an introduction to it, I recommend the talk "Lexical Scanning in Go" by Rob Pike: [this video]
Introduction to the text/template parser as a model for writing a compiler.
Rob Pike’s lexer was very nicely done and struck a chord with me. It’s not nearly as magical as some of the other things here. It’s just great engineering: [this video]
If you enjoyed the state machine + parsing part and also I recommend the older "Lexical Scanning in Go" talk that Rob Pike gave: [this video] It's in Go, but you can apply most of it easily in other languages. I'm so happy that modern languages like Rust, Zig, and Go support unicode/runes/graphemes natively. So many problem just disappear compared to Java, .net, C++, or scripting languages.
Older talk whose lexing approach applies easily to other languages.
Check out the std lib: [link] -- that's canonical Go code. It's my language of choice for personal projects. A neat and compact language that is very trivial to pick up, feels light, and yet sufficiently subtle to intellectually engage your mind. Very productive, good performance, and very portable. [this video] caveat: /imho/ the runtime is not yet fully cooked. ETA is Go1.1 (end of year I believe) for beefed up internals.
type state struct { // add any fields you need to persist here } type transition func(*state) transition same as in Lexical Scanning in Go ( [this video] )
State function pattern matches the one from the talk.
I thought this would be a good opportunity to use 'state functions' to implement a shell for Bytengine. Rob Pike mentions the use of state functions in his talk "Lexical Scanning in Go" [this video]
Pike's use of state functions inspired the commenter's shell implementation.
There are many explanations. The most famous one Rob Pike in his lexer talk 2011 [this video] at 10.48 Or [link] Or several articles why you should not parse with regex, like [link]
Pike's lexer talk explains why not to parse with regex.
> I think Go spent a lot of time implying that goroutines were some sort of special language magic, but I think everybody knows now that they are basically a really good implementation of green threads that can take advantage of some internal knowledge to do optimal scheduling to avoid context switches. It took me a while to work this out, and I got a lot of pushback from the go team when I pointed this out internally. This comment deeply fascinates me, because I get the same feeling every time I go back and read/watch early Go resources from Rob Pike and others, but I've never actually heard it articulated before. I've always thought that surely they weren't that ignorant about PL theory and history? Or maybe Rob Pike himself was, but surely his team knew better? It really feels like they thought they were making some special hybrid between threads and coroutines, combining the advantages of both. Like this 2011 talk[1] which demonstrated a classic coroutine code pattern implemented with goroutines and channels. But as time went on, goroutines became fully pre-emptive, and their programming model is now identical to threads. Using them as coroutines only leads to misery (read: data races) in my experience. They're faster than OS threads, sure, but that's an implementation detail at this point. When do you think they eventually realized they were re-inventing the same green threads Java had created and scrapped 10 years earlier? [1] [this video]
Video list and Hacker News comment labels from HackerTube (prompted by @karpathy), released under ODC-By 1.0, built on the open-index/hacker-news archive. Hacker News data is provided by Y Combinator; comments remain the work of their authors. Video titles, channel names and thumbnails belong to their creators and are shown via YouTube. HackerTube Watch is an unofficial edition. Data export of 2026-10-31.
Plays via YouTube's privacy-enhanced embed
20 Hacker News comments
From the HN archive; up to 20 kept per video, favoring different commenters. Recommendations first.
Read more
Citations, criticism and jokes
Read more