I Shot You First: Networking the Gameplay of Halo: Reach

For those who aren't in the video game development field and like technical content, I cannot recommend watching GDC's content enough (even better, attending!). Although this blog post isn't as detailed as I've seen them do in the past, I appreciate that game developers have the opportunity to share their wins like this externally. If you enjoy watching technical content and video games, the following videos are greatly entertaining deep dives into ludicrously technically complicated bugs or performance optimizations done to deliver the gaming experience that I myself take for granted. - I Shot You First: Networking the Gameplay of Halo: Reach: [this video] - 8 Frames in 16ms: Rollback Networking in Mortal Kombat and Injustice 2: [link] - It IS Rocket Science! The Physics of Rocket League Detailed: [link] - [link]

I appreciate the details this went through, especially laying out the exact timelines of operations and how overlaying those timelines produces unexpected effects. One of my all time favourite bits about distributed systems comes from the (legendary) talk at GDC - I Shot You First[1] - where the speaker describes drawing sequence diagrams with tilted arrows to represent the flow of time and asking "Where is the lag?". This method has saved me many times, all throughout my career from making games, to livestream and VoD services to now fintech. Always account for the flow of time when doing a distributed operation - time's arrow always marches forward, your systems might not. But the stale read didn't scare me nearly as much as this quote: > Since this situation had no established operational recovery procedure, engineers took care in attempting to resolve the issue with DWFM without causing further issues Everyone can make a distributed system mistake (these things are hard). But I did not expect something as core as the service managing the leases on the physical EC2 nodes to not have recovery procedure. Maybe I am reading too much into it, maybe what they meant was that they didn't have a recovery procedure for "this exact" set of circumstances, but it is a little worrying even if that were the case. EC2 is one of the original services in AWS. At this point I expect it to be so battle hardened that very few edge cases would not have been identified. It seems that the EC2 fa