Frame from Conflict Resolution for Eventual Consistency • Martin Kleppmann • GOTO 2016 Plays with YouTube's privacy-enhanced embed

Conflict Resolution for Eventual Consistency • Martin Kleppmann • GOTO 2016

Open on YouTube First HN thread

3 HN users recommended this talk in 7 different months between 2017 and 2020.

2007–2026
5 strong recommendations 6 recommendations 2 citations HackerTube score 9

Treated as the go-to approachable yet deep primer on CRDTs and collaborative editing, recommended repeatedly in threads on Automerge, local-first software and OT versus CRDT.

Martin Kleppmann's GOTO Berlin 2016 talk on how collaborative editors, offline apps, and multi-datacenter databases handle concurrent updates. It covers conflict resolution approaches including operational transformation and CRDTs, with a worked look at text CRDTs.

Pull quote: “Rest of the talk is also highly recommended for anyone interested in an approachable primer on CRDTs.” (HN, 2019)

Also tagged: crdts and eventual consistency, crdt, collaborative editing, operational transformation, automerge, distributed systems

What Hacker News said

13 labeled comments from the HN archive; up to 20 are kept per video, favoring different commenters. Each one links back to the original thread.

Strongly recommends marknadal September 2017 under Distributed algorithms in NoSQL databases
Kleppmann's talk on CRDTs for decentralized collaborative editing.
Read the full comment
Is this website related to the popular [link] ? Holy cow, this is an intense article, with great definitions for a bunch of detailed variations of distribute system jargon, and even has some nice images to explain the ideas! My specialty is in eventually consistent data types, which there section is much smaller on, but they have some links to follow up on CRDT stuff. > are often limited in functionality and impose performance overheads This is for the most part true, with CRDTs. However we just formalized and proposed in a new paper, with some Stanford colleagues, how to construct a generalizable CRDT. One that actually lets other more case-specific CRDTs be built on top of it, which we use at gunDB, so that way you can get the advantages of the more optimized CRDTs but still have the rest of your data glued together. Another set of really interesting CRDTs which they didn't mention, are the ones used for making decentralized versions of Google Doc: - Martin's Kleppmann has an exceptionally great/interesting talk on this: [this video] - And we recently did a "layman" animated explainer of a similar algorithm, that explains it for the non-academically minded, with simple analogies to "distributed systems" that happen in every day life: [link] I'll definitely be referencing the OP's article in the future though. Even though I'm in this world, I constantly get the vernacular mixed up. And this will be a really great dictionary to use.
Strongly recommends marknadal February 2018 under Automerge: JSON-like data structure for building collaborative apps
Kleppmann's outstanding talk on CRDTs.
I've spent almost a decade working on CRDTs, largely from an industry perspective but also some academic side. They're great, and not too hard to understand if you take time to think about them. Here are some animated/interactive cartoon explainers we've made to help teach the concepts: - How a CRDT version of Operational Transformation (Google Docs) works: [link] - Why CRDTs are better than centralized alternatives like PAXOS/RAFT: [link] Martin Kleppmann's work (author of Automerge) is outstanding, especially check out: [this video]
Strongly recommends lachenmayer March 2019 under Applied Monotonicity: A Brief History of CRDTs in Riak
Kleppmann gives great in-depth talks on Automerge CRDTs.
CRDTs are quite actively discussed in the IPFS community, in particular here: [link] (lots of great resources if you're interested in CRDTs) Also check out automerge, a CRDT that aims to be as JSON-like as possible: [link] Martin Kleppmann (one of the creators of Automerge) has given several great in-depth talks about it, eg. [this video]
Strongly recommends marknadal March 2019 under Applied Monotonicity: A Brief History of CRDTs in Riak
Excellent talk on CRDTs.
CRDTs are great, and I've seen a lot of quality in Christopher's work. A few other resources for those interested in CRDT subject: - Our cartoon explainer [link] - An excellent talk by Martin Kleppmann [this video] - Marc Shapiro's in depth talk [link]
Strongly recommends lewisl9029 May 2019 under Why CRDT didn't work out as well for collaborative editing xi-editor
Approachable CRDT primer with illustrative example of semantic pitfalls.
Read the full comment
> Indeed, the literature of CRDT does specify a mathematically correct answer. But this does not always line up with what humans would find the most faithful rendering of intent. This is a very salient point that anyone thinking of using CRDTs to "solve" synchronization in an user-facing application needs to take into consideration. Yes, CRDTs will guarantee that clients converge to an identical, mathematically "consistent" state eventually, but there's no guarantee whether or not that mathematically "consistent" state would make any sense to the application business logic that needs to consume that state, or to the human that needs to reason about the rendered result. That is a completely different can of worms that we'll still have to tackle to build a usable application. Here's great example to illustrate this from Martin Kleppmann's talk on this topic: [this video] Rest of the talk is also highly recommended for anyone interested in an approachable primer on CRDTs. The trade-offs to CRDTs mentioned by the author in the context of text-editors make sense, but I would be curious to hear from the Xray team on what their current thinking on the topic is, given that they have collaborative editing as an explicit core objective (which might shift the value prop in favor of using CRDTs relatively speaking since in Xi it seems to be only an aspirational goal), and that their approach to implementation was similar but not quite identical to Xi's: > Our use of a CRDT is similar to the X
Detailed explanation of how realtime syncing CRDTs work.
Cool concept, but note: I just tested this, and it is NOT a google docs alternative - it has no realtime sync (or at least, it wasn't working). I work extensively on decentralized/P2P distributed systems and realtime P2P sync is non-trivial, you need to use a specialized CRDT for it: - Here is an interactive animated explainer on how to do a P2P realtime syncing CRDT: [link] - Here is an excellent more detailed explanation of how such a CRDT works, by Martin Kleppmann: [this video]
Recommends marknadal July 2018 under Differential Synchronization (2009)
Good follow-up on CRDTs from Kleppmann.
Neil Fraser went on to apply this work at Google with Google Docs (Operational Transformation, see other comment here). He was also one of my heroes that inspired my work to build a decentralized (and offline-first) version, that I later applied to graphs with [link] . Some good follow ups are: - Martin Kleppmann's [this video] - Cartoon explainer [link]
Recommends marknadal July 2018 under Resistant protocols: How decentralization evolves
Thorough technical overview of CRDT algorithms by a researcher.
Yes, we did a whole interactive explainer article on this: [link] (cartoon version) Although, from a algorithm side, you may enjoy a talk by Martin (a CRDT researcher) for a more thorough technical overview: [this video] Our team has a model for applying end-to-end encryption to this as well. It isn't a high priority for us to do it ourselves though, so if you are interested in getting involved, we can help explain all the algorithms for implementation!
Recommends marknadal May 2019 under Open source collaborative text editors
Detailed deep dive into text CRDTs including RGA.
For anyone who wants a quick primer on OT / CRDT text algorithms: [link] ( Cartoon Explainer! ) For a detailed deep dive, including RGA: [this video]
Recommends marknadal January 2020 under OT and CRDT trade-offs for Real-Time collaboration
Good talk with many paper references on CRDT algorithms.
Started, but never finished. :( @nmaro has some pretty good stuff. The cartoon explainers gives an overview. But you'd probably want to do RGA algorithm or one of the newer ones. Martin Kleppmann has a pretty good talk with bunch of paper references: [this video]

Citations, criticism and jokes

Read the full comment
Hey, David/others I'm very impressed with your work on Y-JS (the CRDT library powering this). I'm the author of gun, which is currently (as far as I know from stats) the most popular CRDT-based data sync (MIT/ZLIB/Apache2 licensed, 6.6K+ stars, [link] ). Usually I'm pretty skeptical of most systems, but I'm rather particularly impressed with yours. I also do not think you are getting the recognition you deserve, because the truly novel work here is Y-JS not IPFS (even though is IPFS is great), yet Y-JS is almost just a footnote on the page which is sad. The first couple documents (on architecture and stuff) didn't seem to explore the CRDT that is used. I'll have to follow-through on other sub-documents/pages to read more, but wanted to get your thoughts first (and BTW, thanks for all the documentation you have already provided, it is impressive, and I know how hard it is to try and get everything shipped, so the last thing I want to be is an annoying HN person whining about why you didn't already have X or Y or Z finished, and wanted to give you a chance to reply directly). I was able to rather quickly ( [link] ) get the document to go out of sync, but I understand this could just be a trivial implementation detail (and often is) NOT a problem with the CRDT, but wanted to check. Given your bachelor thesis, I know you are all too aware of these things, P2P collaborative rich-text is one of the hardest problems out there in the space. Your Y-JS system seems fairly "generalizabl

Keyboard shortcuts

/
Search
j k
Next / previous video in a list
Enter
Open the focused video
l
Like (on a video page)
w
Watch later (on a video page)
?
This help
Esc
Close