Spec-ulation Keynote - Rich Hickey

See this talk [0] about this addition to Clojure called core.spec [1]. I read the paper first, and it isn't until the end of the presentation that I understood how it was related to the feature at all. Basically, core.spec is a kind of formal verification tool designed to deal with the growth of software. It is not a type system, though it resembles one in some ways. The objective is to support comparisons between the signatures of a function at two different points and say, are these compatible? Is this a breaking change? You have to design for growth: make no assumptions about unspecified keys and always assume that later versions may return more than they used to. And so on. If you're a fan of semver, be warned. [0] [this video] [link] [1] [link]

> I have a hard time being too impressed because more modern languages are doing this out of the gates The problem with Java is that it's an already established and very popular platform and it's been so for the past decade at least. When starting from scratch, it's easy to just throw it all away and start fresh, it's easy to fix mistakes that were made in the past. The irony though is that we still have a hard time learning from history. Just look at Go. Sometimes this industry feels like Groundhog Day, the movie. Also, no platform or language that I know of has gotten " modules " right, with Java being one of the platforms that has gotten closest to fixing the problem actually (by means of OSGi). To see why modules are still a big problem and why we won't have a fix for the foreseeable future, I invite you to watch this keynote by Rich Hickey: [this video]

The two are not mutually exclusive. Clojure has namespaced keywords and specs[0] to cover that. (There is also the third-party malli, which takes a slightly different appproach.) The advantage is that maps are extensible. So, you can have middleware that e.g. checks authentication and authorization, adds keys to the map, that later code can check it directly. Namespacing guarantees nobody stomps on anyone else's feet. Spec/malli and friends tell you what to expect at those keys. You can sort of do the same thing in some other programming languages, but generally you're missing one of 1) typechecking 2) namespacing 3) convenience. [0]: spec-ulation keynote from a few years ago does a good job explaining the tradeoffs; [this video]

This is all excellent, and I'll add one more sub point to "Avoid becoming a bottleneck at all costs": - Be backwards compatible. If you need to break backwards compatibility, advertise it widely and put your people towards the migration. Otherwise, only expand the abilities of components, never remove them and only require less, never require more. If you want to remove from what you give or ask for more than you currently require then make a new version of the component (e. g. `components/Button/v2`). Plan for this (e. g. don't ship `components/Button`, ship `components/Button/v1`). Rich Hickey's Spec-ulation talk goes into the principles in some more detail: [this video] The reason for this is because your goal is to empower other teams to move faster. The minute that "keeping up to date with the UI library" requires any significant amount of attention you are draining any velocity you added. People will stop (either by not updating or by moving off the library).

> Ultimately, the talk tears into version numbers in general, but not semantic versioning He does talk explicitly about the trouble of making major changes in SemVer[0]. The gist of his argument was that minor changes in semver are relatively useless while major changes have a high probability of breaking your software. Major changes in semver are backwards incompatible and update the program's API in place. This leads to dowstream breakage and fear around doing upgrades. > If I need to upgrade my application due to a security vuln, SemVer lets me know if I can just "drop-in" the upgrade, or if I need to work more. I think the point he was trying to make was that upstream developers could change internals of a library but keep the API consistent so that downstream devs would never have to worry about scary updates. As you said with SemVer, if the security upgrade is a significant change, then you can expect breakage in the library. What he was advocating was patching issues like security vulns under the hood while keeping everything backwards compatible. Major upgrades could even add new namespaces, functions, and arguments but there's no real point to deleting or mutating old code, that just creates breakage. He wants software libraries to be immutable to take care of dependency issues and versioning to better reflect the changes made in the code. [0] [this video] [this video] [this video]

I don't find that too surprising, Rich tends to let things bake internally (in his head and at Cognitect) for a while before releasing anything publicly for consumption and feedback. Most of Clojure's current development is around tooling [1] and some changes to move clojure.spec out of alpha. I believe they want to accomplish this for 1.10, but I'm not sure where I remember hearing that. Alex Miller has mentioned that Cognitect built and continues to work on a tool that tracks all of your functions with spec definitions and generatively tests them only when the function or one of its dependencies changes. This would be extremely helpful since generative testing can be expensive to run, especially if you want to generate huge amounts of test cases. It'd be nice to say "test all of my codebase's functions 10 million times" and then have that information stored locally with the code. In Rich's "Effective Programs" talk [2] he outlines the "10x problems" that Clojure was targeted at solving. In it there are two 10x problems that weren't solved by Clojure: resource utilization and libraries. There's not much Clojure can do to solve resource utilization as it's dependent on the host (JVM), but the library problem (especially Clojure libraries with spec definitions) is solvable (though there seems to be some disagreement on whether only allowing growth in libraries in a practical). Pure speculation: All of their tooling projects (spec, codec, tool.deps, and the above mentioned gene

Rich Hickey also came to mind, but a different talk[0]. There he talks in part about software stability and how spec (the primary way to encode data and function contracts in Clojure), is about communicating what is required and provided by a function or library (among other things). Rust's decision to make signatures mostly explicit (sans lifetime parameters here and there), is self-documenting and useful, but most importantly it creates an explicit contract that people can rely on. Rust code adheres to semantic versioning so at least it will also adhere to not breaking callers without bumping the major version. Which the further discussion in the talk do not agree with. But the fundamental decision to make signatures explicit at least enables people to write stable software. [0] [this video]