Plenary: Performance Matters - Emery Berger - CppCon 2020

This guy has had a few talks at CppCon over the years, and covers the basics of how to attempt benchmarking with actual computer science instead of myth and voodoo rituals: [this video] In practice, it's not critical to go that far. In my experience, huge performance wins are trivial to benchmark with just a modicum of effort, and tiny performance wins are typically not worth any effort at all. At FAANG scale they might be, but I don't work at a FAANG, so not my problem! :) Simple methods I use and can recommend are: - Ideally, collect data from production, where the change behaves exactly as it would in production because it is production! - Interleave tests on the same machine. E.g.: A, B, A, B, A, B, etc... - Run tests on every machine you can get your hands on over an extended period and draw a histogram or a scatter chart of some sort. You don't have to do this at 100% load, it's actually better to mix this in with normal workloads. This gives you an idea of the noise versus signal, platform variance, variation due to time of day, etc... - Run tests after a cold boot to get a sense of worst-case performance instead of idealised and often unrealistic "hot loop" performance. - Run tests with different input sizes and draw a graph. Try find the explanation for every feature of the graph. - Be aware of every cache layer in the system. Don't forget about the potentially huge RAM or SSD cache of the storage subsystem. Typical SAN arrays now have 256 GB or more. The previous graph a

To give an oversimplified answer, that depends on how things are connected and synchronized. You know how bottlenecks are often explained using funnels in real life? Imagine two funnels, one directly connecting to another one. In that case, the effective rate of flow is determined by the funnel which has the smallest tip - the fact that the larger one lets through fluid quicker is irrelevant, as it is slowed down by the smaller one (we're obviously ignoring things like funnels potentially overflowing here, this is a simplified scenario). So then the bottleneck is easy to pin-point, but also it immediately becomes obvious that widening one funnel beyond the size of the other has no effect. Now let's change the scenario: I have a bucket of water that I tip into the first funnel, going into a second bucket. Once the second bucket is full I tip the second bucket into the second funnel, going into a third bucket. In this case both funnels have an impact on the time it takes to fill the second bucket, but the smaller one sill has the bigger impact. And then of course there is the parallel scenario, you can probably see how that plays out. In our software environment is a complicated mix of all of these, and it can be really hard to pin-point what is the most significant effect. The "buckets" and "funnels" translate to all kinds of things - CPU an I/O are the most obvious ones, but there's more to it than that. Also there's tons of side-effects that make this entire picture a lot mo