dotScale 2015 - John Graham-Cumming - I got 10 trillion problems, but logging ain't one

You make incredibly fair points and I don't have the data to answer what you've raised. This could be lack of research or the fact that CF hasn't released it. I expect a 20/80 non-researched/non-available mix. My understanding is that the WAF detects trivial stuff like SQL injection and XSS, and the video also mentions it detected ShellShock and so forth (yeah, this was a while ago). "Thousands of regular expressions" is mentioned verbally in the video around ~11:25. That's where I got that from. There are some bits mentioned about the hardware around 19:30: - E5-2630v3 (16 cores, 32 w/ HT which is turned off), 128GB RAM, 10Gbps - Custom built by Quanta: 4 nodes per 2U (each node is like half-width and 1U high), redundant 2xPSU in the middle - 40 machines run Kafka storing 400TB of log data (ingest @ 15Gbps), 50TB disk per node - 5 machines run CitusDB (sharded PostgreSQL) to power realtime graph generation, 12TB SSD per node - 100+ machines for realtime analytics running Go (and presumably LuaJIT) - LuaJIT seems to be the realtime "hmm, interesting" flagging system, while Go is used for in-depth intelligent analysis that's (I assume) slightly behind realtime. Also, the video is actually over here - [this video] --- I'm curious to understand what tail latency is, and very interested to learn more about language latency and performance in general. What subjects would you recommend I learn about / read? (I would describe myself as interested from a hobbyist-intermediate standpont, b

> Even putting aside that the computer tech part of the business isn't their core competency, custom hardware is hard. Hmmm. Good point. (Maybe I scared OP away, heh.) > Is there an example of any large company, other than Google, who has mastered this? Well, I certainly don't know. I've heard Cloudflare gets their systems made, for some definition of "from scratch", by Quanta. Presumably there's a bit of a decision-making process in there, and presumably Quanta handles everything related to the "why is the devkit sample broken again #51383" side of things and ships a more or less finished product (maybe even with finished enclosures). My reference point for this knowledge is [this video] (2015). Regarding SAS, maybe past a certain ordering quantity (or anything similar that would reveal a massive operation) the market would expect certain behavior like requesting value-adds, ordering with only one supplier, or other things that would generate lock-in. I of course have absolutely no idea [if this sort of thinking is reasonable]. Your points about commodity at scale sound quite reasonable. (You clearly have more experience than I do, which is kind of zero, heh.) As for suspicion about summaries and conclusions, well, (tangential example) the radial graph on [link] (the one from Forrester) doesn't show GitHub :D Yeah, building own hardware is sadly still within the realm of over-specialization at this point. I guess this is because 4+-layer PCB printing is still eye-watering so the