"How NOT to Measure Latency" by Gil Tene 

Key takeaways for me : 1. Most of the tools that capture the response times, report 99 percentile latency of every 30 sec duration. For example prometheus metrics are scraped every one minute. But the real thing to look at is the Max response time. 2. Gatling fixed the co-ordinated omission problem. Most of the other tools like Jmeter, etc still have this problem. So use Gatling for your load generation and reporting purposes. 3. Didn't understand co-ordinated omission fully. But I'm now informed that it is bad and needs to be looked out for. 4. When a graph shows sudden spike, it is an indication of a 'possible' coordinated omission. If a graph is smoothly growing it is an indication that there is no bad data. Exceptions maybe there to this rule. 5. There is no point in looking at percentile graphs if you don't have performance goals set for your service. If you are comparing two systems and your target is 20ms, then you could plot graphs and see what is the maximum throughput each system supports while maintaining latency at 20 ms.

10 Minutes in and I already see the very obvious flaw that +Gil Tene pointed out in my day-to-day monitoring. Great talk!

Coordinate emission: One issue here is one that is often encountered in metrics in business, and that is that the bosses want simple, easy, and reliable numbers to look at. To the guy behind the project it is seen as a system that ions out a rare case, without understanding the maths behind it.

Terrific talk about load testing and lazy thinking. The early part probably applies to anyone thinking about metrics for a complex system.

Nice talk, I've read good things about it. Now starting to listen, looking forward to finding some good stuff about performance testing.

[6:52] I don't have the data. A common problem we have is we plot only what is convenient. We only plot what gives us nice colorful charts. We choose the noise to display.

I'm more used to graphs being split for request kinds. To me the first thing that jumped out was the large difference between 50th and 75th percentile.

@12:04, is this because of 5 page loads/40 resources per page increasing the chance of hitting above p99? If that’s the case couldn’t you just adjust each graph to be on a per-resource or per-page basis? Which seems like it would directly reflect the percentile.

Great talk! I'm curious, we get most of our production response time stats from AWS load balancer logs. Is that an accurate measure of response time?

19:34 should be 100 measurements of 100s each , no? 100 requests will be sent that second and each will be stalled for 100s.

Anyone can write all the tool names mentioned in the presentation? I can't find some of them, probably because I'm not writing the name correctly.

Ok, i'm only a few mins in and my brain hurts.. I can't belive that people seriously ignore the max in things like this.. scary.