# What are percentiles and why do they matter?

#### August 1, 2013

##### By @pauldix

It's nearly impossible to talk about metrics and instrumentation without hearing about percentiles. In this post we'll define what percentiles are and explore some examples that highlight their importance in measuring performance in aggregate.

Before we get into percentiles let's lay out a specific example and talk about averages. Specifically, the mean. Say we're tracking response times in our application and we have the following response times measured in milliseconds from 20 requests to our application:

``````times = [44, 558, 41, 45, 46, 43, 612, 43, 43, 40, 41,
43, 44, 49, 41, 49, 40, 48, 48, 45]``````

Let's look at the mean response time. We care about the mean because usually it's not possible to examine or visualize individual requests because it's too much data. The mean is a useful tool to summarize what the response time was for the average user request. It can be calulcated as the sum divided by the count like so:

``````sum = times.inject {|s, i| s + i} # => 1963
mean = sum / times.length.to_f # => 98.15``````

Looking only at the mean, it seems like we're doing pretty well. Average response time of 98.15ms? Under 100 milliseconds? I read somewhere that humans can't perceive time much faster than that so that seems good to me. However, if you look closer at the individual values, you see that two of those requests took a significant amount of time. Greater than 500ms, which is totally unacceptable. That's where percentiles come in.

Percentiles are another tool in statistics for summarizing a collection of values. Percentiles are a single value that describe what percentage of values fall below a given value. For example, let's look at four different percentiles: 50th, 80th, 90th, and 95th. The 50th percentile, also known as the median, is the number in the collection for which 50% of values are less or equal than it. We can calculate that like so:

``````ordered_times = times.sort
# => [40, 40, 41, 41, 41, 43, 43, 43, 43, 44, 44, 45, 45, 46, 48, 48, 49, 49, 558, 612]

# get the value in the middle of the array
percentile_fifty = ordered_times[(times.length * 0.5).to_i] # => 44``````

So 44 is the 50th percentile. That's the value in the middle of the array. That is, 50% of the requests were <= to 44 and 50% were >= 44. Here are the other percentiles:

``````
# get the value 8/10ths of the way through the array
percentile_eighty = ordered_times[(times.length * 0.8).to_i] # => 49
# etc.
percentile_ninety = ordered_times[(times.length * 0.9).to_i] # => 558
percentile_ninetyfive = ordered_times[(times.length * 0.95).to_i] # => 612``````

Looking at the 80th percentile, things still look pretty good. Less than 49 milliseconds. However, when we get to the 90th percentile we see things slowing down. That means that 1 in 10 requests had a response time >= 558 milliseconds.

Ok then, let's just track the 90th percentile and we'll be all set, right? Well maybe not. Let's say there's one more request in there at the beginning that finished at 40ms. Here's what that does to our calculation:

``````ordered_times.unshift(40)
# => [40, 40, 40, 41, 41, 41, 43, 43, 43, 43, 44, 44, 45, 45, 46, 48, 48, 49, 49, 558, 612]

percentile_nintey = ordered_times[(ordered_times.length * 0.9).to_i] # => 49 ``````

All of a sudden our 90th percentile is looking respectable again at 49ms. So having multiple percentiles values to look at can be very helpful in seeing how we're performing.

Now the question, are these values even reasonable? Maybe I've just set up a strawman to shoot down the value of the 90th percentile measuremnt. I have to admit that I engineered these values, but it's very common in performance measurements to have clusters of performance. The fast requests and the slow requests. How many clusters you have and what percentiles they fall into are determined by a number of factors. It could be garbage collection, a background job that kicks off every so often that slows things down, your cache hit ratio, or events that cause a rush of traffic to your site.

In a previous post I talked about measuring cache hit ratios. Your cache hit ratio can have a direct impact on which percentiles show good vs. poor performance. For instance, if 19 out of 20 requests have a cache hit and one request has a cache miss, you won't see that reflected in your metrics unless you're looking at the 95th percentile or greater. Your mean and all the lower percentiles will tell a story of fantastic performance that just won't be true occasionally.

There are a bunch of tools out there that help you measure percentiles. StatsD is a well known open source project that will do these calculations for you and send them to a metrics system like Graphite or even Errplane. Errplane also has an aggregator that can be fed into directly without using StatsD as an intermediary.

##### We're writing a book on monitoring!

This blog is our place to explore ideas and topics in metrics, insrumentation, and monitoring. We'll be talking to customers, testing out tools and compiling our experiences into a free e-book. Fill out the form below to get updates on new monitoring related posts and receive a free copy of the book when it's released.