Amazon's Reserved Instance Planning QuandaryAmazon EC2 Reserved Instances allow you to buy capacity at a discount compared to On-Demand pricing (38-39% discount for 1yr commitment on m3-instance types). The challenge is planning for the annual contract. Amazon "usage reports" help a little, but only provide 7 days of data at the granularity we need to inform our decision. Seven days is not a confidence-inspiring period on which to make an annual commitment (especially for businesses with seasonality). Also, the tools don't show the percentile of a given instance count (more on this below).Thankfully we use our own analytics platform internally and I was able to use it to generate the view of AWS instances I needed:hourly counts (or better, if necessary)IMby instance typeby Availability Zoneover a customizable period of time (months is great, years will be better)counts at various percentilesOur Amazon Instance-Count DataWith a little scripting and some time gathering data, we know exactly what our ec2 usage looks like. There are strong daily and weekly patterns, corresponding to automated testing - some of that's scheduled and some is triggered by peeps checking code in. The crux: after 2 months, we never dropped below 130 m3.medium instances, even at oh-dark-hundred.Based on that data, we began to purchase RIs within a week and continued to ramp up over 2 months, as data continued to guide our purchases:After the first week , we bought 50 m3.medium instances. (total RIs: 50)One month in, another 50 m3.mediums. (total RIs: 100)Second month in, 30 m3.mediums. (total RIs: 130)We'll continue to purchase until we hit the 30th percentile. (probably only another month)Think "laddered bond"Discount PercentileGiven the discount rate, an instance doesn't need to be up 100% of the time to be worthy of an RI purchase- it just has to be up more than 62% of the time. Plotting the 38th percentile, we can get that count. However, the closer to the threshold we get, the less of a discount we realize, and the less sense it makes to take a chance on the annual commitment. Ultimately, I used the 30th percentile, to allow margin for error. I also wrote the Juttle with "input" dropdowns to facilitate toggling among different percentiles.Where Using Jut ShinedMost importantly, without Jut, I couldn't have shown more than 7 days of granular data. However Jut also provided three qualities I could not get going the traditional db/sql solution: shareable output (via URL), interactive reporting, real-time analysis.A few more key features:Dropdown inputs to allow quick toggling amonginstance types - we also purchased m3.2xlarges and m3.largesavailability zones - RIs commit you to using a given az. Toggling among the different AZs showed that we could consolidate to us-west-2c and make the cat herding easier. Alternately, we could purchase in those AZs.percentile - useful to get a feel for data, and to gauge how big of a risk to takeperiod -again, useful to get a feel for data and how quickly usage is changingany data the AWS api provides and that we queryZoom slider at the bottom facilitates drill down - useful around inflection pointsAs an example of the programmable "inputs", here is the Juttle I used to get instance types from the AWS data, dynamically. The "input" allows the user to choose their desired instance type from a list: input input_instancetype: combobox
-label 'Instance Type'
-juttle "read -source_type 'metric' -space 'aws' -from :1d ago: -to:now: | reduce by instance_type | sort instance_type | filter instance_type != null"
-valueField 'instance_type';How to: Analyze your AWS Usage Data into JutIngest your data: hit the AWS api every 5 mins (simple script, linked here ), query for instance count (by type and availability zone), and send it to Jut's http data receiverUse these Juttle programs (https://github.com/sgipe/jut-blog) to analyze your data.