Amazon Web Services

In November AWS announced Reserved Instance Purchase Recommendations in the Cost Explorer.

Great! I thought, finally some free native support for what a lot of cost-analysis companies offer at a price.

And yet when I look at the recommendations, they seem completely wrong to me. Each recommendation provides a link to see the usage that it was based on – but the main issue there is that Cost Explorer only knows your total EC2 usage hours per day.

Why is this important? Because RI applied utilisation only span usage within an hour.

Here’s an example:

I have an autoscaling web app that runs c5.large instances with a normalisation factor of 4. It idles at 2 instances outside of working hours, where it autoscales up to 10 for 8 hours a day.

Over the day my normalised units are:

  • 2 instances x 4 units x 24 hours = 192
  • 8 instances x 4 units x 8 hours = 256
  • Total units: 448.

That’s consistent every day. AWS sees 448 units per 24 hours = 13.3 normalised units per hour. At 4 per instance, that’s a recommendation of 3 c5.large Reserved Instances.

So I buy 3 instances. Since I have at least 2 all day, that’s 100% utilisation for those. But the remaining one is only being used for 33% of the day, in the 8 hour period when I scale above 2.

To recap:

  • I’ve bought 3 Reserved c5.large Instances. No upfront, N.Virginia, at a price of $39.42 each per month which I’m paying whether I use them or not.
  • If I buy the third instance on demand, for 8 hours a day, it costs me only $20.68 per month. But I’ve bought a Reserved Instance for it as recommended, and it’s now costing me $39.42 instead.

So that’s the problem – the recommendations are completely useless for anything but baseline steady-state load. If you scale during the day, particularly if you have very short windows of high peak usage – it’ll equate to a high number of normalised unit/hours that the recommendation is based on, which completely fail to map to the application of the reserved instance billing as this spans only across an hour.

This is a significant loss with this small example, and factored out over a large estate the recommendations and indicated savings are irresponsibly misleading. To work properly the tool needs to see a moving hour-window of concurrent instance usage across the day to work out if a lack of full utilisation would result in an overall saving. Currently, for an average auto-scaled application, I can’t see that it would – but in it’s current state it’s impossible to calculate with any accuracy.


TL;DR: Recommendations are calculated over an aggregated day, Reserved Instance billing is only applied over an hour, so they’re worse than useless.


Recently I was frustrated in a Jenkins build when I was running Docker-in-Docker to build and push a container to AWS Elastic Container Registry (ECR).

The error on push was a familiar `no basic auth credentials` which means some issue with the credentials stored in ~/.docker/config.cfg (or perhaps ~/.dockercfg in earlier versions).

In this case I initially couldn’t understand the error, as the Jenkins declarative pipeline was using a docker.withRegistry function for the registry login, and this was being successfully written to, so what was going on?

Eventually it occurred to me, although it’s not obvious at first – as we’re running docker-in-docker, you might assume that the credentials are looked for relative to where the Docker daemon is running (i.e. on the host), but actually it’s being looked for relative to where the client is calling the daemon from. In this case – within the container. The docker.withRegistry that I was doing with Jenkins was creating credentials on the host – not within the container where the client itself was running.

There were two possible solutions here – one is to ensure you run the docker login command within the client context of the docker-in-docker container, or to mount the .docker directory on the host into the container using something like `-v /root/.docker:/root/.docker` depending on what user you’re running your containers as.