Meter’s running - How to avoid spending all your money—

Ever been in a cab, stuck in traffic watching the meter run. You watch each 50-cent increment nudge the fair up dollar by dollar, wondering if it would be quicker to walk.

Cloud hosting is charged the same way, but it’s very easy to take your eye off the meter, fire up dozens of services, store gigabytes of files, and end up spending a fortune.

I’ve left instances sitting around for days and even weeks. Between myself and our lead architect, we somehow forgot about an RDS instance for a few days, racking up charges for a machine that wasn’t even being used.

It doesn’t matter what platform you use: If it’s pay as you go, you’ll want to monitor it to prevent your $1,000-a-month bill turning into $10,000 a month.

In the tradition of programmers being lazy but resourceful, I’ve thrown together a little script using the Ruby cloud computing library fog that shows you which services are running and roughly how much they’re charging in the current hour.

The script simply connects to AWS, finds each of the main chargeable services (I’ve only done EC2, RDS, and S3 so far … feel free to contribute), and uses a table of costs to find the upper limit of costs. Because AWS tiers its price (the second million gigabytes of storage cost less than the first), the estimated costs will probably be higher than you are charged at the end of the month, but it’s better to be safe ….

Write the Script

The first few lines list the instances I use and their costs. Because I’m only working in a couple of regions, I’ve left the costs approximate.

instance_costs = {
   't1.micro'   => 0.02,
   'm1.small'   => 0.095,
   'c1.medium'  => 0.19,
   'm2.2xlarge' => 1.14,
   'm2.4xlarge' => 2.28
}

Then I list the regions. This could have been dynamic, but I don’t want to check in the [asia??] regions, so I’ve listed them.

regions = [
   'eu-west-1',
   'us-west-1'
]

And finally, I need the account details. Because I run multiple accounts for dev, test, and production for multiple clients the account details are in a hash.

accounts = {
   'myaccount' => {
      'key_id' => 'YOUR-KEY-HERE',
      'access_key' => 'YOUR-ACCESS-KEY-HERE'
   }
}

The script uses 'fog' [URL], which means you could adapt it to work with GoGrid, Rackspace, or whichever cloud provider you use.

The rest of the script simply loops through the accounts, then through the regions, and shows you how much you’re spending in the current hour and what that might be at the end of the month. It’s easy to forget that $1 an hour is $744 dollars a month.

Run the Script

$ ruby cost-checker.rb
For region 'eu-west-1'
   i-abc123ab....................: $  0.19
   i-def456de....................: $  0.095
   i-hij789hi....................: $  0.095
Total for account                : $  0.48
Projected monthly for account    : $353.40

Pretty useful. If an instance is stopped, it is shown because you won’t be charged for it.

   i-aaa111aa....................: $--.-- [stopped]

EC2 isn’t by any means the only way to rack up charges. You need to consider storage (S3, EBS snapshots, and CloudFront), bandwidth, and any other services you’re using. This script should help you keep an eye on the meter.

The script is available on github at https://github.com/danfrost/Cloud-costs. Just clone it and run … then realize how much you should be saving.

Here's a quick tip for managing AWS keys a little differently—

All the tutorials you'll find refer to downloading the generated private key from the AWS console. This is fine for the first 10 times you create instances or cloud setups, but the time will come when you want to use the same key for lots of instances or you want to use your own keys all the time.

AWS has an "import keypair" method than can be reached easily using the AWS CLI tools. This command takes a public key and uploads it to your AWS account so that instances can be launched with it.

If you haven't already installed the AWS CLI tools, grab them from the Amazon EC2 API Tools website and put them in /usr/local/ec2-api-tools/. Add the environment variables:

export EC2_HOME=/path/to/ec2-tools/
export PATH=$PATH:$EC2_HOME/bin
export EC2_PRIVATE_KEY=`ls -C $EC2_HOME/pk-*.pem`
export EC2_CERT=`ls -C $EC2_HOME/cert-*.pem`

With this installed, you have access to loads of the API that isn't always exposed via GUIs – a nice chance to get under the hood.

Start by generating an ssh key-pair in the usual way:

ssh-keygen

And follow the instructions. This generates two files: the public key, id_rsa, and the private key, id_rsa.pub.

It's the public key that you need to upload, using the ec2-import-keypair command:

ec2-import-keypair dan-key --public-key-file ~/.ssh/id_rsa.pub

Using the ec2-describe-keypairs command, you can see the newly uploaded keypair:

ec2-describe-keypairs
  KEYPAIR dan-key
fc:39:b2:60:90:4c:0f:66:fc:b8:a6:54:af:19:0c:ef
  KEYPAIR my-key
a6:54:af:19:0c:ef:fc:39:0f:66:fc:b8:b2:60:90:4c
  KEYPAIR your-key
fc:39:b2:60:90:4c:af:19:0c:ef:0f:66:fc:b8:a6:54

Instances can now be launched using the key by firing off the ec2-run-instances command:

ec2-run-instances ami-abc123ab -k dan-key

Finally, if you want to generate keys on the fly, the CLI tools can help you do this with the ec2-add-keypair command. This works in reverse to the ec2-import-keypair command by generating the key on AWS and passing the private key back to you:

ec2-add-keypair my-new-key
  KEYPAIR dangen1
aa:ce:ec:ae:bb:18:f3:cb:cc:ee:95:c3:fe:86:5b:09:f9:ae:18:ff
  -----BEGIN RSA PRIVATE KEY-----

+PgLSIcy+rHHELXhv6bvZGamd2R2u2DFhBo36w9DEjAhocW2hraXMIIEowIBAAKCAQEAhXo2cUYv

B8/P/BP0ges6i7VJ9Oj1bDHfILtu805syqwN5J6IBcgvesthq4Xpj4zuIVsCctU5SEIkx9texM+b
....

fuy3QFJdl3rM0w/ry1QDRy5WgfZsIpAQZUuCaZgZx2BavviuVcFGrd67RfP6gt2yBk7EhN0gQCN2

X5YHEueK7qLqQYNsX4X9JROKgB0zMdaJvmCqekvoGhQepx5C4TqrDWIOHAwHOw6V6nqZ
----END RSA PRIVATE KEY-----

So, when would you import a key rather than generate one? If you have limited permissions on your local machine or if you want to create instances from a known set of keys, uploading the key makes more sense.

If you have a strong internal security policy and need to keep keys in rotation, uploading the public keys will be better than having any number of keys generated on AWS.

You can find out more about this command at the AWS CLI docs: http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/index.html?ApiReference-cmd-ImportKeyPair.html