Cloud Startup

Launching instances on Amazon Elastic Compute Cloud (EC2) can be a lengthy affair for those used to instant gratification, taking entire minutes to complete. Historically, Windows instances have brought up the rear by taking considerably longer to initialize than Linux-based instances because of sysprep and the reboot that follows. The open source world is somewhat more streamlined, but there is still considerable variance between Linux distributions, with one to two minutes being a reasonable expectation of first availability.

One Ping Only, Please

Trying to connect multiple times as an instance boots is inelegant. Fortunately, you can use shell-fu to script your way out of this. The BSD version of ping [1], notably on macOS, includes a convenient "one ping only" option (-o) that I would like to think honors Sean Connery's famous quote in Hunt for Red October . The option terminates ping once the first reply is received. Like Connery's character, Captain Marko Ramius, you can use this option to ask for "one ping only, please" (Listing 1).

$ ping -o 52.90.56.122; sleep 2; ssh ubuntu@52.90.56.122
PING 52.90.56.122 (52.90.56.122): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
64 bytes from 52.90.56.122: icmp_seq=3 ttl=48
time=40.492 ms
[ output truncated ]
Welcome to Ubuntu 20.04.1 LTS
(GNU/Linux 5.4.0-56-aws x86_64)

Whether you launch the instance with the Amazon Web Services (AWS) Console [2] or the AWS command-line interface (CLI) [3], you can easily find the moment it becomes accessible; however, that is only part of the picture. An instance is only useful as the service it exists to provide can be reached by users. Completing service bootstrap is distinct from the point in time when the kernel network stack becomes active. The SSH daemon is present in nearly all official distribution images, and it provides a convenient testing standard for service availability.

Perhaps less steeped in movie lore, but nonetheless equally effective, is this GNU-compatible one-liner, waiting in a loop for the SSH service to start up (Listing 2).

$ until ssh ubuntu@52.90.56.122; do sleep 1; done
ssh: connect to host 52.90.56.122 port 22: Connection refused
ssh: connect to host 52.90.56.122 port 22: Connection refused
[ output truncated ]
Welcome to Ubuntu 20.04.1 LTS
(GNU/Linux 5.4.0-56-aws x86_64)

Booting Benchmarks

These two tests provide good insight into the initialization speed of stock operating system images, with the notable exception of Microsoft Windows, where SSH is absent (see the "Hibernate Support" box). FreeBSD Security Officer Emeritus and author of Tarsnap [4] Colin Percival has done all the hard work and released a new tool to time these events. The ec2-boot-bench tool [5] takes four measurements: (1) the duration of the RunInstances API call, (2) the time to transition from a pending to a running state, (3) how much longer the network stack took to initialize (RST TCP response), and (4) actual service availability (SYN+ACK TCP response). The ec2-boot-bench tool uses the SSH service's availability for the fourth metric, just as the shell script in Listing 2, but you can easily change the port with a recompile and use Colin's code to benchmark the specific service of your own Amazon Machine Image (AMI).

The first two metrics do not vary with distribution, and Colin himself has published [6] data indicating an average of 1.5 and 6.9 seconds, respectively; however, these numbers may vary with AWS region, availability zone, or instance type and are part of the delay experienced by the user. My own measurements in the us-east-1 region are consistent with Colin's findings and average around 1.7 and 8.5 seconds, respectively (Figure 1). I used T3.micro instances for my tests instead of C5.xlarge, accounting for some of the variance between our results.

Figure 1: A few runs with ec2-boot-bench. A shell loop is recommended.

The tool itself is is still a rough instrument, with no distribution packages, man pages, or GitHub docs – and no way to check API errors. (I will be submitting a couple of patches for these before this article goes to press.) However, it does its job remarkably well already. Instances are benchmark-terminated, but if you Ctrl+C the tool, you might have stragglers left running: Remember to clean up after tests or your AWS bill will suffer.

Ubuntu's Lot

Ubuntu on 64-bit instances appears to have made consistent progress over the years, cutting startup time by nearly a third, from 14.6 to 9.9 seconds (32%), as shown in Table 1 for 10 sample averages after a warm-up run. The latest Ubuntu Server LTS release, Focal Fossa (20.04), can be expected to be ready for use in 15 seconds once the EC2 virtual machine has been initialized, compared with 18 seconds with Trusty Tahr (14.04). The more than doubling of the SSH startup time between Trusty and Xenial may show how the transition to systemd [7] affected Ubuntu's overall optimization, or it may just be an artifact of startup ordering because the resulting total is not significantly affected.