Comparing startup times of Linux distros in the cloud

Cloud Startup

Fastest and Slowest

The slowest numbers were posted on RHEL 8 with nearly 70 seconds, to the consternation of some on the RHEL team. A network misconfiguration in the RHEL AMI used for testing caused a failed DNS lookup to hold up the cloud-init [9] configuration of userspace. The issue has already been remedied as I write and should no longer be there by the time you read this column [10] [11], as the RHEL AMI is updated. On the winning side, Intel's speedy Clear Linux spins up in under three seconds most of the time, although some unexplained outliers in the 10-second range occur with fast-starting distributions (Table 2). The service startup time is essentially zero, and not even the ever popular Alpine Linux can keep up, on account of its longer service initialization times. Debian and AWS Linux 2 have times in line with Ubuntu, so you can pretty much expect to have your newfangled T3.micro instance in about 20 seconds.

Table 2

RHEL 8, Alpine, and Clear Linux

Distribution	AMI Tested	OS Boot (s)	Service Start (s)	Total (s)
Amazon Linux 2	ami-087c17d1fe0178315	10.480	2.307	12.787
Debian 10	ami-07d02ee1eeb0c996c	7.438	4.202	11.640
RHEL 8	ami-0b0af3577fe5e3532	15.497	53.561	69.058
Clear Linux 35000	ami-078bad975fd5aa9f3	2.811	0.025	2.836
Alpine Linux 3.14.2	ami-026896c9df188bad2	2.562	8.213	10.775

If you can spend a few minutes re-building the code to change some configuration, you now have a way to measure your AMI's startup times consistently that can be integrated into your continuous integration/continuous deployment (CI/CD) pipeline. As RHEL's example shows, it is a good idea.

Infos

ping(1) man page: https://manpages.ubuntu.com/manpages/focal/en/man1/ping.1.html
AWS Management Console: https://console.aws.amazon.com/
AWS CLI: https://aws.amazon.com/cli/
"From bsdtar to tarsnap: Building an online backup service" by Colin Percival; September 28, 2013: https://www.tarsnap.com/download/EuroBSDCon13.pdf
ec2-boot-bench: https://github.com/cperciva/ec2-boot-bench
"EC2 boot time benchmarking" by Colin Percival; August 12, 2021: https://www.daemonology.net/blog/2021-08-12-EC2-boot-time-benchmarking.html
systemd: https://www.freedesktop.org/wiki/Software/systemd/
"New – Hybernate your EC2 instances" by Jeff Barr, AWS News Blog ; November 28, 2018: https://aws.amazon.com/blogs/aws/new-hibernate-your-ec2-instances/
cloud-init: https://cloudinit.readthedocs.io/en/latest/
Bugzilla issue 1862930: https://bugzilla.redhat.com/show_bug.cgi?id=1862930
Bugzilla issue 1994804: https://bugzilla.redhat.com/show_bug.cgi?id=1994804

The Author

Federico Lucifredi (@0xf2) is the Product Management Director for Ceph Storage at Red Hat and was formerly the Ubuntu Server Product Manager at Canonical and the Linux "Systems Management Czar" at SUSE. You can read more from him in the new O'Reilly title AWS System Administration .

« Previous 1 2