A Look Inside

Working with containers has become a standard task for administrators, but in addition to plain vanilla container operation, it is also important to take care of security – a task that is sometimes neglected when faced with relatively new container technology. Aqua Security offers the open source Trivy [1] tool, which scans filesystems, Git repositories, and Kubernetes clusters and resources, as well as ensuring container image security. Additionally, the software can find operating system packages and software dependencies (the software bill of materials, SBOM), known security vulnerabilities (CVEs), infrastructure-as-code (IaC) misconfigurations, and sensitive information and passwords.

Installation

Trivy can be installed on all popular Linux distributions and macOS. Alternatively, you can run Trivy as a container. Detailed installation instructions can be found online [1]. Type the commands in Listing 1 to set up the scanner on Debian/Ubuntu.

sudo apt-get install wget apt-transport-https gnupg lsb-release
wget -qO -https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -
echo deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main | sudo tee -a /etc/apt/sources.list.d/trivy.list
sudo apt-get update
sudo apt-get install trivy

Security Scanning

Once the installation is complete, you can start scanning, which I demonstrate with an example of the well-known NGINX image. First, download the image then start the scan:

sudo docker pull nginx
trivy image nginx

The output is fairly lengthy; Figure 1 shows only an excerpt. The output shows the library, its version number, and the CVE number with a description. For an exact description of the vulnerability, you can use either the National Institute of Standards and Technology (NIST) National Vulnerability [2] or the Aqua Vulnerability [3] database.

Figure 1: An image scan with Trivy generates a long list of vulnerabilities, shown in abbreviated form here.

In particular, when you examine third-party containers, you will find that most container images have a significant number of vulnerabilities. For this reason, you will want to focus on the critical vulnerabilities in most cases by defining limits for the scanning process:

trivy image --severity CRITICAL nginx

In the output, you can see two critical vulnerabilities in the current NGINX. For in-house development, it makes sense to use the smallest possible container image as the base image. Small images come with fewer system components and libraries, which means the number of vulnerabilities is very likely to be lower. Another advantage of small images is that they can be built faster, reducing IT costs.

Today, experts advise either a distroless image or a lightweight image such as Alpine. The concept of a distroless image was largely coined by Google and describes images that contain only the application itself along with the runtime dependencies (i.e., only the components that are needed for the purpose of the application). Popular lightweight images include Debian, Alpine, and Ubuntu, which has slimmed down considerably if you look at the size in the latest version.

If you compare these images with the Trivy command-line interface (CLI), the current official Debian image tallies 81 vulnerabilities (61 low, 16 high, and four critical). Ubuntu does very well with only 17 (15 low and two critical), with nothing at all to complain about in the current Alpine image at the time of writing this article.

Despite the low number of vulnerabilities in these images, you cannot assume that an image verified as secure can be considered free of vulnerabilities forever. Instead, you need to implement a process to keep all images up to date and replace them as needed. In many cases, however, a patch is not immediately available for every CVE. Typical threat mitigation approaches include running some code in a sandbox or in a non-privileged context; you need to look into this on a case-by-case basis. Help is available for CVEs in the Aqua Vulnerability Database, showing you how to deal with each CVE.

Finding Misconfigurations

Although Trivy has become known as a tool for scanning container images, it can now do far more. For example, the software can also check the configuration of Docker build files and Kubernetes, Helm, Terraform, and AWS CloudFormation artifacts. Ansible support is also planned for the future.

I look into scans of this type with Terraform as an example, a system with a risk of deploying infrastructure components potentially fraught with vulnerabilities. Listing 2 shows some code that calls a Terraform module published in GitHub. You can launch the code check with the command:

trivy config <directory>

01 locals {
02       cluster_type = "simple-zonal"
03 }
04 provider "google" {
05       ...
06 }
07 data "google_client_config" "default" {}
08 provider "kubernetes" {
09        ...
10 }
11 /*******************************************
12 * Import terraform outputs from VPC
13 ******************************************/
14 data "terraform_remote_state" "vpc" {
15       ...
16 }
17 /*******************************************
18 * GKE Cluster
19 ******************************************/
20 module "gke" {
21       source               =  "terraform-google-modules/kubernetes-engine/google"
22       project_id           =  var.project_id
23       name                 = "${local.cluster_type}-cluster${var.cluster_name_suffix}"
24       regional             =  false
25       region               = var.region
26       zones                = var.zones
27       network              = data.terraform_remote_state.vpc.outputs.network_name
28       subnetwork           = data.terraform_remote_state.vpc.outputs.subnets_names[0]
29       ip_range_pods        = data.terraform_remote_state.vpc.outputs.pod_cidr_name
30       ip_range_services    = data.terraform_remote_state.vpc.outputs.service_cidr_name
31       service_account      = var.compute_engine_service_account
32       ...
33 }

Aqua Security provides a database similar to the container scanning database that provides detailed information for security-related configuration worries. Among other things, you will find information on how to adapt the IaC code so that the infrastructure rolled out with it is robust in terms of security.

Comparable scans are also possible for the subcommands image, repo, and fs:

trivy image --security-checks config <image_name>
trivy fs --security-checks config </directory_path>
trivy repo --security-checks config <repo_name>

The built-in policies provide a good basis and are all implemented in Rego query language. If you need more, you can supplement with your own rulesets. The Trivy documentation includes some examples.

The currently still experimental support for direct scanning of cloud resources is interesting despite its status. At this point, it only works with AWS, but it shouldn't be too long until the other major hyperscalers are supported. To use the scanner, install the AWS CLI and configure access to your AWS account accordingly; then, start the scanning process with the trivy aws command.