Protecting Data

The second major security concern relates to storing data. Especially when processing personal information for members of the European Union (EU), you have to be careful for legal reasons about which of the cloud provider's regions is used to store the data. Relocating the customer database to Japan might turn out to be a less than brilliant idea. Even if the data is stored on servers within the EU, the question of who gets access still needs to be clarified.

Encrypting data in AWS is possible [5]. If you do not have confidence in your abilities, you could equip a Linux VM with a self-encrypted volume (e.g., LUKS [6]) and not store the password on the server. With AWS, this does not work for system disks, but it does at least for data volumes. After starting the VM, you have to send the password. This process can be automated from your own data center. The only possible route of access for the provider is to read the machine RAM; this risk exists where modern hardware enables live encryption, as well.

As a last resort, you can ensure that the computing resources in the cloud only access data managed by the local data center. However, you will need a powerful Internet connection.

Solving a Problem

Assume you have a local Elasticsearch cluster of three nodes: a master node, which also houses Logstash and Kibana, and two data nodes with data on board (Figure 1).

Figure 1: A secure network architecture (here the rough structure) should connect the nodes in the local data center with those on AWS.

You now want to provide this cluster temporarily two more data nodes in the public cloud. You could have several reasons for this; for example, you might want to replace the physical data nodes because of hardware problems, or you might temporarily need higher performance for data analysis. Because it is not typically worthwhile to procure new hardware on a temporary basis, the public cloud is a solution. The logic is shown in Figure 1; the machines started there must become part of the Elastic cluster.

The following explanations assume you have already written Ansible roles for installing the Elasticsearch-Logstash-Kibana (ELK) cluster. You will find a listing for a Playbook on the ADMIN FTP site [7]. Thanks to the structure of these roles, you can add more nodes by appending parameters to the Hosts file, and it includes installing the software on the node.

The roles that Ansible calls are determined by the Hosts file (usually in /etc/ansible/hosts) and the variables set in it for each host. Listing 1 shows the original file.

10.0.2.25 ansible_ssh_user=root logstash=1 kibana=1 masternode=1 grafana=1 do_ela=1
10.0.2.26 ansible_ssh_user=root masternode=0 do_ela=1
10.0.2.44 ansible_ssh_user=root masternode=0 do_ela=1

Host 10.0.2.25 is the master node on which all software runs. The other two hosts are the data nodes of the cluster. The variable do_ela controls whether the Elasticsearch role can perform installations. When expanding the cluster, this ensures that Ansible does not reconfigure the existing nodes – but more about the details later.

Extending the Cluster in AWS

The virtual infrastructure in AWS comprises a VPC with two subnets. One subnet can be reached from the Internet; the other represents the internal area, which also contains the two servers on which data nodes 3 and 4 are to run. In between is a virtual firewall, by Fortinet in this case, that terminates the VPN tunnel and controls access with firewall rules.

This setup requires several configuration steps in AWS: You need to create the VPC with a main network. On this, you then assign all the subnets: one internal (inside) and one accessible from the Internet (outside). Then, you create an Internet gateway in the outside subnet. Through this, the data traffic migrating toward the Internet finds an exit from the cloud. For this purpose, you define a routing table for the outside subnet that specifies this Internet gateway as the standard route (Figure 2).

Figure 2: A firewall separates an external subnet and an internal subnet (right). In detail, the AWS connection looks slightly different.