Swarming

Availability is one of the three most prominent protection goals of IT security. In contrast to encryption and integrity, both of which can be ensured with cryptographic processes, availability is often ensured by access control and redundancy. However, access control in particular is rather difficult for resources that are basically publicly accessible.

To be prepared for emergencies, it is important to determine the performance of your systems to give you reliable figures on how many requests your services can handle well and at what point an overload situation will occur. In this article, I introduce you to Locust [1], a powerful tool that can simulate an incredibly large number of users.

The Right Time

Of course, you should not run a stress test against a production system, if possible. Therefore, a good idea is to run a shadow system with the same resources, connections, and software versions as the production system. If you do not yet have a shadow system, you can also test recovery from current backups when you create them. At best, this option will give you a running system in a short time and evidence that your backups are comprehensive and quickly available.

If you need to run a stress test against a production system, your best choice is to time the test when the number of regular requests is low and preferably no customers or employees will be affected. To prepare your test as well as possible so that it can be kept short, you need to redirect regular users to another system with a maintenance notice for as long as possible. The easiest way to do this is with the Domain Name System (DNS).

Set the Time to Live (TTL) of your DNS records to a very small value (e.g., in the single-digit minute range) for one to two days and then store an IP address of another server for the duration of the test to provide the required information for the users of your service. After the load test, check that the service is available again without errors and reset the TTL value.

Preparing Locust

Locust is written in Python and can be easily adapted to your application with a little Python code. You can install Locust with the command:

pip3 install locust

For the first test run, create the locustfile.py file:

from locust import HttpUser, task
class ItaTestUser(HttpUser):
    def simple_task(self):
       self.client.get("/")

Import the HttpUser modules required for a user accessing your service over HTTP and the task, which specifies your defined functions as the task to be performed during the load test. Create your own ItaTestUser class that inherits from HttpUser, and use the HTTP client's get function in the simple_task function to query the index page of a service.

Here you can add other arbitrary subpages of your service. For different user scenarios you will want to create different functions, all with the task decorator. These functions will be randomly selected and executed as part of the load test. Typing locustfile.py launches the first test.

Launch the Locust server with the locust command and connect to the specified address (usually http://localhost:8089/ ) in your browser. In the dialog shown in Figure 1, enter the parameters for your first test. In particular, you will want to adapt the URL of the service to your own service.

Figure 1: Three entries and off you go – above all, it is important to use the correct URL.

Now click on Start swarming and dispatch your plague of locusts to the service to be tested. Locust's web view provides statistics on the calls made and, under Charts , gives you a continuous display of the number of active users, concurrent requests, and errors encountered. While your tests are running, you can adjust the number of concurrent users and new users created per second to fine-tune the tests to the load limits of your systems.

Locust maintains an HTTP client for each user started. In addition to get for requesting web pages, the post function,

def on_start(self):
  self.client.post("/login.php", json={"user": "username", "password": "userpassword"})

can be used to register or log in (e.g., before further requests are made in the valid sessions of logged-in users). If this login is in the on_start function of your ItaTest-User class, it will be executed directly when the user is created; from then on, all of this user's requests will be executed with the session created. In contrast, the on_stop function can be used for logging out or for the user checkout process of a store system. Locust takes care of managing the session cookies on its own.

To ensure that different users log in during your tests, you can also use Python's random module to store an array of login combinations and randomly select entries.

If some functions in your applications are called more frequently than others (e.g., in a store system, viewing various products compared with checking out), you can give the @task decorator a weighting factor. The higher this weighting factor, the more often the function is called. The @tag decorator lets you assign tags to individual functions, which you can select or deselect when Locust is launched, allowing you to create general Locust configurations for all your services and select them accordingly, depending on the service or host being tested.

Distributed Requests

Locust allows work to be split between a controller and several worker instances. Several worker instances are assigned to a controller. If a certain number of workers is available, the controller starts the prepared tasks, which allows the integration of different systems for load generation that communicate with each other on the network. This capability makes it very easy to stress your load balancers through different data centers and, therefore, by way of different upstream providers.