Test Fast and Break Stuff

Even if you know your website can handle a load of 30 users a second or that more than 500 users causes a small fire in the data center, the big question is: Will the next release work under the same load? Finding an answer to this question offers peace of mind when getting ready to put up a new version of your software live in the middle of the Christmas season.

When I first started down this path, it did not take much to imagine that I might not have enough power on my laptop or that the test server could run out of connections – but none of that happened. What I did discover was configuration errors in my Apache setup in the test environment.

Testing can mean a lot of things to a lot of people, but for me, it comes down to testing Internet applications and services when under high load, which usually means testing REST APIs or connecting to and stepping through various scenarios on a website. Some fairly pricey solutions from software vendors can assist in this process (e.g., LoadRunner), but the good news is that the open source world has come up with some pretty compelling solutions of their own.

I first learned about JMeter [1] while looking for solutions for my LoadRunner problems. From the Internet, I found that JMeter users also had some testing issues, but in general, they were all quite glad to get away from LoadRunner. Inspired to take a closer look, I took JMeter out for a test spin.

Understanding JMeter Scripts

After opening JMeter, you start with an empty testing project. To support your tests, you need to add a few building blocks. The first step is to add a thread group by right clicking on Test Plan in the left pane and selecting Threads (Users) | Thread Group (Figure 1). A thread group controls how many threads (users) are used when running the test and is the number of parallel tests that will be run at any given time. The Ramp-up period defines how many seconds it will take before all threads have started.

Figure 1: Setting up a thread group.

The thread group also controls how long the test will run by the value in Loop Count or with the thread lifetime fields. When the Infinite checkbox is not checked, the test will perform the steps a Loop Count number of times. You can also run the tests a specified period of time by checking the Infinite and Specify Thread lifetime checkboxes and entering the runtime in seconds in the Duration field.

The thread group defaults are sufficient for getting started. Because it is possible to have multiple groups, you can give them a meaningful name other than Thread Group , as originally assigned.

A thread group is simply a collection of one or more steps that are run as a test. A step can be as simple as requesting a page from a website, or it can be more complex, such as performing a login with a REST API. To do either of these operations, you need to add an HTTP request: Simply right-click on your new thread group and choose Add | Sampler | HTTP Request .

The HTTP request is typically used for performing GET and POST requests; however, you can use any of the common HTTP messages here (e.g., DELETE, PUT, PATCH). The HTTP request contains not only the request, but the protocol, server, port, and relative path (if one exists), as well.

If the goal of the test is fetching a web page, then simply creating a group with a single request would almost be a complete test. The final step of any good test is to verify the return code to ensure that the test step and the server both agree on the outcome.

This verification is done by adding an assertion step that tests for a specific return code (Figure 2). Simply right-click on  HTTP Request in the left pane and choose Add | Assertions | Response Assertion . Most often you are interested in whether the server returned success code 200 , but under certain circumstances, you also want to verify that failures do occur.

Figure 2: Setting up an assertion.

Many different options are available in the assertion step. For example, you can test on the basis of headers, response codes, response text, or even on the document delivered. Conveniently, you can save the results of a JMeter variable from one step and test that value in a subsequent assertion.

During development, you can see how each step in the test is performing (e.g., whether it succeeded or how long it took) by adding a listener to the thread group. Similar to the previous steps, right-click on the thread group and choose Add | Listener | View Results Tree .

The most interesting of listeners during development is the Results Tree viewer, which lets you see whether a step succeeded. In the View Results Tree listener window, you can see whether a step was successful by a green checkmark, but you can also examine what request data was sent and what response data came back. You will see only one entry in the listener for each step executed.

If you are impatient to see some of the test results, you can choose Add | Listener | Summary Report (Figure 3), which displays timing information in a table view. Each step is represented by a single line showing how the step performed.

Figure 3: The Summary Report listener displays the run results of each step and a total.

Creating a script for the most part is just doing a series of HTTP requests and following them with an assertion to verify the step was successful. However, different environments commonly exist in a workplace – perhaps all hosted from the same web server. Having the domain-specific information in each and every step of a test will make it time intensive when changing to run the script on other environments. These values can be defined in a central location to make this process easier.

Adding HTTP request defaults (Add | Config Element | HTTP Request Defaults ) to your project allows you to set up the base information for each HTTP request. These values are used when the corresponding field in a normal HTTP request is left blank. These defaults (Figure 4) mean you can leave the protocol, server, and port entries blank for each of your HTTP requests. If any value needs to be changed later, then only a single value would need to be modified.

Figure 4: HTTP request defaults.

HTTP defaults other than Request Defaults can be defined, including:

• HTTP Header Manager
• HTTP Cookie Manager
• HTTP Cache Manager

These are just a few of the possible default configuration setups that can be added, and it is possible to define variables and use them in any of the steps of your test.

Running a Script

The script should be run from the command prompt without a GUI to ensure the values being collected are not influenced by any unnecessary factors. Running the script from the command line requires a few parameters (Table 1) for running and saving the collected information. First, though, I want to discuss another important JMeter feature.

JMeter properties can be defined and used in test scripts that break the link between some values that might otherwise be hard-coded in the test script, such as runtime, username and password, domain names, or even some of your API parameters.

The use of JMeter properties does require a few small changes to the script setup. The example in Figure5 shows that a property named host will be passed in from the command line to be evaluated with this expression. With the ability to define properties within the script, you can now very easily run the same test script for development, quality assurance, or the pre-production environment quite easily. Only the arguments passed in need to be changed when running the script:

jmeter -n -t TestPlan3.jmx -j TestPlan3.log -l TestPlan3.jtl -e -o /tmp/jmeter-report -Jhost=blazedemo.com

Figure 5: Example of using a JMeter property.

Putting this command into a shell script allows all kinds of functionality. You can create a separate dated directory for each run by running either multiple scripts in parallel to generate even more load or a number of different scenarios one after another overnight.

When running JMeter from the command line, the test run generates an HTML report that can be viewed with a web browser.

Full-Fledged Example

One of the things in relatively short supply on the Internet is websites that test REST API programs. However, BlazeMeter does have a test site [2] that simulates a travel agency. Much like any travel site, it has drop-down boxes containing various origin and destination cities; when chosen, the website displays the various flights and costs for those cities (Figure 6).

Figure 6: Simulated flight results on the BlazeMeter website.

Pressing a specific button for a flight confirms what you have selected, and you are then prompted to enter your personal address and payment details. In the final step, you press the Purchase flight button, which will confirm your payment details and thank you for your purchase, which is displayed as the final step.

Translating these steps from web page clicks to a test requires a small bit of reverse engineering. The good news is that your favorite web browser probably already has a developer mode that lets you examine the elements of the web page, as well as examine the network activity. The Google Chrome browser allows you to switch into this mode by pressing the F12 key, as described in the Google Chrome DevTools guides [3].

To begin, create a new project, add a thread group, and add HTTP request defaults. Next, do a retrieve from the website by configuring the HTTP Request to use a GET message for the / path. The protocol, server, and port will be defaulted from the HTTP request defaults. The HTTP Request will do a POST, but the path will be to the reserve script /reserve.php. To simulate that you have pressed one of the flight buttons, you need to pass in values for the flight (Figure 7).

Figure 7: POST request to perform the flight purchase.

The final step of the airline simulation would require you to pass in all the information for the passenger. Looking at the website, this is essentially two different sets of information. The first set defines who will be purchasing the ticket and where they live, and the second set holds the credit card details. This information is sent as an HTTP POST to the website (Figure 8).

Figure 8: POST request to confirm the flight purchase.

These steps might feel a bit disjointed, in that there are almost no ties between the different steps, but this is only a simple web page developed to allow people to create a small JMeter test script and not a real travel site. If this had been a real web page, you would probably see quite a few logical links between pages and be required to log in to the website. Each page retrieved most likely would have held a token that would need to be parsed out and passed into the next step.

However, it is really neat that BlazeMeter makes this test website available, and I don't want to abuse their generosity. I did add pauses to the test script, so running it does not end up being a distributed denial-of-service (DDoS) attack on their test site.