Automation, Automation, errr … Automation

Now that the security bit is working, the next task is to write some suitable code or, in Gauntlt parlance, an attack file.

You should find that if you try to run your SSLyze attack file without having installed the package (or "attack adaptor") first, you receive useful errors (Listing 4; the output was altered slightly for clarity). The attack file will generate errors and offer instructions about how to install the SSLyze package on your system if you forgot to do so. As you can see, you are not abandoned without assistance, and Gauntlt offers some helpful debugging advice.

Feature: Run sslyze against a target
  Background:                   # sslyze.attack:3
    Given "sslyze" is installed # gauntlt-1.0.13/lib/gauntlt/attack_adapters/sslyze.rb:1
      sslyze.py not installed or $SSLYZE_PATH not set!
      1. Download sslyze from: https://github.com/iSECPartners/sslyze
      2. In your .zshrc or .bash_profile (or whatever), set $SSLYZE_PATH
         export SSLYZE_PATH=/path/to/sslyze.py
      3. Make sure you have python installed:
             $ which python

Notice that because I installed SSLyze with pip, I neglected step 2, which relates to setting the SSLYZE_PATH environment variable. Incidentally, you should note that the GitHub URL shown in step 1 now redirects to the new page [8].

The environment variable error suggests that Gauntlt needs to be pointed in the direction of the sslyze.py python script; however, having looked, that file apparently isn't present on my system. The ungraceful hacky fix I used was to point the environment variable at the Python script that lives in the /usr/local/bin directory, instead, with the command:

$ export SSLYZE_PATH=/usr/local/bin/sslyze

Have a deeper look yourself to determine whether this fix is ill-advised or not.

Where Did I Put It Again?

The next step is the sslyze.attack shown in Listing 5. Note that I have used the GitHub example but changed the hostname to my website and reduced the scope of the attack to SSLv2 only. I've also altered the should not contain line to the word Rejected. In other words, the success of the test depends on whether the word Rejected is present in my website's test output. I want an SSLv2 connection to my website to be rejected because that protocol has some nasty security issues. So, for clarity, the word Rejected is a Good Thing (TM) on this occasion.

Feature: Run sslyze against a target
Background:
  Given "sslyze" is installed
  And the following profile:
    | name     | value      |
    | hostname | www.devsecops.cc |
Scenario: Ensure no SSLv2 certificates are accepted
  When I launch an "sslyze" attack with:
    """
    sslyze --sslv2 <hostname>:443
    """
  Then the output should not contain:
    """
    Rejected
    """

Figure 1 shows the output from the test in Listing 5. The command to run that test, this time via Gauntlt and not directly using SSLyze, is simply:

$ gauntlt sslyze.attack

Figure 1: Running an SSLyze test via Gauntlt.

Notice that www.devsecops.cc has provided the word Rejected in its output, because the test has passed. If you squint a little, you can see some of the comments (in dark text) that the clever Gauntlt adds as it goes through the motions.

On or Off?

To prove the attack file is indeed working as expected, I'll now change what I'm looking for. In Listing 5, I'll change Then the output should contain to Then the output should not contain. By adding the word not, I'm essentially saying that I will accept SSLv2 certificates, and the test output should not contain Rejected. (Obviously, Accepted would be the reverse of the expected text in the attack file's case). In Figure 2, you can see what happens by adding the word not.

Figure 2: Three little letters added to the attack file, and the whole test fails.

The detail in Figure 2 might be a little tricky to read, but trust me when I tell you that there's much more red-colored text than in Figure 1 and that the last three lines of the test output now read:

1 scenario (1 failed)
4 steps (1 failed, 3 passed)
0m0.624s

With those results in hand, I think it's safe to say that the attack adaptor and attack file are working correctly.