Listing 3: Another Section of SSLyze Output

 Trust
       Hostname Validation:               OK - Certificate matches www.devsecops.cc
       Android CA Store (8.1.0_r9):       OK - Certificate is trusted
       iOS CA Store (11):                 OK - Certificate is trusted
       Java CA Store (jre-10.0.1):        OK - Certificate is trusted
       macOS CA Store (High Sierra):      OK - Certificate is trusted
       Mozilla CA Store (2018-04-12):     OK - Certificate is trusted
       Windows CA Store (2018-04-26):     OK - Certificate is trusted
       Symantec 2018 Deprecation:         OK - Not a Symantec-issued certificate
       Received Chain:                    www.devsecops.cc --> Let's Encrypt Authority X3
       Verified Chain:                    www.devsecops.cc --> Let's Encrypt Authority X3 --> DST Root CA X3
       Received Chain Contains Anchor:    OK - Anchor certificate not sent
       Received Chain Order:              OK - Order is valid
       Verified Chain contains SHA1:      OK - No SHA1-signed certificate in the verified certificate chain
 
     Extensions
       OCSP Must-Staple:                  NOT SUPPORTED - Extension not found
       Certificate Transparency:          WARNING - Only 2 SCTs included but Google recommends 3 or more
 
     OCSP Stapling
                                          NOT SUPPORTED - Server did not send back an OCSP response
 
 * OpenSSL Heartbleed:
                                          OK - Not vulnerable to Heartbleed
 
 * ROBOT Attack:
                                          OK - Not vulnerable
 
 SCAN COMPLETED IN 9.97 S

The end of Listing 3 shows how long the query took to run – a smidgen under 10 seconds, in this case.

For the curious, according to the SSLyze docs, the Trust Stores Observatory, which “… monitors the content of the major platforms’ root certificate stores” contributes to the SSLyze trust certificates.

A weekly check of various vendors and for the certificates used on their devices means you can be confident you are dealing with current information.

Automation, Automation, errr … Automation

Now that the security bit is working, the next task is to write some suitable code or, in Gauntlt parlance, an attack file.

You should find that if you try to run your SSLyze attack file without having installed the package (or “attack adaptor”) first, you receive useful errors (Listing 4; the output was altered slightly for clarity). The attack file will generate errors and offer instructions about how to install the SSLyze package on your system if you forgot to do so. As you can see, you are not abandoned without assistance, and Gauntlt offers some helpful debugging advice.

Listing 4: Attack File Errors and Instructions

Feature: Run sslyze against a target
 
  Background:                   # sslyze.attack:3
 
    Given "sslyze" is installed # gauntlt-1.0.13/lib/gauntlt/attack_adapters/sslyze.rb:1
          
          sslyze.py not installed or $SSLYZE_PATH not set!
      
          1. Download sslyze from: https://github.com/iSECPartners/sslyze
 
          2. In your .zshrc or .bash_profile (or whatever), set $SSLYZE_PATH
      
             export SSLYZE_PATH=/path/to/sslyze.py
      
          3. Make sure you have python installed:
      
             $ which python

Notice that because I installed SSLyze with pip , I neglected step 2, which relates to setting the SSLYZE_PATH environment variable. Incidentally, you should note that the GitHub URL shown in step 1 now redirects to the new page.

The environment variable error suggests that Gauntlt needs to be pointed in the direction of the sslyze.py python script; however, having looked, that file apparently isn’t present on my system. The ungraceful hacky fix I used was to point the environment variable at the Python script that lives in the /usr/local/bin directory, instead, with the command:

$ export SSLYZE_PATH=/usr/local/bin/sslyze

Have a deeper look yourself to determine whether this fix is ill-advised or not.

Where Did I Put It Again?

The next step is the sslyze.attack shown in Listing 5. Note that I have used the GitHub example but changed the hostname to my website and reduced the scope of the attack to SSLv2 only. I’ve also altered the should not contain line to the word Rejected . In other words, the success of the test depends on whether the word Rejected is present in my website’s test output. I want an SSLv2 connection to my website to be rejected because that protocol has some nasty security issues. So, for clarity, the word Rejected is a Good Thing™ on this occasion.

Listing 5: Fire SSLyze at a Website

Feature: Run sslyze against a target
 
Background:
  Given "sslyze" is installed
  And the following profile:
    | name     | value      |
    | hostname | www.devsecops.cc |
 
Scenario: Ensure no SSLv2 certificates are accepted
  When I launch an "sslyze" attack with:
    """
    sslyze --sslv2 <hostname>:443
    """
  Then the output should not contain:
    """
    Rejected
    """

Figure 1 shows the output from the test in Listing 5. The command to run that test, this time via Gauntlt and not directly using SSLyze, is simply:

$ gauntlt sslyze.attack

Notice that www.devsecops.cc has provided the word Rejected in its output, because the test has passed. If you squint a little, you can see some of the comments (in dark text) that the clever Gauntlt adds as it goes through the motions.

Figure 1: Running an SSLyze test via Gauntlt.

On Or Off?

To prove the attack file is indeed working as expected, I’ll now change what I’m looking for. In Listing 5, I'll change Then the output should contain to Then the output should not contain . By adding the word not, I’m essentially saying that I will accept SSLv2 certificates, and the test output should not contain R eject ed . (Obviously, Accepted would be the reverse of the expected text in the attack file’s case). In Figure 2, you can see what happens by adding the word not .

Figure 2: Three little letters added to the attack file, and the whole test fails.

The detail in Figure 2 might be a little tricky to read, but trust me when I tell you that there’s much more red-colored text than in Figure 1 and that the last three lines of the test output now read:

1 scenario (1 failed)

4 steps (1 failed, 3 passed)

0m0.624s

With those results in hand, I think it’s safe to say that the attack adaptor and attack file are working correctly.