The activity of penetration testing has been around for decades, but the past five years or so have seen radical changes in networking, beginning with the dissolved perimeter. Remember the good days of threats "inside the firewall" and "outside the firewall?" Those days are long gone, because of the advent of the cloud and the use of mobile devices.

A second problem is radically changed endpoints. From mobile phones and new peripherals, to the cloud, to the Internet of Things (IoT), the current environment is a long way from the typical end-user PC. Today's endpoint can be the cloud, a wearable device, or a traditional PC, notebook, mobile phone, or tablet.

A third threat is social engineering. The con man has been around for thousands of years in one form or another, but the majority of successful attacks start with clever pretexting. Good pen testers know this and use it to their advantage.

Finally, you must consider social media. Hackers have focused on end users. Why? One reason is because so many are revealing nearly everything about themselves to the world.

Considering all of the changes mentioned above, it is all the more important to consider automating certain steps of the penetration test, and it is especially important to obtain accurate information about the organization you are testing. Still, the step of gathering information for a penetration test can be quite time-consuming, so given all of the changes you need to address, what can you do to speed up that process? Also, what is the relationship between the first part of the penetration test, and all of the other parts?

To begin, I talk about the hacker lifecycle and its relevance to the penetration test.

Typical Hacker Lifecycle

In 2011, researchers from Lockheed Martin created their version of the hacker lifecycle called the Cyber Kill Chain. Figure 1 shows each of the steps.

Figure 1: Lockheed Martin Cyber Kill Chain model.

Each "kill chain" step describes typical activities an attacker takes when infiltrating a network. The idea behind calling it a kill chain is that it helps defenders think about how to thwart each step. You can learn more about the Lockheed Martin model on their website [1].

Reconnaissance involves gathering information, including vulnerabilities. Weaponization is where the attacker obtains or creates malware. The Delivery and Exploitation steps include where the attacker is able to place and activate malware and use it to exploit network resources. The Command & Control stage involves the ability to usurp control over networked systems, including endpoints, servers, and connectivity devices. In the Actions on Objectives stage, the attacker begins stealing and modifying data and taking additional steps, such as covering up evidence of intrusion.

Since its introduction, the Lockheed Martin Cyber Kill Chain model has received quite a few critiques. Some focus on how the model is based on the old perimeter-focused model, which has largely dissolved. The model was clearly created before the days of the IoT, and even before the mobile/bring your own device (BYOD) revolution. As I noted above, the environment has changed greatly.

Customized Lifecycles and Frameworks

When it comes to penetration testing, I maintain it is important to create a lifecycle that makes sense for the environment you are investigating. You might want to emphasize certain steps or break out a certain step into multiple steps. In Figure 2, for example, I present a scheme I created for a few networks I have audited over the years.

Figure 2: Custom hacker lifecycle for a network.

In the above model, the Discovery stage involves reconnaissance. In Discovery, attackers do more than simply identify network hosts; they also identify and begin profiling individual users susceptible to social engineering and start identifying the likely steps for exploitation. As you will see later in this article, I spend some time discussing how to automate the process of gathering information during this stage.

The Penetration/Foothold step involves substeps, such as introducing malware, hijacking a session, or starting a reverse shell session. It is basically the initial compromise in which one gains unauthorized access. From here, the attacker generally escalates privileges, then ends the process, and gets out. This happens with the typical "smash and grab" hacker. They just grab something (e.g., an account and some information) and then leave.

Many hackers, however, tend to use this initial compromise as a way to learn more about the network and then lurk. These attackers are often part of an advanced persistent threat (APT), in which attackers lurk in systems for a long period of time – months or even years – before moving on to penetrate additional systems. They also engage in modifying networked systems (e.g., endpoints, routers, firewalls) to facilitate remaining in the network over a long period of time. This is the activity I call Persistence.

During either the Penetration or Persistence phase, attackers will conduct various activities, including:

• Data egress: stealing data for later use. Data can include user account information, databases, or sensitive intellectual property. It can also include personally identifiable information (PII).
• Data manipulation: modifying sensitive data. Attackers today often would rather alter data to their advantage over a long period of time than simply steal it. After all, why simply grab a quick data snapshot when you can manipulate information in your favor?
• Malware installation. Ransomware is quite common, but it can also include keylogging software, viruses, and other payloads.
• Sandbox escape. Lately, I have seen networks use mandatory access control (MAC) and "sandboxed" environments that do not allow software to be launched as in a traditional Windows, Linux, or Mac environment. You have to choose from menus or from choices on a web form; yet, in these cases, attackers have simply gone after code flaws in the web browsers and stored session cookies to escape these environments.

Usually, attackers will try to cover their tracks throughout the entire hacker lifecycle. Covering tracks involves modifying or removing logs, as well as behaving in ways that fall below usual intrusion detection thresholds. Pen testers, because they are working under terms of a contract and a Statement of Work (SOW), do more than cover tracks. They also clean up the systems and generate reports.

To begin, I will look at the Discovery/Reconnaissance stage and see what kinds of automated software the penetration tester can use to conduct a "black box" audit.

Discovery Stage: Nmap and Maltego

Open source intelligence tools (OSINTs) are extremely useful during the discovery stage. Pen testers were once relegated to using whois, nslookup, and good old nmap, along with plenty of other useful tools. However, you have many more automated choices available now.

One of the first additions is the Shodan API. Shodan [2] is a search engine service that provides granular information about Internet-aware devices. A few years ago, the maintainers of Shodan created an API that allows you and your security software to "plug in" to their database. Various tools now have interfaces, including Nmap.

Figure 3 shows a typical Nmap scan using the OSScan option (-0) that you've seen a thousand times. Notice with this particular scan that Nmap is having a hard time figuring out the exact nature of the operating system, although it outputs some details, including Oracle Virtualbox , but no more. Enter Shodan.

Figure 3: Typical Nmap scan.