All Threats at a Glance

Apache Spot has extensive visualization tools that allow further analysis of suspicious activities detected by the ML algorithm. The web-based interface displays a list of suspicious network activities detected by Apache Spot as potential security threats under Suspicious . The overview displays different information for each critical event, such as the time, IP address, rating, type of traffic, port, and protocol.

The starting point of the visualization is the dashboard, which presents a summary of the network infrastructure to be monitored and shows the number of networks, endpoints, and critical events. The dashboard also lists the most critical user, endpoint, and network threads. The assessment of the status is visualized by corresponding status information (e.g., Very High, High, Medium), together with a corresponding color. In the Network menu, Apache Spot displays the network structure in a separate Network View (Figure 3). In addition to a graphical representation of the traffic over a defined time period, a tabular overview lists the busiest systems. A visual map gives you a bird's eye view of the infrastructure.

Figure 3: The web interface displays suspicious events, allows selective restriction of the view, and visualizes the network structures and connections.

Depending on the information requirements and user type, you can switch between simple and expert modes. In the Notebook panel, you can determine the risk configuration for each connection. If you switch from simple to expert mode, you can adjust the criteria by which the data is filtered and discarded. Possible filters include source and destination IPs and ports and a rating. In this way, you can filter out exactly the information that matters to you. For each event, you can call up additional information in the Details View, such as the type of service and the router and interfaces used.

Threat investigation takes place in the Threat Investigation tab and is the final step of the analysis before Apache Spot displays the Storyboard. At this point, security analysts can perform a custom scan for a given threat. The Storyboard provides you with overviews of incident progression, impact analysis, and geographic location, along with an event timeline. The Ingest Summary presents a visualization of the events on the timeline.

Data Model for Universal Threat Detection

Most companies already have threat detection systems, with an intrusion detection system here, a threat detection and response solution there, and other tools in other network segments. This haphazard approach creates a problem that makes it more difficult to detect security problems: mountains of incompatible or duplicated data. A uniform data format for the detection of threats would be very important.

Apache Spot therefore uses the Open Data Model (ODM), which bundles all security-relevant data (e.g., events, users, networks, endpoints, etc.) in a single view. A consolidated view makes the relevant contexts visible at the event level. Another advantage is that the ODM allows sharing and reuse of threat detection models, algorithms, and analyses.

The ODM has a common taxonomy for describing security data used to detect threats, with schemas, data structures, file formats, and configurations in the underlying Hadoop platform for collection, storage, and analysis. The Spot system defines the relationships between the different security data types and links the protocol data to user, network, and endpoint identity data.

The Apache Spot ODM is particularly useful for out-of-the-box analysis to detect potential threats in DNS, flow, and proxy data. In addition to the standard analyses created in Apache Spot, you can create custom analyses according to your specifications. In the future, third-party plugins are conceivable that analyze the data that is collected and consolidated by Spot. The development team hopes to see significant contributions from the Apache Spot community to the development of threat detection models, algorithms, pipelines, visualizations, and analytics on the basis of the data model.

Quick Start in Spot

With the release of Apache Spot 1.0 and the ensuing assessment of its production capability, a mature version is available for evaluation. Because Apache Spot integrates with any infrastructure, no changes to the existing environment are required. However, commissioning is not trivial. To get a first impression, you can also install a demo system. All you need is a Docker installation; then, execute the container:

docker run -it -p 8889:8889 apachespot/spot-demo

To access the web interface, go to http://localhost:8889/files/ui/flow/suspicious.html#date=YYYYYY-MM-DD . For example, if you use 2016-07-08 at the end of the URL, you end up in the overview of suspicious network events on July 8, 2016, where you can conduct a detailed investigation with Spot. Alternatively, a manual installation is possible: At the core of Apache Spot is a Hadoop system. HDFS, Hive, Impala, Kafka, Spark, YARN, Zookeeper, and HDFS must also be present before installation. Once you have set up the required users, you can download the Spot code. After unpacking, use the spot.conf Spot configuration file, which is located in the /home/safety/incubator-spot/spot-setup directory.

Using keys such as NODES and DSOURCES, you determine the network nodes and data directories to be monitored. Use the next steps to install the ingest and ML components. The analytical tasks are handled by the spot-oa module. The final step is the installation of the UI component. The Spot Dashboard can then be accessed.