Fishing in the Flow

Administrators monitor key network connections to detect issues (e.g., congestion) at an early stage. The Simple Network Management Protocol (SNMP) is often used for this purpose to query the metrics of the network interfaces. The measured values can be visualized as time series diagrams, and the user can define threshold values that trigger notifications if exceeded.

What happens, though, when the admin is notified? A quick look at the time series chart reveals that the network connection is busy, but this doesn't tell you which conversations and which applications are using the connection. Information from flows can fill this gap. Today, many network devices let you export this kind of information, but the opportunity often remains unused.

In this article, we look into the use of OpenNMS Horizon and monitoring with SNMP to visualize the make-up of network traffic with flow protocols. Given appropriate visualization in Grafana and unrestricted access to the flow data by Elasticsearch, OpenNMS Horizon can support administrators in their troubleshooting, capacity planning, and security tasks.

What Are Flows?

Flows are not essentially related to a connection on the transport layer, but to a set of Internet Protocol (IP) packets with similar characteristics that pass through a measurement point within a defined period of time [1]. As shown in Figure 1, these properties include the IP source and target addresses, the ports, and the transport protocol.

Figure 1: A flow is a set of IP packets with common properties, such as the source and target addresses.

The aim of NetFlow, originally designed by Cisco Systems and introduced to their routers in the mid-1990s, was to carry out CPU-intensive computation of routing decisions once per target and cache the results on the network component. However, NetFlow did not scale well because of the high volume of short-lived network flows in the backbone or core segment and was ultimately replaced by Cisco Express Forwarding (CEF). Luckily, the packet flow data originally collected with NetFlow was excellently suited to traffic analysis and capacity planning. Cisco therefore finally also implemented a protocol that allows the collected data to be forwarded to a flow collector to evaluate IP data streams.

NetFlow thus became the de facto standard, and other manufacturers also started to export data in the NetFlow format. To this day, version 5 of NetFlow is considered the most widely used because of its easy-to-implement packet format, which consists of an introductory header and a set of records. However, it should be noted that this version does not yet support IPv6 addressing and is therefore only suitable for collecting IPv4 traffic data. Finally, after the release of version 9 of NetFlow as an open standard [2], a template-based protocol approach was chosen that allows for flexibility in terms of exporting the information needed for evaluation.

The modeling and component standardization processes began in 2001 with the founding of the IP Flow Information Export (IPFIX) working group within the Internet Engineering Task Force (IETF). The goal was to develop a flexible message format that supports forwarding of specifically definable metrics. Finally, NetFlow protocol version 9, developed by Cisco Systems, was chosen as the basis for design and development. IPFIX, like NetFlow, uses a template-based approach, which means that a collector is given information on the composition of the data before it is transmitted [3]. IPFIX defines Information Elements, which are centrally registered and managed by the Internet Assigned Numbers Authority (IANA).

The sampled flow (sFlow) standard [4] also plays a role in collecting and analyzing traffic data on networks. This protocol describes a sampling-based method for sending individual packet headers to a collector at definable intervals. The data from packet flows collected by this protocol reference the header information at the time of sampling and are therefore a kind of snapshot, making sFlow particularly suitable for protocol-independent evaluation of header information.

In order to start the collection of flow data you need to configure a network component as a flow exporter for it to send the appropriate traffic data to a flow collector. Figure 2 shows an example of configuring NetFlow v9 on a Cisco Internetworking Operating System (IOS) device. A monitoring system must then provide a matching flow collector to receive the NetFlow data.

Figure 2: How a flow collector and exporter are configured for NetFlow v9 on a Cisco device.

Flows in OpenNMS

The free OpenNMS monitoring system has supported the acquisition and classification of this kind of traffic data since 2018. The developers paid attention to good scalability from the outset of the design process. OpenNMS supports NetFlow versions 5 and 9, as well as IPFIX and sFlow. Figure 3 shows the rough structure of the components needed to evaluate flow data with OpenNMS Horizon.

Figure 3: The OpenNMS Horizon stack for collecting flows.

OpenNMS uses a PostgreSQL database to store the monitoring inventory, events, and status information. In a simple installation, the time series are stored in round-robin database (RRD) files. Grafana can enter the play for custom and application-specific dashboards, which requires the installation of the Grafana plugin OpenNMS Helm [5] to provide the corresponding data sources: OpenNMS Performance, OpenNMS Entities, and OpenNMS Flow.

The traffic data has different characteristics than typical time series and is therefore stored in an Elasticsearch instance with the OpenNMS Drift plugin [6] installed for the necessary functionality of storing, querying, and aggregating traffic data. A minimal setup for flow evaluation with OpenNMS Horizon is shown in Figure 4.

Figure 4: A minimal OpenNMS for flow evaluation with OpenNMS Horizon.

Installation and Configuration

The telemetry daemon (telemetryd) component in OpenNMS Horizon assumes the role of the flow collector and decides which flow protocols it will receive on which ports. In the default configuration, this component is enabled and can receive and process typical protocols such as NetFlow v5/v9, IPFIX, and sFlow on port 9999/UDP. The protocol is identified on reading the flow header so that the data finds its way to the matching internal parser. You can customize telemetryd behavior in the telemetryd-configuration.xml file.

To store the flow data, the Elasticsearch nodes need an OpenNMS Drift plugin installed. In the next step, you need to tell OpenNMS Horizon how to reach Elasticsearch in a configuration file named $OPENNMS_HOME/etc/org.opennms.features.flows.persistence.elastic.cfg with the lines:

# Elasticsearch persistence configuration
elasticUrl = http://my-elastic-ip:9200
elasticIndexStrategy=daily

Depending on the expected traffic volume, you can choose different indexing strategies for storing traffic data in Elasticsearch (e.g., hourly, daily, monthly, yearly). In production environments, factors such as redundancy and indexes play an important role. The OpenNMS documentation [7] describes all the available configuration parameters. After creating the file, restart OpenNMS Horizon by typing

systemctl restart opennms

so that it starts saving flow data. Now you just need to configure the central network components to send their traffic data to port 9999/UDP of the OpenNMS Horizon instance. To enable meaningful evaluation of the flow-exporting devices, you need to use SNMP to monitor them as nodes in OpenNMS.