eBPF On Board

Another Teleport feature, which the documentation very unjustly just touches on, is worthy of mention. Teleport offers comprehensive auditing and logging capabilities for the systems on which it runs, a feature you will not often find in comparable solutions.

Teleport handles tasks that are specific to the architecture of the solution. If a component is responsible for centralized login management, it is easy for this component to keep records of logins. Teleport refers to this as the Audit Log. The solution records who logged on to which server in the Teleport cluster at any given time and where the connection originated (Figure 3). The service then provides the information as a structured JSON file, which can be easily processed downstream in various services outside of Teleport.

Figure 3: Teleport logs the commands and traffic it sees in several ways – Audit Logs are the most basic form of logging.

Of course, log data of this kind also needs to be available externally. A successful attack could leave you with nothing but manipulated logs. With the use of an external data source, you might at least be able to determine the time of the hack to facilitate forensics. However, the market offers virtually nothing in the line of ready-made integration for logfiles originating from Teleport, such as for Loki or the Elasticsearch-Logstash-Kibana (ELK) stack, which forces you into a do-it-yourself integration of the Teleport Audit Logs with a choice of logging system, if this functionality is needed.

However, Teleport's logging capabilities go far beyond recording the metadata for connections (Figure 4). The developers have taken advantage of a technique offered by the Linux kernel itself – one that is brilliant but still attracts too little attention in the free/libre open source software (F/LOSS) world: the extended Berkeley Packet Filter (eBPF).

Figure 4: Checking the logs, you will find structured logfiles in JSON format, which can be easily processed on other systems. Currently, however, native integration with Loki or ELK is missing.

eBPF has been covered in our sister publication Linux Magazine [2] on several occasions. In practical terms, eBPFs are small virtual machines written in Rust that can be loaded into the running Linux kernel. Once there, they can implement arbitrary functions over a defined API. If you set this up correctly, eBPF can be given complete access to the traffic flow of a system or individual applications. eBPF is significantly more powerful than comparable nftables-based packet filters.

Increasing numbers of applications that rely on eBPF are making their way onto the market. Google has presented a framework in the form of Kernel Runtime Security Instrumentation (KRSI). The framework monitors complete data streams and sets up checkpoints at neuralgic points to enable applications to interrupt traffic flows. KRSI does this by combining eBPF, which implements the monitoring functionality, with the Linux Security Framework; applications such as SELinux and AppArmor use this in a similar way. Teleport can be made to terminate or interrupt ongoing sessions automatically if the content of the transmitted information meets certain criteria according to predefined benchmarks.

Preventing Data Leaks

A real-world example makes it clear that once an attacker has gained access to a system, no one can prevent them (if they are root) from arbitrarily offloading data from that system. Publicly defacing websites is no longer the focus for most hackers; rather the real target of attack is databases containing credit card data and passwords. From the point of view of the attacked company, uncontrolled and unwanted data leaks are a far bigger problem that, depending on the country, can have serious legal consequences.

In these cases, dynamic interruption of Teleport comes in handy, at least assuming the Teleport proxy nodes themselves do not fall victim to an attack. With the Restricted Session feature, you can define keywords that must not appear in the transferred sessions and that will automatically cause Teleport to terminate the connection. These keywords can be configured with the use of Teleport's own API. The feature is dynamic; for example, you could store a keyword in a hidden row in a database and tell Teleport to jump to that row during the transfer. This example is rudimentary, of course – anyone who wants to exploit the capabilities of Teleport and eBPF will have no alternative but to implement a holistic security setup and get involved in a little development work. It is remarkable, however, that Teleport allows this kind of setup at all; nothing comparable could realistically be implemented with OpenSSH and nftables.

Teleport's logging features also take effect in a far less invasive mode. If you "only" want an audit trail, Teleport can log all the traffic from SSH sessions. In terms of accountability, some certifications now go so far as to require more than just individual accounts for individual users; they also want to have proof of who executed what commands on a system and at what time.

For the principle to work, of course, it is important to have the Teleport proxies operated by different people from the target systems, because bad guys could otherwise easily bypass the self-installed protection mechanisms. However, given this clear demarcation of responsibilities, Teleport is ideally suited to meet the comprehensive audit requirements of various certifications.

If you intend to use Teleport in this way in Europe, you should be aware of the need to comply with data protection requirements under labor law. This is a little easier if you do not log and monitor the communication content and instead restrict it to allowed hosts. This process also works with eBPF. If the filter notices that packets are designated for an IP that has not been cleared, it automatically interrupts the connection on an application-specific basis (Figure 5).

Figure 5: eBPF sample code used like this in Teleport takes down a connection if it sees data migrating to undesirable IP networks (denylist). © Teleport

Kubernetes? No problem!

The benefits of Teleport described so far are primarily under the hood, but Teleport also offers many more visible features. The most prominent of these is undoubtedly Teleport's ability to act as a tunnel agent not only for SSH, but also for a variety of other protocols. It's true that SSH was the nucleus, and the Teleport documentation clearly states in many places that Teleport is an SSH server at its core. In the meantime, however, the solution has grown to see itself more as a kind of mediator between the worlds, as a few examples will illustrate.

In addition to the SSH protocol, the Teleport proxy supports the API of the Kubernetes cluster fleet manager. After a user's initial login against the Teleport proxy (tsh login), tsh builds the user's environment variables so that any requests to, say, Kubernetes or PostgreSQL are automatically redirected through the proxy server to the target system. On the system where the admin runs ssh user@host to access their target system after logging in to the Teleport shell, commands for connecting to PostgreSQL could subsequently be executed with kubectl or psql. Teleport is even available for remote desktop protocol (RDP)-based access to desktop systems (Figure 6).

Figure 6: Teleport enables authenticated access to virtual machines in the cloud if combined with Azure Active Directory or Azure Active Directory Domain Services. © Teleport

From the user's point of view, this diversity further increases the convenience of using the solution: Where you previously had to worry about correctly setting up authentication for each individual application, Teleport simply takes this work off your hands. Anyone who has tried to manage the credentials for accessing a MySQL database in a meaningful way will eventually bundle their .myrc off into Ansible or some other software provisioning, configuration management, or application deployment tool and be annoyed about having to do so. However, this setup is anything but dynamic and requires regular adjustments if access credentials change.

By configuring the database to accept user certificates from the Teleport CA, issuing passwords is practically a thing of the past and is very likely to please compliance officers, too.