A few years ago I was spending a great deal of time on call – day and night. It rapidly became obvious that we needed to implement company-wide server security that would be effective when on-call personnel were away from the office.

In this article, I describe a solution the team developed that combines TCP Wrappers with port knocking, allowing engineers to work remotely, without compromising the integrity of the company security policy.

Almost every computer application demands a level of security to enforce privacy – from software houses developing the latest ground-breaking game to time-sensitive academic discoveries. Across the many facets of computing, security is one of the most fascinating, especially online security, because security of the Internet must contend with many unseen hurdles thanks to its bare-all public-facing nature.

Hit the Road

One of areas we looked at included using one mobile cell phone carrier universally within the business for engineering staff and tying SSH access specifically only to their IP address blocks.

Sadly, two show-stopping issues quickly became apparent. First, because large carriers (Verizon in this case) allocated hundreds, if not thousands, of IP addresses to its GPRS and 3G customers, this approach opened up our servers to many more IP addresses than in our old system, with its 10 or 20 authorized IP addresses at the company offices and at the homes of those on call. Granted, the rest of the Internet couldn't gain access, but the difference between a handful of authorized IP addresses and thousands is huge.

Second, this approach added to administration because the IP address blocks changed every month or two; as the carrier's IP address pool grew larger, sometimes despite our best efforts, the on-call engineer was inevitably locked out during emergency callouts.

Next, we tried dial-up access via back doors and other out-of-band access (which I will keep to myself for security reasons because it's still in use). Unfortunately, it was all too easy to become overly reliant on the providers offering this access, and when they had connectivity outages, disaster struck at our end.

Being as diligent as possible, we all carried SSH keys around on phones, memory sticks, and laptops, but as hard as we tried, at some point, we didn't have access to those SSH keys in the middle of night, which meant an emergency couldn't be dealt with without harassing other members of staff at home who weren't on call.

Finally – and, as you can tell, we really searched for a solution – we paid for a couple of remote shell accounts with static IP addresses and allowed them access in addition to the office IP addresses.

Yet again, the reliance on these providers caused headaches at one point or another. Additionally, we were conscious that we were advertising our ever-so-precious login information to third parties (which could have been compromised).

Enough Already

Eventually, after too much coffee and a little testing, we were certain that we had found a solution. What the team liked about this solution in particular was that for some systems we could still arbitrarily meld it with our other preferred security methods, such as SSH keys and firewalling, if required.

As with anything relating to computing, there's always one hurdle or another. In this case, we only began implementing our security solution after some unexpected head scratching, because our preferred route used iptables (or more accurately Netfilter, the built-in Linux firewall) by default.

The iptables scripts were getting lengthier, and as powerful as iptables is, we didn't want to let junior members of staff break access for everyone by inadvertently making a config mistake while setting changes live under pressure.

Deviating a little from the norm, therefore, our solution incorporated what I like to think of as a natty fix using my favorite security tool for SSH: TCP Wrappers. By adding TCP Wrappers and a highly effective security methodology called port knocking , you can switch SSH access on and off securely and arbitrarily from any IP address on the Internet.

Knock on Wood

If you haven't heard of port knocking, it allows a sys admin to configure a sequence of connection attempts of varying complexity that acts as the first step in gaining remote access to servers.

The next step would be to provide a valid SSH key or password (e.g., if SSH was used). One such package available in the Debian and Ubuntu software repositories is the well-thought-out and intelligently built knockd.

The sequence I mentioned involves using a client program to knock on what appear to be otherwise closed ports on a server. Because knockd listens at the network link layer, it's suitably sophisticated to allow monitoring of both closed port traffic as well as open ports.

This means you can check without discrimination what goes into your network interface, making no effort to read that information and otherwise add load to your server, but instead just read where the traffic is destined.

So clever is port knocking that if you use iptables to hide your port completely, then you can also disguise the SSH server port entirely, pretending that it doesn't even exist to attackers (so there's nothing to attack).

Until the predetermined sequence of ports or secret knocks has been accepted, allowing the server port to then become visible, these ports are entirely invisible. This approach is also very slick because it means you can catch nefarious port scanners probing for services running on your servers and banish them outright without fear of them guessing your magic port sequence [1].

At this juncture, it's also worth noting that the long-standing armed guard of port monitoring, PortSentry (which has been around since calculators were oversized and on your wrist), also captures – or more accurately monitors – connections to closed ports in the same way. To my mind, the "Customizing PortSentry" article  [1] complements the topic of this article very nicely and might help you build a super-simple but highly effective security solution if you're looking to do so.