Fault Tolerant

Source code for the Fault Tolerant Router project [4] and some fairly extensive documentation [1] can be downloaded from GitHub. Typing

gem install fault_tolerant_router

lets you install the software, which was programmed in Ruby (Figure 1). After launching, a daemon monitors your links and switches over when needed. When called with the generate_config option

sudo fault_tolerant_router generate config

Figure 1: The gem command installs the FT Router Ruby program easily on a Linux system – in this case, Ubuntu 14.04.

the program generates the /etc/fault_tolerant_router.conf configuration file (Listing 2). You need to modify this to suit your requirements. The file uses a YAML format [5] and contains the configuration groups uplinks, downlinks, tests, log and email.

uplinks:
- interface: eth2
  ip: 10.1.1.1
  gateway: 10.1.1.254
  description: Example Provider 1
  weight: 1
  default_route: true
  # Other Interfaces
[...]
downlinks:
  lan: eth1
tests:
  ips:
  - 192.168.254.254
  required_successful: 1
  ping_retries: 1
  interval: 60
log:
  file: "/var/log/fault_tolerant_router.log"
  max_size: 1024000
  old_files: 10
[...]

An uplinks definition typically comprises of the name of the interface, the matching IP address, the IP address of the gateway for the interface, a description, a flag, and a weighting. The flag determines whether the interface belongs to the default routes. The higher the weighting, the more often this link is used for new connections. You would use weighting if your links run at different speeds (Figure 2).

Figure 2: The configuration file for FT Router can be generated via the generate_config option. It comes with several sample interfaces that point to potential providers.

In the downlinks section, you need to define an internal interface – if so desired, a DMZ interface. The content below tests shows a list of IP addresses that the route is allowed to ping regularly in random order in the scope of functional tests. The addresses should be accessible on the Internet. You will also need to define the number of tests a link needs to pass before it is deemed functional and how often the router sends pings. You also specify a timeout parameter and a retry parameter.

The log section of the configuration file defines the maximum size and number of logfiles. If its status changes, the daemon can dispatch email. The parameters required for this are also stored in the configuration file. The last parameters act as start counters for routing tables, firewall markers, and priorities, which you want the router to assign automatically.

The script also defines the required mangle and NAT rules on the basis of the configuration. The installation guide recommends merging these with an existing configuration. Because the router typically also assumes the role of the firewall in the infrastructure, this definitely makes sense. The command:

fault_tolerant_router generate_iptables

generates the rules in iptables-save format.

To avoid any downstream routers requiring a configuration that takes the FT Router into consideration, you would NAT all outgoing sessions on all interfaces to match the IP address of the interface. Then, launch the monitoring process with the fault_tolerant_router monitor command. If you additionally set the --debug option, you can follow in detail on the console which routes the FT Router enables and how the various functional tests work out.

Three-Track

The ADMIN test lab was set up with an FT router with three uplinks in a KVM environment. The VMs are connected using Open vSwitch, which was also used to restrict the bandwidth on the router's uplinks. In other words, it emulated typical bandwidth patterns in a consumer environment. Figure 3 shows a sketch of the network. I then ran wget to download data from a web server.

Figure 3: The schematic shows what a test network with Fault Tolerant Router looks like.

On the interface between the client and FT router, iptraf-ng detected the traffic, revealing that wget commands started sequentially actually did run one after another with the assigned bandwidth as a result of session balancing. Starting multiple downloads at the same time activated all the links. On the route between the client and FT router, the traffic detector revealed that the load was occupying the full bandwidth.

The next step was to block one of the links. The download running on it then stalled. Until the daemon started the next ping test, new connections that would normally have used this link were not established.

Only when the daemon removed the link from the pool did all the connection establishments work again. Again, I enabled the connection again, and after another ping cycle, I was able to use the link in the normal way.

Conclusions

If a system administrator were tasked with creating a link that users would never notice failing because the technology compensated within milliseconds, FT router would be the wrong choice. However, for administrators who can accept a short wait to avoid a triple-figure monthly fee, FT router offers a solution that is easy to set up.

Beyond this, FT router uses all links if there are no failures. If your links run at different speeds, a download can take considerably longer if it happens to be running on the slowest link. Although you can try to prevent this outcome with weighting, you have no way to rule out this behavior fully.