RAID for the Network

Stable Internet connections (uplinks) are mission critical in many enterprises. Unfortunately, they often break down. If you want to connect two uplinks redundantly using two or more providers, you will typically experiment with the Border Gateway Protocol (BGP). This solution can be a fairly expensive, though, because providers charge dearly for enterprise-level connections. With a few restrictions, you can achieve redundancy far less expensively by opting for Linux and the Fault Tolerant (FT) Router [1].

A Linux host typically sends its packets with the help of a routing table. All packets that do not belong to a specific route follow the default route, which usually leads to the Internet. If this link fails, all the users on the inside are cut off from the Internet. The reasons for failure can be many, including bulldozers digging up cables, Layer 2 or 3 software failures, or routers that fail one hop downstream on the provider's network.

To avoid hard disk failures, administrators have relied on RAID for a long time; in the simplest case, this means simply doubling the number of disks in a mirroring RAID [2]. This isn't quite as easy for access lines. In the classic setup for this scenario, at least two Internet providers safeguard the network; that is, your own connection has two uplinks.

The administrator needs to inform the rest of the world using BGP (on internal networks, this can also be an internal routing protocol such as OSPF, or Open Shortest Path First). If one link fails, the protocols notice this and stop sending packets over the dead link.

The protocols detect failures automatically. If the Internet Protocol (IP) fails even though the link is working perfectly at the lowest level, the routing protocol notices this through active monitoring. Although it can take a while, at least the changeover happens without intervention (i.e., without forcing the administrator out of bed in the middle of the night).

Although the "I" in RAID stands for "Inexpensive," a redundant network connection is not something you can have for $19.99 a month. Instead, you are probably looking at a three-figure amount. However, administrators with less cash at their disposal would probably relish the prospect of turning their Redundant Array of Links (RAL) into a RAIL.

Fortunately, Linux comes with almost all the components that you need to implement such a RAIL. However, the network administrator does need to enable it kernel-side. Linux fails to detect link failures automatically; instead, it would continue sending data packets into a black hole at the end of the failed link, and the user would receive non-reproducible load errors. Before you integrate FT Router into your setup, it is a good idea to familiarize yourself with the underlying software.

Wild Routing with iproute2

The ip command in the iproute2 package can configure everything the Linux kernel offers in terms of network technology. Many functions can only be configured with this tool (e.g., routes in additional routing tables); in other cases, it replaces standard tools such as ifconfig, route, or vconfig, which creates VLANs.

For example, ip can create multiple-path routes. To do so, you need to assign one of your new routes the table <X> parameter, where <X> can be a number or name that points to a specific routing table in the /etc/iproute2/rt_tables file. The command

ip route add table 5 192.168.0.0/24 via 10.1.1.1

assigns the route for network 192.168.0.0/24 to routing table number 5, for example.

Policy Routing

If multiple routing tables exist, the administrator needs to tell the kernel when to use which routing table with the ip rule command, which is used to create rules for the source and target IP addresses – network blocks are also permitted. You can define which network interface to use and a value for the type of service (TOS) field in the IP protocol.

To keep the rules as flexible as possible, you can also use the fwmark argument. This hexadecimal value lets you track the firewall marker set by iptables; iptables references this along with the mangle attribute to route by ports or protocols (Listing 1). For more detailed information, search the web for "Policy Routing" [3].

01 iptables -t mangle -A PREROUTING -i <source_interface> \
  -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
02 iptables -t mangle -A PREROUTING -i <source_interface> \
  -j CONNMARK -p tcp --dport 80 -m conntrack --ctstate NEW --set-mark 0x10

To send all the packets from subnet 172.16.0.0/24 to routing table 8 with the default route 10.2.2.1, you would enter the following two commands:

ip route add table 8 default via 10.2.2.1
ip rule add from 172.16.0.0/24 lookup 8

Administrators need to be aware of one small thing when using multiple routing tables: You do not have the routing entries the kernel creates automatically on creating a new IP address. You need to add these manually if you want to use them (e.g., in Policy Routing). Otherwise, Linux sends all response packets via the wrong route.

If you only want to forward outgoing HTTP traffic over a different route, you can create the two iptables rules from Listing 1. The first will restore the markers on the outgoing packets, and the second assigns packets that have a target of port 80 a hexadecimal marker (--set-mark 0x10). You will want the routing rule to watch out for this in future:

ip rule add fwmark 0x10 lookup 8

Using this approach, you can redirect packets, but the kernel does not distribute them across different routes.

Multipath Routing

Using the ip route command, administrators can set up multiple default routes – even with different weighting. This assumes that the kernel was built with multipath routing support up front.

Linux typically balances data packets at the packet or connection level. Distributing the packets individually will achieve maximum utilization of both links, but it does entail a couple of risks in practical applications. For example, a counterpart at the other end of the link organizes the packets again in a similar way; otherwise, only the upstream is load balanced.

If the links use different speeds, it's possible that the sender transmits the packets in the right order, but they arrive in the wrong order. Because TCP connections only compensate for differences up to the buffer size, the speed settles to that of the slowest link. Additionally, address translation is impossible on this route.

Alternatively, admins can set up each new connection on a different link. Although the individual streams are, at most, as fast as the quickest single link, this still leads to improved data throughput for users.

To set up to default routes with load balancing, you would do this:

ip route add default nexthop via 10.1.1.1 dev eth1 \
  nexthop via 10.2.1.1 dev eth2

If you want to work at the packet level, you need to add the equalize keyword after the add command. To achieve weighting, type weight <number> behind the individual hops. If a route now fails, every second connection will hit a timeout.

At this point, you have all the building blocks in place to send data across multiple links, monitor with a couple of simple commands, and remove one link from the cluster in case of failure. So, it's time for FT Router to enter the game.