Safety Net

If high availability or load balancing is required on servers across geographically separated locations, many of these services require direct access over Layer 2. However, if the Layer 2 link, which is based on classic 802.1Q VLAN technology, is interrupted by a routed link in Layer 3, the required transparency is lost. A virtual extensible local area network (VXLAN) solves this problem by extending Layer 2 accessibility over the existing Layer 3 structure with an overlay network.

Layers

System administrators face the challenge of planning scalable networks while maintaining appropriate security and availability requirements. Some server systems need to be located in a redundant infrastructure on a subnet. Additionally, virtualized systems need to be capable of migration between multiple sites on the same subnet, whether in live operation or in a disaster recovery scenario. Moreover, today's networks need be able to meet the increasing demand for bandwidth.

In classic scenarios, the required security can usually be achieved by implementing virtual local area networks (VLANs) to reduce the size of broadcast domains in combination with firewall rules or static packet filters (access control lists, ACLs) on routers or Layer 3 switches. In some cases, routing virtualization implemented by virtual routing and forwarding (VRF) is used at the routing level to enable multiclient capability.

Layer 2 Limitations

Routed networks are typically used to ensure scalability and avoid Layer 2 loops. Layer 2-only networks use methods such as Spanning Tree defined in IEEE 802.1d, its extension Rapid Spanning Tree (IEEE 802.1w), or Multiple Spanning Tree (IEEE 802.1s). All three have the advantage of ensuring loop-free operation. Layer 2 loops overload the switches and can be complex to troubleshoot.

Freedom from loops is ensured for redundant links by path blocking, but this comes at the cost of compromising maximum throughput, because the maximum combined or load-distributed data rates cannot be fully leveraged in this way. Additionally, some switches impose restrictions on the number of spanning tree instances, which, in turn, limits scalability for methods that run one dedicated instance per VLAN, which is the case with Rapid Per-VLAN Spanning Tree.

Classical Network Architecture Limits

A routed design (Figure 1) addresses these issues – again, with different variants, such as a hierarchical three-layer model with core, distribution, and access switches. Routed access, in which VLANs are only implemented locally on the respective access switch or access switch stack, is increasingly being used on classic campus networks.

Figure 1: In the classic routed three-layer design, host A and host B must be on different subnets.

An alternative in data centers is the spine-leaf model, where central switches, the spines, are connected to the leaves by routed links, and the leaves are connected to the servers. These leaves are usually designed as top-of-rack switches. Normally, IT managers would limit the number of VLANs required to one top-of-rack switch at any time and route them on their own uplink levels to leverage the routing protocols' properties for redundancy and load balancing across multiple routed links. However, this setup results in VLANs being restricted to the rack only, which contradicts the requirement for flexibility in redundancy and virtualization solutions referred to earlier.

Many data center services still require direct Layer 2 connectivity. One example is VMware vMotion, which enables live migration of virtual machines from one hardware setup to another. This configuration would not be possible in routed designs in different racks because IP-based re-addressing is impractical in most cases.

On local networks with end users, this connectivity does not play a major role. After all, the host usually doesn't care about the IP subnet it resides on, as long as it can access the required services over its gateway without the firewalls blocking the attempt. However, data centers need more, so how can you ensure – despite routed designs – that redundant services in distributed data centers can communicate with each other over Layer 2 while allowing virtual machines to move between racks without IP address changes?