Free Interface, Proprietary Failover

To help users and administrators keep track, LizardFS provides a simple but sufficient web GUI in the form of the CGI server (Figure 4). The admin can basically install the component on each system and thus have an overview of all the servers on the network, the number of files, their replication states, and other important information.

Figure 4: The LizardFS web interface displays the servers in the cluster, the number of files, and their replication states.

If you enter a support agreement with Skytechnology, you can use the proprietary uRaft daemon, which is independent of LizardFS and provides automatic failover according to the Raft consensus algorithm [6]: All masters on which the uRaft daemon is installed use heartbeats to select a primary master. For the quorum to work reliably, the number of masters must be odd. They also all need to reside on the same network, because the primary master's IP (floating IP) moves with the primary master.

In the case of uRaft, simple shell scripts make sure that the master is promoted or downgraded appropriately. If you run LizardFS with the uRaft daemon, you abandon the typical master and shadow roles and leave it to uRaft to manage the respective roles. uRaft even takes care of starting and stopping the master daemon, which makes the init script of the LizardFS master useless. Figure 5 shows uRaft providing information on cluster health.

Figure 5: Output of uRaft daemon on the current cluster health.

Monitoring

For basic monitoring of a LizardFS environment, the aforementioned web interface is available; however, it cannot replace classic monitoring. LizardFS uses a special output format for almost all administrative tools, which also allows for querying of states. Thus, there is nothing to keep you from setting up a connection to Nagios or some other monitoring tool.

Thanks to compatibility with an earlier version of MooseFS, many monitoring modules or plugins for the original project – of which there are many on the Internet – also work with LizardFS (Figure 6). Resourceful system administrators can write their own checks, as well. An even simpler solution is possible: I worked for a company that has entered into a support agreement with Skytechnology. In the context of the support services, we received Nagios checks that, apart from a couple of details, allowed reliable monitoring of the LizardFS components.

Figure 6: The Nagios Adagios interface showing the result of five LizardFS checks on a host.

Where to Use It

The POSIX-compliant LizardFS can in principle be used wherever a client requires disk space. The most sensible use arises when user data needs to be stored redundantly for better availability and you want to avoid using classic (think "expensive and inflexible") storage appliances. In contrast, LizardFS distributes the data on the basis of its own system across a more or less large and widespread, in terms of network topology, pool of inexpensive standard machines.

However, this scheme can also be a disadvantage: You cannot specify that related data is stored contiguously on a chunk server. Although you can use topologies to influence the storage locations (e.g., one copy in rack 14, another in rack 15), this not a guarantee of fixed locations. Contiguous data that spreads across more than one chunk server (and therefore across rack or even data center boundaries), naturally leads to changing network latencies, resulting in significantly different response times for application read and write operations that is noticeable to users.

Distributed storage for large, heavily used databases proves to be detrimental in practice – system architects do well to be slightly skeptical of LizardFS in this case. For everything else, however, it is perfectly suitable, including, for example, for rendering farms or media files in which all clients simultaneously access nearby (in terms of network topology) chunk servers, thus pushing server performance to the limit.

The ability to use LizardFS as storage for virtual machines is little known. According to its own statements, Skytechnology is working on better VMware integration.