Photo by Dan Dennis on Unsplash

Sharding and scale-out for databases

Shards

Article from ADMIN 78/2023

By Martin Loschwitz

Apache ShardingSphere extends databases like MySQL or PostgreSQL, adding a modular abstraction layer to support horizontal sharding and scalability – but not replication or encryption at rest.

Scalable databases have mushroomed in recent years and, in some cases, rely on completely new architectural approaches. For example, YugabyteDB is a key-value store, but it offers a MySQL compatibility layer. Vitess, on the other hand, uses native MySQL databases in the background but inserts an abstraction layer between the user and the database management system (DBMS) that takes care of sharding and, in turn, horizontal scalability. The Apache project takes a similar approach with its ShardingSphere [1] variants but promises easy handling, seamless extensibility with plugins, and support for most common databases.

Redundancy and Scalability

Databases are a central component of most complex setups. True to the motto "a special tool for every task," a long-established practice is to let databases take care of data management because they solve the task best and most efficiently. The changes that have occurred since the advent of cloud computing – containers and the cloud-ready principle – also lead to tougher requirements for databases. In a scalable environment, the database also needs to be able to scale; single instances of MySQL or PostgreSQL are no longer sufficient.

Out-of-the-box DBMSs come with quite a few limitations that make them unsuitable for operation in cloud environments. Two problems are immediately apparent: First, modern environments assume that the services running in them are implicitly redundant. Typically – and this is especially true in microarchitecture services – any number of instances of a service are available for each task, monitoring each other and seamlessly taking over the tasks of a failed instance in an emergency. Anyone who has ever faced the challenge of achieving high availability for MySQL will be aware that, out of the box, MySQL does not come with any functions for this task.

...

Use Express-Checkout link below to read the full article (PDF).