digifuture, 123RF

Moving your data – It's not always pretty

Moving Day

Article from ADMIN 15/2013

By Jeff Layton

The world is swimming in data, and the pool is getting deeper at an alarming rate. At some point, you will have to migrate data from one set of storage devices to another. Although it sounds easy, is it? We take a look at some tools that can help.

Almost everyone reading this article has, at some time in their life, moved from one place to another. This process involves packing up all your possessions, putting them into some type of truck, driving them to the new location, unloading them, and ultimately unpacking everything. I've done it several times and, to me, it's neither fun nor pleasant. A friend of mine loathes moving so much that he has threatened just to sell or give away everything and buy new things for his new home. Honestly, I'm not far from his opinion, but at the same time, it's always interesting to see what you have accumulated over the years and, perhaps more interestingly, why you have kept it.

I think the same thing is true for data storage. At some point you're going to have to move the data from your existing storage solution to a new one, for any number of reasons: Perhaps your current storage solutions will become old and fall out of warranty, or perhaps you need additional capacity and can't expand your current storage. Of course, another option is just to erase (burn) all of the data on your existing storage solution.

Approach

I recently went to the Lustre User Group conference [1], and I was impressed with a presentation by Marc Stearman from Lawrence Livermore National Laboratory. The title of his talk was "Sequoia Data Migration Experiences" [2]. Sequoia is a large petascale system with 55PB of storage running Lustre on top of ZFS [3] and reaching about 850GBps. Marc talked about moving data from older systems to Sequoia and some of the issues the team faced (it wasn't easy). His talk inspired me to examine tools that can be used to migrate data.

One of the many ways to migrate data is a block-based approach, in which you copy a

...

Use Express-Checkout link below to read the full article (PDF).