Moving massive blocks of data
In February 2007, a large e-commerce company decided to migrate its transaction databases from five storage arrays into one big one. "Big" may be something of an understatement: this company had about 206 TB of data. It was using Veritas Storage Foundation with Volume Manager to manage its complex data storage needs.
The company called in a third-party supplier to perform the migration. The third-party software had a new feature that would allow the migration to take place while the storage array was up and running. This sounded like a good idea, but since the feature was new and the data was mission-critical, the e-commerce company decided to try it out with a test migration of about 90 TB.
Migration to nowhere
The test did not go well. Not only did the migrated material fail to appear on the new disks, the original disks were altered, making the data inaccessible. The e-commerce company desperately needed its data back. Since the disks were running Storage Foundation, the third-party vendor requested support from Symantec.
The call came to the attention of Rich, a Symantec support engineer with about 16 years' experience. Because of his history with Veritas, and with the third-party supplier, Rich's colleagues knew he'd be the best person to address this issue. Rich, however, was on vacation at the time. "I was relaxing and decided to check my email," Rich recalls. "Somebody saw me log on, so they started IMing me. And then my cell phone rang."
"Destroy" routes to wrong disk
Rich and his colleagues identified the problem: The third-party migration tool sent a "vxdisk destroy" command to the target disk, and deleted the disk's private region. Although the data remains on the disk, vxdisk destroy removes the pointers to that data. However, as a safety precaution, Storage Foundation won't allow vxdisk destroy to execute while disks are imported. Since the new disks were up and running during the migration attempt, the disk destroy command failed against the new disks.
However, it succeeded against the original disks, which were not imported. This is why the pointers to the data on the original disk disappeared. Once Rich and his team discovered the problem, Symantec's front-line tech support helped the e-commerce site recover the data within a couple of hours.
Volume Manager comes through
The missing data was back, but the customer no longer trusted the third-party migration tool, and it still had a migration to complete. "They said, 'Don't we already have a tool built into Volume Manager that will do this?'" Rich recalls. The answer was: Yes, they did.
The Volume Manager tool is slightly more disruptive than the third-party tool would have been—if it had worked correctly. "With their tool, it would take two to three days to migrate all the data in the background, but the actual switchover would have taken an hour or less," Rich says. "With us, the migration would take four or five days to copy the data, and one or two hours of down time to flip the switch."
The customer decided this was a worthwhile tradeoff for a more reliable migration tool, and instructed its third-party consultants to complete the job using Volume Manager. The end result was a successful migration, and a very satisfied customer.
