One to two failures a week
A telecommunications company was using Veritas NetBackup Storage Migrator to automatically save files to tape and then remove them from disk, minimizing disk space usage. Because the data backed up to tape included telephone records, and the company's policy was to move the data to tape after a week, users frequently sought to retrieve data from tape—about 5,000 times a week.
Frequently, instead of performing the operation normally, the system would hang up and fail to retrieve the data. This led to the company calling Symantec for support once or twice a week, to ask for help obtaining the missing tape data. Jan, a support engineer with seven years' experience, a long history with this particular customer, and expertise in NetBackup Storage Migrator, responded to these calls.
Jan reviewed error messages and copies of system logs from the customer, looking for the causes of the failures. They turned out to have several causes, mostly related to hardware issues.
"A lot of times it was their tape drives," Jan says. "Either there was a bad sector on the tape (in which case the data could be retrieved from a different tape because NetBackup Storage Migrator always makes a duplicate copy), or too many tape drives were down and there wasn't one available, or all the available drives were already in use. In other cases, the server sending the request was overloaded, so the request was not getting through to the tape drive at all."
Working with her contacts at the company, Jan was able to resolve each failure over the phone, one at a time. But the frequent failures highlighted an underlying problem: the company's tape library and servers were not up to handling the high volume of requests to retrieve backed-up data that the company was experiencing.
Getting the setup right
When the company decided to address the problem by installing a new server, IT staff requested in-person assistance from Jan, because she was well acquainted with the company's systems and known to its personnel. "I was on site for a week," she says. "I sat with the system administrator who manages NetBackup Storage Migrator and I went through their logs to see where the issues were."
In addition to helping make sure the server was configured optimally, Jan provided training to the "sysadmin" on how to make NetBackup Storage Migrator work most efficiently. For instance, Storage Migrator has an internal database that keeps track of where data is stored on the tape, and can overtax the system if it grows too large. She taught the sysadmin to keep tabs on the size of this database, and how to start a new one periodically to keep any single database from growing too large.
She also provided training to end users who regularly use NetBackup Storage Migrator to retrieve the data they need. "When an operation failed, they would call IT and say, ‘It didn't work!'" Jan says. "I taught them to report specific error messages and provide other information that could help the sysadmin resolve the problem more quickly."
The new server, along with the help Jan provided, made a huge difference. From one to two support calls a week, the company is now down to one a month, or less. This is quite an impressive improvement, especially considering that its users retrieve data from tape 20,000 times or more every month.