The challenge was this: Some organizations were requiring as many as four or five attempts to upgrade to NetBackup 6.0 successfully. Why?
Dan, a NetBackup tech support specialist at Symantec, pondered that question as he worked with one of them, a large county government.
He and the upgrade team had just spent a day preparing to upgrade the county’s Microsoft Windows-based environment, which consisted of one NetBackup master server, 10 media servers, and about 100 clients.
The team had run a NetBackup catalog consistency check to ensure all databases in the backup process were consistent. There were still two small configuration changes they wanted to make in two remaining media servers—but because of change controls, they didn’t have administrative access to those servers.
The upgrade failed. However, to the relief of the county IT manager, Dan rolled back to NetBackup 5.1 in time to successfully run the night’s backup. Then he sat down to look for patterns in upgrades that did not go smoothly.
The “aha!” factor
He gathered tech notes from an earlier, similar situation, and combined them with details from that day’s upgrade attempt. “I created an upgrade preparation checklist,” he says.
One of the key steps on the checklist is making sure that all host names in NetBackup databases were active and assigned to storage units. “When some companies decommission NetBackup media servers, they don’t follow the decommission process completely,” Dan explains. “If a customer doesn’t remove an old host name from the database, the upgrade process fails.”
He also discovered it was best practice to upgrade all media servers at the same time. “If you are going to leave legacy media servers in your environment, where you have a master server at version 6.0 and media servers at 5.x and using Shared Storage Option or SSO, you’ll potentially have serious problems.”
“By following the checklist, we were able to verify and clean up better than 90% of all the known issues,” Dan adds. “When we made the upgrade attempt again the next day, everything took off and ran smoothly. We had 10 hours set aside for the upgrade. We were done in four.”
When it came time to upgrade that customer’s UNIX environment, the team went through the same checklist drill and got the same smooth results.
An upgrade portal is born
The Upgrade Guide is available for download at the NetBackup 6.0 Upgrade Guide Portal, and it has received tens of thousands of hits ever since. “Go to the upgrade portal, get the checklist and follow each step,” Dan says. “With due diligence, you get a clean and smooth upgrade experience.”
Some enterprises, however, weren’t entirely convinced of the checklist’s importance.
In late 2006, one large company was attempting to upgrade to NetBackup 6.0, and its backup time began to exceed its backup window. Don, another NetBackup tech support engineer, was investigating.
“The cause seemed to be a problem of name resolution, similar to what Dan had discovered with the large county,” he explains. “To help them see it, I created an evaluation script that cross-checked the consistency of their global device database against their storage units.”
The script produced a long list of naming discrepancies when it ran, and the company saw that they weren’t fully prepared to upgrade. After taking the checklist and conducting an extensive server-by-server cleanup, the upgrade was completed without a problem.
Looking down the road
Plans are now in the works to incorporate the upgrade checklist and the name consistency evaluation script into the next version of NetBackup. Additional portals supporting upgrades from NetBackup version 5.x to 6.5, and from 6.0 to 6.5 are being prepared. These checklists will help ensure that future customer upgrades go smoothly.