upgrade master without downtime

[Razvan_Cosma]

Hi - reading through the most recent docs (7.6 at the time), several HA solutions are described, none though that would allow a straightforward upgrade (even for minor release numbers) without bringing down the master. This seems to imply that any hotfix requires a serious maintenance window, with all NB processes stopped and I'm looking for a more resilient scenario; the most promising one seems to be multi-site single-domain catalog replication (synchronous or not, let's assume there are no performance concerns) - in such a setup would it be safe to assume that one node can be disabled, patched, rebooted, etc and then still communicate properly when brought back online? No version of OS or appliance constraints, whatever will work.

[Nicolai]

All commodity software need re-start upon cluster fail-overs (netbackup is one of them). Only very special OS has the capability to do switch over in a running condition (e.g HP Nonstop). 

One thing is what you can do, another is what the vendor support. Stick to a supported configuration - and for Netbackup that is a active/passive cluster configuration.

Symantec NetBackup 7.6 in Highly Available Environments Administrator’s Guide

http://www.symantec.com/docs/DOC6485

[Razvan_Cosma]

Hi - and thank you for the clarification, of course what is possible may exceed the vendor's supported implementations and I am looking for a supported one; my question was a different one though: would a setup with catalog replication allow for at least minor release updates of a master by failing to the other node, patching, bringing it back online and patching the other one? Most clustering solutions have this as an option, NB seems to be the only one that explicitly says in the doc you mention "Upgrade NetBackup on the primary site master server by following the standard upgrade procedures (refer to the NetBackup Upgrade Guide)" thus my confusion.

[Mark_Solutions]

The thing with NetBackup upgrades, especially with 7.5 and 7.6 is that they make significant changes to the catalog and NBU is not functional until that has been done

So clustered or not there would be no way to upgrade without this downtime - do not try and shortcut that - it just won't work i am afraid.

[Jaime_Vazquez]

The upgrade process has significnat dependencies on the overall environment. It has multiple database connections in use and the upgrade process itself may be making changes to the schema on any or all of the databases as part of the upgrade.  These changes cannot be implemetned with the database open for normal NBU activity updates at the same time. Even if you fail over to the secondary server while upgrading the server, the databases are still open and locked by the database manager service.. 

Beyond that, running services need to be stoped/started in order to load into kernel space the latest executable images. 

[Razvan_Cosma]

Hi - so for a major release the only way to go would be to upgrade in-place, keeping services down for as long as it takes and with restore from backup/snapshot as the only way to reliably rollback should anything fail; this doesn't sound very enterprisey, how about installing a fresh instance with the new version, importing the catalog from the old one and switching ip/hostname when done?

Even for a minor release, having to update all NetBackup nodes (including the active one) incurs significant downtime that I'm trying to avoid

http://www.symantec.com/business/support/index?page=content&id=TECH214571

http://www.symantec.com/business/support/index?page=content&id=TECH214575

[Marianne]

There is no such thing as catalog import from a lower release.
NBU catalog recovery can only be done to same version and patch level.

Furthermore, master server hostname (or Virtual hostname) is extremely important. 
You cannot install NBU on one hostname and then change it. Hostname and/or virtual hostname is built into databases during NBU installation.

I am not sure why you would need this level of high availability for a backup product?
 

[LucSkywalker195]

Your disaster recovery capability is important enough to take the outage. Backups are mostly a backend/off hours activity to allow your Tier 1 assets the infrastructure resources during primary hours when end users need them. HA means you can do a restore 24/7 to a given point in time when a backup image was captured. Look at the rate of differential data changes during the weekly cycle and identify what day you experience the smallest amount data backed up on average. If you plan your upgrade for a mid-week backup cycle and communicate to your users in advance you can take the outage with minimal interuption and risk and still get your systems back online in time to complete your normal backup window.

[Razvan_Cosma]

Thank you everyone for the hints, though I must say I'm surprised by the strictness of this upgrade in-place rule, especially given the way NB master saves its data: it's all in the Sybase DB that can be opened using the freely available developer tool and a quick test I've just done shows the services do start after simply copying the data folder to a new system and doing a quick search/replace in the obious places (there are 139 tables); the upgrade could be performed there and afterwards just redirect the media servers by editing their registry/bp.conf

As for why such HA level is needed - not my decision, it just is, but I do think it's a worthwile thing to research.

[Jaime_Vazquez · Accepted solution]

Exactly how much of a service interruption hit do you think you are going to take? The largest upgrade effort involves going from a 7.5.0.X installation to a 7.6.0.X installation. The reason for that is because the Sybase engine used is upgraded from V11 to V12. That causes NBU to perform some actions on the databases to be consistent. The larger the database files, the longer the upgrade takes. Some time is also spent making a backup of the running system environment in case it needs to be downgraded. 

But, again, from my experience, we are talking less than an hour for this, unless there are some very serious mitigating circumstances.