Upgrading to 7.5, concerns about SLP auto replication between two masters

[bilson123]

Hello,

I am trying to plan our migration to netbackup 7.5. Our current infrastructure is on 7.1.0.4. We have 1 master 2008R2 7.1.0.4 server that acts as a storage server (hosting a puredisk and advanced disk) and two linux media servers on 7.1. The master is a media server as well. We also have another master 2008R2 7.1.0.4 as well, in a separate location that serves as the destination for SLP configured with Auto replication between the two sites.

I have read the 7.5 upgrade guide, the 7.5 upgrade center and portions of the install guide that are referred by the upgrade guide, and watched a video on Symantec.com that shows 7.1 to 7.5 upgrade. The one part that I am not clear about is, do I bring down both masters and update them at the same time? I am mainly concerned about the version mismatch between the two. If I upgrade the source master first and bring it back up, will it start auto replication as soon as catalog migration is done? Are there any issues with duplicating SLP on 7.5.0.6 (latest as of now) to a server that still runs 7.1.0.4. One thing I noted is SLP policies don’t seem to have a disable button in GUI, so I probably need to look up the command (7.5 release notes say a button has been added). Actually, I found the command in the best practices pdf for 7.1:  nbstlutil inactive –destination "remote_server_hostname"

 I have briefly discussed this with Symantec support while working on another issue, and initially they said to upgrade both servers at the same time, but later that it may be good to bring up the master first and make sure DB migration has completed ok. If I do just that, will I have to have replication suspended?

I am fairly new to netbackup, so I would greatly appreciate your input.

[revarooo]

Personally I would upgrade both at the same time. Do one, confirm its able to do backups, then do the other.

Highly recommend upgrading to 7.5 then upgrading to 7.5.0.6 immediately before you bring up the services. There were some known issues with catalog import into the database at 7.5. As soon as you start the services it will attempt to import the catalog into the database

[Jaykullar]

I upgraded from 7.1.0.4, I did pretty much what revaroo said. I upgraded the production master to 7.5, straight to MP5, ran the catalog import into db, checked all was well. Then did the BC master we AIR to, once confirmed then re-enabled SLPs. (I did however upgrade all the media servers before reactivating policies)

[bilson123]

I am thinking about upgrading the master server all the way to 7.5.0.6, bring it up and let it finish metadata migration. Then do one backup job that uses the advanced disk (no deduplication so SLP should not be affected), verify that I can browse older backups in Backup, Archive and Restore and then proceed with the other master server. Next upgrade media servers. Then enable backup policies and media servers on the master.

I think this would be also a good time to decomission the linux media server running 7.1 and replace it by a windows host. According to the version compatibility chart at (http://www.symantec.com/business/support/index?page=content&id=TECH29677) media server at 7.1 is still compatible with master at 7.5. I may not be able to get all of this done on a single day, so it's nice to know that I can let backups run and finish upgrade next day.

Thanks

[Jaykullar]

Sounds good, dont forget to set all your SLPs to inactive before you start. Also if you have NBAC enabled, added new string in HLEY_LOCAL_MACHINE > SOFTWARE > Veritas > CurrentVersion>Config USE_VXSS - PROHIBITED. Have seen alot of people including myself have problems with upgrades when NBAC is enabled, resulting to DR!

A few notes for image metadata migration.

1. After installation of NetBackup 7.5 and patch, restart the NetBackup services. (After NetBackup is restarted, phase 1 of the image metadata migration begins automatically.)
2. Start the NetBackup Administration GUI, and select the Activity Monitor.
3. To start phase 2 of the image metadata migration, run the "install_path\NetBackup\bin\admincmd\bpimage –cleanup -allclients"command and wait for the image clean-up job to complete
4. This will spawn a clean-up job in the Activity Monitor to confirm that the Image Metadata migration is progressing. Once complete, you can check that the two phases of migration have completed by running “install_path\NetBackup\bin\admincmd\nbemmcmd -listsettings -brief -machinename masterservername” and check for the following: LIST_FS_IMAGE_HEADERS/ SLP_DSSU_MIGRATION_STATE lines:
   1. If the result shows SLP_DSSU_MIGRATION_STATE="1", phase 1 has completed.
   2. If the result shows SLP_DSSU_MIGRATION_STATE="0", phase 1 has not completed.
   3. If the result shows LIST_FS_IMAGE_HEADERS = "0", phase 2 has completed.
   4. If the result shows LIST_FS_IMAGE_HEADERS = "1" or if no result appears, phase 2 has not completed.

[bilson123]

"Sounds good, dont forget to set all your SLPs to inactive before you start. Also if you have NBAC enabled,"

So did you use the nbstlutil command to disable slp?

I don't think we use NBAC, when I click on Access management, I get not configured on this system. I guess that's a good thing.

[revarooo]

Sure is a good thing, it's not configured in that case :)

[Jaykullar]

Yes, use nbstlutil inactive -lifecycle (name)

Also de-activate all policies and media servers whilst upgrading.

[bilson123]

Performed the upgrade fine, however, Puredisk volume is staying at down state, even after UP and Reset commands. Advanced disk is ok and backups can write to it but not puredisk. All media servers have been enabled. I verified that LIST_FS_IMAGE_HEADERS = "0", which means phase 2 is complete.

Cleanup job of the migration process ended with status 1 in Activity monitor. All image imports were successful but it complained that NBU status: 2074, EMM status: Disk volume is down

Here is the last portion of my cleanup job log. Prior to this all imports show success. Replication master is still at 7.1.0.4.

08/05/2013 12:30:45 - Info bpdbm (pid=11852) [000:03:04] Overall progress: 3647 images imported, 0 skipped, 0 corrupt.  Import rate = 19 images/sec
08/05/2013 12:30:45 - Info bpdbm (pid=11852) Finished importing all images into the database.  (Count = 3647)
08/05/2013 12:30:45 - Info bpdbm (pid=11852) Cleaning up tables in the relational database
08/05/2013 12:30:45 - Warning bpdbm (pid=11852) Hot catalog backup is not configured for 'mediaservername', catalog cleanup will return partial success until hot catalog backup is configured.
08/05/2013 12:30:45 - Info bpdbm (pid=11852) deleting images which expire before Mon Aug 05 12:27:39 2013 (1375720059)
08/05/2013 12:31:55 - Info nbdelete (pid=8132) deleting expired images. Media Server: mediaservername Media: @aaaab
08/05/2013 12:31:55 - requesting resource @aaaab
08/05/2013 12:31:56 - Error nbdelete (pid=8132) Cannot obtain resources for this job : error [2074]
08/05/2013 12:31:56 - Info nbdelete (pid=8132) deleting expired images. Media Server: mediaservername Media: @aaaac
08/05/2013 12:31:56 - Error nbdelete (pid=8132) Cannot obtain resources for this job : error [2027]
08/05/2013 12:31:56 - Warning bpdbm (pid=11852) nbdelete failed with status (174)
08/05/2013 12:31:56 - Info bpdbm (pid=11852) deleted 1822 expired records, compressed 0, tir removed 0, deleted 39 expired copies
08/05/2013 12:31:56 - Error nbjm (pid=14292) NBU status: 2074, EMM status: Disk volume is down
08/05/2013 12:31:56 - Error nbjm (pid=14292) NBU status: 2074, EMM status: Disk volume is down
08/05/2013 12:31:56 - requesting resource @aaaac
08/05/2013 12:31:56 - Error nbjm (pid=14292) NBU status: 2027, EMM status: Media server is not active
08/05/2013 12:31:56 - Error nbjm (pid=14292) NBU status: 2027, EMM status: Media server is not active
the requested operation was partially successful  (1)

========================

D:\Program Files\Veritas\NetBackup\bin\admincmd>nbdevquery -listdp -stype PureDi sk -U

Disk Pool Name   : nbu-vmc_dp1

Disk Pool Id     : nbu-vmc_dp1

Disk Type        : PureDisk

Status           : UP

Flag             : Patchwork

Flag             : Visible

Flag             : OpenStorage

Flag             : SingleStorageServer

Flag             : CopyExtents

Flag             : AdminUp

Flag             : InternalUp

Flag             : LifeCycle

Flag             : CapacityMgmt

Flag             : FragmentImages

Flag             : Cpr

Flag             : FT-Transfer

Flag             : OptimizedImage

Raw Size (GB)    : 30147.47

Usable Size (GB) : 30147.47

Num Volumes      : 1

High Watermark   : 98

Low Watermark    : 80

Max IO Streams   : -1

Comment          :

Storage Server   : (Master hostname) (UP)

===================================================

D:\Program Files\Veritas\NetBackup\bin\admincmd>nbdevquery -listdv -stype PureDi sk -U

Disk Pool Name      : nbu-vmc_dp1

Disk Type           : PureDisk

Disk Volume Name    : PureDiskVolume

Disk Media ID       : @aaaab

Total Capacity (GB) : 30147.47

Free Space (GB)     : 16882.37

Use%                : 44

Status              : DOWN

Flag                : ReadOnWrite

Flag                : AdminUp

Flag                : InternalDown

Flag                : ReplicationSource

Num Read Mounts     : 0

Num Write Mounts    : 1

Cur Read Streams    : 0

Cur Write Streams   : 0

Num Repl Sources    : 0

Num Repl Targets    : 1

Replication Target  : (replication master hostname):PureDiskVolume

[bilson123]

Support figured it out. On ver 7.1.0.4 or prior to that the Deduplication engine service on the master was changed to run as domain account with admin rights. I was told that on 7.5 that's a no go. So we took NB down, changed it to Local System and this time disk pool stayed UP. I wish that info was somewhere in the upgrade notes, or on this forum.

[CRZ]

Well, it is now thanks to your post but that wouldn't have helped you earlier this week.  Sorry about that!  I have poked around and found an unpublished TechNote which might have helped you which I have just published:

After Upgrading to netbackup 7.5.0.5, MSDP Disk volume goes Down
 http://symantec.com/docs/TECH205786

It seems a little unusual to have to change permissions on individual NetBackup services from the defaults; any idea how that came about and what the rationale was? 

(pokes around some more)

Could you have experienced THIS problem back in 7.1.0.1 and the configuration change just stuck around through 7.1.0.4 (and 7.5.0.6)?

NetBackup Deduplication Engine (spoold) doesn't start properly after an upgrade to NetBackup 7.1.0.1 on Windows 2008 R2
 http://symantec.com/docs/TECH165758

I do enjoy seeing documents which completely "contradict" each other, but there must have been an issue in 7.1.0.x which was resolved later (looks like 7.1.0.2 from the TN above) but no mention that you would need to undo that original workaround because we would change things in 7.5 and they'd break for you.

I'm very glad it's working now!