Netbackup upgrade and migration to VM

[IBSus]

Hi,

We are currently in the process of decommissioning 2008 servers and it is not the turn of our Netbackup environment.  We currently have:

1 physical master running server 2008 r2
4 physical media servers running server 2008 r2
1 virtual OpsCenter server running 2008 r2

Our current Netbackup version is 7.7.2

They all need migrating to 2016 but to start with we would like to get the master server migrated, and then upgraded to 8.1.2 (or vice versa).

We have tape drives connected to the media servers, I have read the master server cannot be virtualised if tape drives are used in the environment but is that when they are directly connected to the master server?  Our master server doesn't have any tape drives attached so I am hoping we can still virtualise it.  We will keep the physical media servers and upgrade those at a later date.

I am just after some clarification around the tape drives and also, a recommended route to move the master server onto 2016 OS and then upgraded to 8.1.2.

Many thanks!

[Michal_Mikulik1]

Hello,

1/ yes Master Server can be virtualized supposing tapes are not used on it. Tapes on Media Servers can be used.

2/ you must first upgrade to at least NBU 8.0 and then migrate, because 7.7.2 is not supported as a Master on Windows 2016.

Regards

Michal

[IBSus]

Thanks for the info, it's slightly good news I guess!

The problem will be when we need to migrate our media servers to 2016 but we can cross that bridge when we get to it.

[Systems_Team]

Hi IBSus,

An extra bit of info to help plan your upgrade.  Just remember in the order of upgrade, Master is not first - If you have Ops Center (which you do), then the upgrade order is:

1. OpsCenter
2. Master Server(s)
3. Media Server(s)
4. Clients

Hope this helps,

Steve

[IBSus]

Hi Steve,

Thanks for the reply.  We had already upgraded OpsCenter when we attempted an upgrade to 8.1.1 but unfortunately, the master server upgrade failed several times and then also failed to rollback so we had to completely re-install 7.7.2 and restore from catalogue.

We are trying to avoid this happening again and were wondering if there was a way we could build a new master server on our virtual platform and then restore catalogue to it for use?  I guess we are trying to avoid anymore failures and costly fault finding time which we don't have much of between jobs that run throughout the week, however, I am certainly not an expert in Netbackup so would be happy to be told that there is a better/quicker way!

Thanks, appreciate the time :)

Ian

[Marianne]

@IBSus

Have you tried to find the reason for 8.1.1 upgrade failure? 

Do you still have the installation log that you can share here? 

[Systems_Team]

Hi IBSus,

Following on from Marianne's recommendations, here's my 2 cents worth:

I'm assuming you have current support, as you are trying to upgrade to 8.x?  Did you raise any tickets with support during your multiple problem upgrades from 7.7.2?  Also, did you run a catalog consistency check and send the results to support for checking prior to your previous upgrade attempts?

I would recommend opening a case with support prior to your next attempt.  Have them do a health check prior, for your planned upgrade path.  Include in that the catalog consistency check (NBCC).  I would also have the engineer assigned to that case be available during your upgrade, so they can review issues as they happen.

The path you have to follow is a bit of a nasty chicken-egg scenario.  As Michal stated, 7.7.2 is not supported for Master (or Media), only client on Server 2016.  So you need to either attempt the upgrade to 8.x again on your physical server, or build a new Master 2008 R2 (or 2012 R2) VM, and install 7.7.2, recover your catalog and attempt your upgrade again.  Once you are successfully at 8.x then you can do a catalog backup, and build your new 2016 8.x VM master and then recover the 8.x catalog to that.  Painful, I know but I don't see another easier path.

I'm guessing you know all the caveats, but just in case:  NetBackup version and patch level needs to remain the same as when you took your catalog backup (so you can recover it successfully), NetBackup must be installed in the same drive/path, IP address of Master server can change, but hostname of Master must remain the same.  If you want to change the hostname of the Master then you MUST engage third party consultants that are certified in the use of CATMAN tools (Catalog Manipulation tools for NetBackup). 

Hope this helps,

Steve 

[IBSus]

Thanks for the helpful info, much appreciated.

@Marianne  With regard to finding out what went wrong with the upgrade, I did ask support (i spent over 2 hours on the phone to them) and the guy told me unfortunately it is unlikely I would find out now a restore had been done.  I thought it was an odd answer, he basically said that it is what it is, he got me back to a working state and told me to have a support person on the line when I attempt it again.  If you let me know where I could find logs to upload/read then I am happy to do so.

As a side note, we have discovered this morning that none of our tape drive backups worked over the weekend leaving us with a 35TB backlog.  It has taken me all morning to fix but they are now wokring again.  This wasn't discovered after the master server catalogue recovery last week and I am fairly certain it was the reason the robots were offline.

When we have issues with Netapp or VMware, they always ask for log files after and then let us know what happened.

It sounds like it might be easier to attempt another upgrade of the current live master, which I am happy to do with support lined up. 

Many thanks again, my experience with Netbackup before a couple of weeks ago was merely to check our backups had completed on a daily basis, I have had to learn quickly thats for sure! 

Ian

[Marianne]

Installation logs can be found here:

C:\ProgramData\Symantec\NetBackup\InstallLogs\
NetBackup.Install.xxxxxxxx.xxxxxx. log

A common reason for upgrade/install failure is because of incorrect webuser/group config. 

[IBSus]

@Marianne

Thanks for that, I eventually found them and have (hopefully) uploaded it to view.  I think you are right about the webuser/group config, I did notice the install failing at the same point and it was something to do with that.

Hopefully, it is something that can be sorted and I am able to do another upgrade which completes!

[Marianne]

Errors look similar to this : https://www.veritas.com/support/en_US/article.100041276

Be sure to double-check web user/group config before next upgrade attempt: 
https://www.veritas.com/support/en_US/article.000081350

Verify account permissions: 
https://www.veritas.com/support/en_US/article.000125106

Good luck!

[IBSus]

@Marianne

So I did some digging around after looking at your link and found a log file containing this:

-------------------------------------------
12 September 2018 17:19:47
Error 1208: An extended error has occurred.
  Error opening E:\Program Files\Veritas\NetBackup\wmc\bin\install\security_policy_config.db.
----Configuration engine was initialized with one or more errors.----

----Un-initialize configuration engine...

starting at the "wmc" folder level, I am locked out and have no permissions despite me being in the administrators group on the server so it would appear to be a user rights problem.

I would assume that the WMC folder is locked by design?

[Marianne]

Looking at https://www.veritas.com/support/en_US/article.000125568, it seems that   <installPath>NetBackup\wmc\WEBSERVER\logs folder should be accessible.

This doc explains how to edit the security permissions of the folder C:\Program Files\Veritas\NetBackup\wmc and that installation should be done by launching browser.exe using " Run as Administrator". 

Be sure to double-check all requirements in Installation Guide and Upgrade Guide. 

[IBSus]

Thank you Marianne.

I have read so much, including the guide and I even made notes but I still have come across things that I feel like I should already be aware of. 

This has certainly been a learning curve and I feel like I am in a better position to support the upgrade going forward.

Thanks for the help, will let you know how the next attempt goes!

[quebek]

Hi 

When I was upgrading to 8.1 most of it failed on the secedit command which is called by some batch file from NBU installer. In order to avoid it such procedure was developed:

1. Create a test folder on c: drive

   mkdir c:\test
   cd /d c:\test

   Export current setting of security user/groups by executing the following

   secedit /export /cfg security_policy_config.db /verbose /log export.log
   secedit /export /cfg user_rights_conf.db /areas USER_RIGHTS /quiet /verbose /log userexport.txt
   notepad user_rights_conf.db

   In this notepad search for these privileges: SeServiceLogonRight; SeTcbPrivilege and SeAssignPrimaryTokenPrivilege - at the end of each add such users: nbwebsvc, nbwebgrp. Than run this

   1. secedit /configure /db security_policy_config.db /cfg user_rights_conf.db /overwrite /areas USER_RIGHTS /verbose /log merge.txt

      Review the merge.txt file with notepad. If there was a failure check for a USER IDs and remove these from this file user_rights_conf.db Example failures:

      	Configure S-1-5-6.
      	Configure Virtual Machines.
      Error 1332: No mapping between account names and security IDs was done.
       	Cannot find Virtual Machines.	Configure NT SERVICE\NT SERVICE.
      Error 1332: No mapping between account names and security IDs was done.
       	Cannot find NT SERVICE\NT SERVICE.

      So now you have to open this file user_rights_conf.db and remove these two users from it: NT SERVICE\NT SERVICE and Virtual Machines

      notepad user_rights_conf.db

      and rerun the last command

      secedit /configure /db security_policy_config.db /cfg user_rights_conf.db /overwrite /areas USER_RIGHTS /verbose /log merge.txt

      Again check the merge.log to see if there are no errors, if no continue with upgrade immediatelly as the gpupdate can overwrite those setting and if you will be unlucky the upgrade will fail due to this on step SetupWMC.bat.

[IBSus]

Thanks for the help!

One last question - I am going to attempt the upgrade tomorrow morning and was just downloading the 8.1.2 files ready for it, I noticed there were no Live update files like there were for 8.1.1, is there a reason for this?  Have they been renamed to something else?

In the 8.1.1 downloads they are called:
8.1.1_LU_1of2.tar.gz
8.1.1_LU_2of2.tar.gz

Any idea's?

Thanks,
Ian

[quebek]

Hey

The below is taken from release notes for 8.1.2 - I am hoping you read it ... ;)

VxUpdate replaces LiveUpdate
VxUpdate replaces LiveUpdate in NetBackup 8.1.2. With this release of VxUpdate,you can now:
■ Upgrade any client platform from any master server platform.
■ Install client EEBs.
■ Schedule upgrades, pre-check, and staging (download packages) separately.
■ Run ad-hoc upgrades using the GUI or from the client itself.
■ Track upgrades through parent and child jobs in the Activity Monitor. VxUpdate also includes a package repository on the master server and improved signing and verification of packages.
For more information about VxUpdate, see the NetBackup Upgrade Guide.

[IBSus]

oh dear ...

I have read so much, but I keep being told about things which I have missed/not read... I think I need to go away and read more, everytime I think I have read enough and got the gist of it, someone points out something else I should have read.

I have the install guide, which I am guessing doesn't contain the info you have pointed out! It's no excuse but I am fairly new to the deeper workings of Netbackup and upgrades, our usual guy is on another project so I have taken over looking after NB.  I will take a look at the release notes document now just to ensure I don' miss another important point!

I am learning as I go though, and appreciate everyone's help, thank you! 

Ian

[quebek]

understood...

than take a look in here 

https://www.veritas.com/support/en_US/article.100040135

and

https://sort.veritas.com/netbackup

and

https://sort.veritas.com/checklist/install

[IBSus]

Hi,

I have read all the stuff given to me, ensured all requirements are met for the nbwebsbc and nbwebgrp accounts.

It was mentioned above that I run browser.exe as admin, which I did but when I then click on install for the netbackup master server I get the error attached.

Is it something to do with the fact we have our install on a different drive to the default?  

Sorry for all the questions, I really want this next attempt to work!