NBU 7.6 Upgrade Issues (from 7.1.0.4)

[DaveM16]

Hi

I running a test bed under VMware workstation to replicate the production environment running at the site I'm working on. This is currently running a SFWHA/VCS clustered NetBackup master server on 7.1.0.4 (SFWHA is at version 5.1 SP2).

I am trying to upgrade the master and as per the upgrade guide offline the VCS NetBackup resource. Then run the setup from the instalation media (I am still using the 7.6FA release, but from what I've read, the 7.6GA release doesn't have anything relevant to this issue - I'm downloading that now, and will try once done - just in case).

Anyway the upgrade proceeds normally on the "active" node and completes without error. The problem is with the passive node. When the instalation software starts the process it runs a check on each node - nbcheck. This fails with a complating about the EMM server not being supported on a remote serverunder 7.6. The output from the nbcheck utility folloows:

ok nbdb_maintenance_space: not an EMM server: skipping
not ok remote_emm: As of NetBackup 7.6, Symantec no longer supports
  NetBackup configurations in which the Master Server and the EMM Server
  are on different hosts.  See

      https://www.symantec.com/docs/TECH206478

  for more information.

ok nbdb_ntfs_dir_symlink: no problematic symbolic links were found

Now as this is the passive node - the EMM server is remote but as it is clustered this should not matter.

I've tried upgrading one node at a time but the test fails in the same way (I didn't fail over to the passive node as the EMM database has been upgraded, so not sure what the effect will be).

I'm stuck at present - am am in discussion with Symantec support, but was wondering if anyone had seen anything similar and how they managed to get around it.

I've attached some of the logs and output from various comands run on each node in the cluster. FYI the primary node is syd02s008npa1, the passive node is syd2s008npa2 and the cluster name is syd02s008nca1.

Thanks

[Marianne]

Do you have any NB config output as it was before the upgrade? (nbemmcmd -listhosts -verbose, main.cf, bp.conf on both nodes, etc)

Have you confirmed that rsh is enabled before starting upgrade of passive node?
This is crucial as the 2nd node needs to communicate with the active node for cluster and EMM config.

What is status of config on active node? 
hastatus -sum
nbemmcmd -listhosts -verbose

What is contents of NBU_RSP on both nodes? (in /usr/openv/netbackup/bin/cluster)

I have seen in the past how rsh not being enabled caused node 2 upgrade to fail and effectively turned the cluster into a one-node cluster again. 

We (customer actually) have only been able to fixed it with a call to Support.

*** EDIT *** 

OH DEAR!!!

Did not read properly....  My answer is for Unix/Linux cluster. 
Only noticed now that you have a Windows cluster.

Please ignore everything except recommendation to contact Support!

[RiaanBadenhorst]

Hi,

I don't access to the install media but can you get a hold of the script which is reporting the issue,remote_emm.

On unix you could probably easily find it, maybe not with Windows.

If you're able to we can probably see what its doing and change it, but ultimately support would need to fix it (if its dodgy)

On a certification side 7.6 is not supported on SFWHA yet, it will be in May when SFWHA 6.1 is released.

On a certification side 7.6 is not supported on SFHA yet, don't know when it will be.

[DaveM16]

Riann

SORT indicates that SFWHA 5.1SP2 (which is what is instaled) is supported on a 7.6 master - where are you seeing otherwise? I also tried looking (using strings) at the nbcheck.exe utility without anything useful coming from it.

Marianne

Thanks for your input - yes it is windows (sigh). If it was Unix, I would be doing the upgrade one node at a time. This doesn't work for Windows (I have tried and the instalation fails with the same error).

The only thing I have not tried, is to fail the cluster over after the initial installation - but there is nothing in the manuals to indicate this is required.

I think support will have to figure out what's gone wrong.

I was wondering whether anyone else had run into this issue already, or whether (preferably) someone has successufully upgraded a Windows cluster and how they did this. I'm sure I can't be the first person to try this type of upgrade.

Cheers all.

[RiaanBadenhorst]

Hi,

Ok I'm looking here

http://www.symantec.com/docs/TECH138722  <<<5.1 SP1

http://www.symantec.com/docs/TECH153742  <<<6.0.1

The Netbackup cluster compat list says "Supported since 7.1" which to me is very vague. These are internal product (to Symantec) compatibilties and should be kept up to date better.

I therefore checked with an Symantec SE and was told its only supported in 6.1

[Marianne]

The only thing I have not tried, is to fail the cluster over after the initial installation - but there is nothing in the manuals to indicate this is required.

My experience with clustered master on Unix/Linux is that EMM is only properly updated after failover to 2nd node.

My personal experience:

Both nodes in cluster, active on node1. But look at this:

# /usr/openv/netbackup/bin/admincmd/nbemmcmd -listhosts

NBEMMCMD, Version:7.1

The following hosts were found:

server             nbumas

cluster            nbumas

master             mvdb-lnx1

Command completed successfully.

Shows only one node… Offline, online, still the same output. 

Failover.

Only NOW is emm updated with 2nd node!

# /usr/openv/netbackup/bin/admincmd/nbemmcmd -listhosts

NBEMMCMD, Version:7.1

The following hosts were found:

server             nbumas

cluster            nbumas

master             mvdb-lnx1

master             mvdb-lnx2

Command completed successfully.

[DaveM16]

Riann

The SE I have been working with does not see an issue with SFWHA 5.1 SP2 (it has been supported since NBU 7.1 and support hasn't been dropped).

-------------------

The 7.6GA release made no difference with the install. I'm currently trying a two stage upgrade

1. 7.1.0.4 -> 7.5.0.6
2. 7.5.0.6 -> 7.6.0.1

I'll post the result

[RiaanBadenhorst]

David,

I had not seen you're from Sym. I'm also not of the opinion that it would be an issue, but we live on the other side of the wall so to speak, so we have to abide by these rules becuase at the first sign of issues, support will refer us to the list. And then say sorry, its not supported.

The lists, in my opinion, are seriously out of date and in need of attention.

[RiaanBadenhorst]

Hi,

What happened here?

[DaveM16]

Hi Riann

Still waiting for a definitive answer from support.

That said I was able to complete the upgrade on the passive node by making sure that node could not connect to the Internet (and SORT.symantec.com). It appears that the installation pre check (nbcheck.exe) is making some type of query and update to/from SORT. When it cannot connect, it does not seem to run the remote_emm check during this pre-installation check (as per the installation log).

Once the case is resolved, I'll post an update.

Cheers
David

[RiaanBadenhorst]

Excellent, thanks.

[DaveM16]

Hi All

I have recently been made aware of this http://www.symantec.com/docs/TECH214320 which describes the failure I am seeing. This refers to MSCS rather than VCS/SFWHA but I am seeing the exact same problem.

It decribes the use of a new environment variable to override the check failure to use on the passive node (NBPREINSTALL_CRITICAL_OVERRIDE) which allows the installation to continue on the passive node if the check fails - see the technote to describe where and when to use this variable.

I have referred support to the similarity with my issue to have them include VCS in the technote.

I am also highlighting to them that although this addresses the immediate issue, it does introduce a potential problem in that any other errors encountered during the prechecks will also be ignored. The problem is with the nbcheck utility - it is broken IMHO and needs to be fixed (maybe something for 7.6.0.2)

Once I have an answer from support about the these issues, I'll either make this as the solution or provide a further update.

Cheers
David

[DaveM16 · Accepted solution]

One further update to the issue from one of the software engineers.

"You're correct about not running the remote_emm test if there's no access to SORT. This was by design and technically, it's still possible at 7.6 to run with remote/shared EMM, we just want to discourage it. The scenario described here and in the TechNote is something we are actually trying to fix right now. Once we have a fix, we'll update the content at SORT and the issue you are seeing will disappear. "

So the solution to the issue is either block access to SORT (for now at least), or use the environment variable as mentioned in the technote.

[Marianne]

Oh Dear! Sounds messy.........

Wonder how many users with Clustered master servers will run into this issue....
(I need to bookmark this post!)

[Michael_G_Ander]

Good thing I saw this post before upgrading to 7.6, it seems to me that Symantec once again have forgotten us with clustered master when implementing changes. It is a little strange as cluster master is one of Symantecs recommendation for Disaster Recovery

[Marianne]

One of my pet hates is hearing "This was by design ....". 
Meaning engineering is under no pressure to fix it any time soon.........

And NO warning or note in the Clustered master server Guide or Installation Guide or Release Notes or 7.6 LBN!

OH! Wait! NBU 7.6 LBN has been updated a couple of days ago:

(NEW) (ET3416670) The passive node of an MSCS Master Server Cluster fails to upgrade to 7.6.0.1

 http://www.symantec.com/docs/TECH214320