Clustered Master server shows "The host is not an active node of a cluster" when do backup

[tigerlover]

Does anybody know what caused below error on a Clustered Master server ( Solaris 10, NBU 7.1):

 06/21/2012 16:23:44 - Info nbjm (pid=28389) starting backup job
 (jobid=204) for client mgifxx18-nbu, policy test20, schedule Full
 06/21/2012 16:23:44 - Info nbjm (pid=28389) requesting STANDARD_RESOURCE
 resources from RB for backup job (jobid=204, request id:{FFECA8CE-BBDE-
 11E1-B8C2-002128A4683C})
 06/21/2012 16:23:44 - requesting resource mgifxx20-hcart-robot-tld-0
 06/21/2012 16:23:44 - requesting resource pdnbxx01-nbu.NBU_CLIENT.MAXJOBS.
 mgifap18-nbu
 06/21/2012 16:23:44 - requesting resource pdnbxx01-nbu.NBU_POLICY.MAXJOBS.
 test20
06/21/2012 16:23:44 - Error nbjm (pid=28389) NBU status: 2061, EMM status:
 The host is not an active node of a cluster
 The host is not an active node of a cluster  (2061)

My cluster is not doing any failover when I start to run the backup policy and emmcmd info as below:

nbemmcmd -listhosts
NBEMMCMD, Version:7.1
The following hosts were found:
server             pdnbxx01-nbu.sdps.org
master             pdnbxx01-nbu.sdps.org
master             mgifxx19
cluster            pdnbxx01-nbu
master             mgifxx18
media              mgifxx20
media              mgifxx20-nbu
Command completed successfully.

Thanks in advance for your help!

[mph999]

At first glance it looks like the cluster is configured incorrectly.

Your server

nbemmcmd -listhosts
NBEMMCMD, Version:7.1
The following hosts were found:
server             pdnbxx01-nbu.sdps.org
master             pdnbxx01-nbu.sdps.org
master             mgifxx19
cluster            pdnbxx01-nbu
master             mgifxx18
 

Test server

rdgv240sol21 # nbemmcmd -listhosts

NBEMMCMD, Version:7.0.1

The following hosts were found:

server          rdgv21-22

app_cluster     app-cluster-test

ndmp            rdgv21-22

ndmp            rdgf270c-01

cluster         rdgv21-22

media           qtpdmedia

master          rdgv240sol22

master          rdgv240sol21

Command completed successfully.

Your two nodes are listed as masters - correct

My two nodes are listed as master also - correct

My cluster name (rdgv21-22) is not listed as a master (as the virtual node name is not a physical node)

Your cluster name (rdgv21-22) however, is listed as a master - which I think is incorrect.

If I am right, then I suspect this might have something to do with the issue.

Martin

[tigerlover]

I also noticed that problem, but I couldn't find any way to delete that Virtural name as Master server and I am not very sure if this is the real cause of the error since that virtrual name can also be treated as Master server name. right?

[Marianne]

This is what output looks like when master is clustered correctly:

# /usr/openv/netbackup/bin/admincmd/nbemmcmd -listhosts

NBEMMCMD, Version:7.1

The following hosts were found:
server             nbumas
cluster            nbumas
master             mvdb-node1
master             mvdb-node2
Command completed successfully.

nbumas is the virtual name
mvdb-node1 and mvdb-node2 are physical node names

I have also compared this with the previous cluster installations that I have done for customers - all output look like this:

server              virtual-name
cluster             virtual-name
master             node1-hostname
master             node2-hostname

Is this a new installation?
I cannot see how your installation could possibly be fuctional the way it looks...

Also never a good idea to configure NBU with a mix of shortnames and FQDN - one or the other is fine.
I prefer to configure all shortnames and add aliases for FQDN.

If this is a new installation, best to start from scratch....

[mph999]

Thanks for confirming that Marianne - I was just on my way out, so it was a very quick look.

Something along these lines will probably remove the node ...

nbemmcmd -deletehost -machinetype master -machinename pdnbxx01-nbu.sdps.org

If I've missed an option, when you run the command, it will tell you what you are missing ..

NOTE: I've never done this on a cluster, I believe removing the node will be fine, but some cluster config files might need changing (cd.conf from memory).

The best thing is to start over if possible, to get rid of that FQDN - I don't like a mix, yes it should work, but it's not really very neat.  My preference is shortname personally.

Martin

[Marianne]

@tigerlover - please give us more info about your setup?

Is this a new cluster installation that has never work before?
Or only on one node?

Please post installation logs of both nodes as attachments. You can find it in /usr/openv/tmp. look for install_trace.####.
Please rename them to install_trace-node1.txt and install_trace-node2.txt before uploading.

[tigerlover]

I called Symantec Support and they spend 4 and a half hour on the phone with me but couldn't get rid of that FQDN, So they sugguested me to remove NBU and reinstall. I did remove all pkgs related with NBU and tried to reinstall it, but right now it seem the node I am installing doesn't recognize another node as a part of cluster, I put the Virtral name as the Netbackup Server name but it didn't ask me for any other questions related with Cluster ( Not like when I installed it the first time ).

From the system level I can failover the clustered service group, but VOM shows my another node not reported to the cluster. I called Symantec support again, they suspect VOM reporting wrong message.

Do I need to change any configure file after deleted NBU before reinstall? Did anybody had the same problem? I have no clue what to check now.

Thanks in advance for your help!