NetBackup 7 VCS 5.1 .. not happy with each other.

[MILB65]

All

TIA for your response and assistance.

My environment:
NetBackup 7
VCS 5mp1
Solaris 10

Problem is the following:
we are not able to get VCS to failover properly from one node to the other.  The process starts, but NBU daemons never start.  No errors in the engine_A log, no hint as to why it fails to start.

Has anyone seen this before?

[Dev_Roy]

Try following to isolate the issue:
1.> Freeze the netbackup service group.
2.> Start netbackup manually outside of VCS.
3.> Check if all relevant processes for netbackup has started. If it does not starts then it is the issue with the netbackup installtion and integration with
VCS.

If netbackup does not starts correctly outside of VCS then you may have to consider reinstall and run custer_config script once again ( I assume this is a new setup). Check PBX is also running fine. Could you please let me know the steps that you followed while running cluster_config. Make sure that you are not using FQDN for Netbaackup Master Server, I remember some issue with resolution while providing FQDN despite of a good DNS but that was again with older version of Netbackup. It would be good if you provide us the screen shots of all the questions / answers while you are running cluster_config and provide us with the main.cf file. If the issue is with netbackup then engine_A.log file alone would not give any pointer except like "service went offline on its own- not initiated by VCS" or so. Also check your bp.conf file and try to provide both the nodes name in the server list. Make sure that you have kept your catalog database in a shared storage. Let me know how it goes after trying out  aforesaid.

Regards,
Dev

[Marianne]

I'd like to know more about your installation/configuration. I've got it working 100% in a lab environment and it's also working fine at at least one of our customers with these exact software versions installed. (We have many other clusters at customers with different combinations of SF/HA and NBU.) Was the installation/configuration carried out as per the NBU HA Guide? Can you confirm that the NetBackup agent is running on both nodes?

You say 'failover does not happen properly'. Are you referring to a Service Group switch or failover in the event of a failure?
You are also saying that 'The process starts, but NBU daemons never start.' Can you see in engine_A log that a start of nbu resource is attempted?

NBU7 no longer requires separate 'cluster_config'. When I first installed NBU 7 master on the 1st cluster node, the installation process realized that the node was part of a cluster and started prompting for cluster info. I was surprised to see that it never asked for a second node name and that it configured a one-node cluster. When we installed the 2nd node, it asked if it should join the existing cluster.

This new behaviour is described as follows in the NBU HA Guide:
4 When NetBackup is installed on the clustered setup a confirmation prompt is displayed.
■ When installing NetBackup on the first node, you are prompted to confirm creation of a NetBackup cluster. Type Yes to set up NetBackup in HA mode.
■ When installing NetBackup on the subsequent nodes, information of already created NetBackup cluster group is displayed. You are prompted to join the group.
5 When you are prompted for cluster specific configuration details, refer to the checklist and provide details  accordingly.
6 Allow NetBackup to be installed in a cluster. When a NetBackup failover server is installed:
■ On the first node, a single node cluster resource group for NetBackup is created and brought online.
■ On the other nodes, the installed node is added to the cluster resource group.


I was somehow surprised/disappointed to see that the NBU 7 config STILL requires rsh connectivity (still no ssh option).

I agree with Dev - I'd like to see the NBU installation log on both nodes (can be found in /usr/openv/tmp).

[alexleikin]

Hi,

I am trying to find some documentation about how to add 3rd node to fail over VCS cluster

for NetBackup. I added the node to ClusterServices group, but what is the best way

to make it visible to NetBackup. I can manually change main.cf and copy over all

/usr/openv/netbackup to the third node, create required links and mount point and it should work, but

hopefully there is better way to do it.

Thanks in advance,

Alex

[Marianne]

There is simply NO documentation...

*****EDIT*****

 see Kimberley's post below for TN.

*************************************

I have personally gone through a long, painful exercise to add a node to an existing cluster. Because cluster_config can ONLY be done on new installations, everything must be done by hand... Copy NBU agent across, create sym links, use 'ha' commands to add new node to cluster and SG, followed by a l-o-n-g nbemmcmd command to add the new node to existing NBU config.

So, just when I thought it was time to write an article to document the process, NBU 7 came out! If the new node is already part of a cluster, the NBU installation will prompt to join existing cluster.

My advice: Upgrade to NBU 7 on existing cluster, then install the new node. PLEASE ensure that rsh is enabled in all directions between all nodes (existing and new) before you start installation of the new node.

[Kimberley · Accepted solution]

I forwarded this thread to the product PM team for both VCS and NBU as thought they'd be interested in your input here. They've made a documentation request and also had a suggestion re:

"A scripted process (cluster_add_node) was added for Unix/Linux in NetBackup 6.5 and significantly improved.  As far as the script is concerned it’s actually called ‘cluster_add_node’ and should be in /usr/openv/netbackup/bin/cluster/util.  The tech note about it is http://www.symantec.com/business/support/index?page=content&id=TECH62860 – this relates to the original version, I can’t find anything about the enhancements in 7.0.

For Windows NetBackup 7.1 (now in beta testing) has an improved cluster installation wizard which performs sanity checks on the underlying VCS setup and also includes an ‘add node’ option. 

In NetBackup 7.0.1 we separated clustering of master servers from the general aspects of HA and created a separate “Master Server Clustering Guide.”

Hope this is useful.

Best,

Kimberley

[Marianne]

Extremely useful, Kimberley! Thanks for the TN - I had no idea that there was such a doc!

I will also find the “Master Server Clustering Guide” right away since I need to replace an cluster node next week!

[Kimberley]

Here's some more info that PM thought would be useful, in terms of a snapshot of the current and previous releases and what has changed.

Pre-7:

a)      Windows: Installing on a node in cluster provides the option for add node to an existing resource group (provided one exists)

b)      UNIX: Undocumented script (cluster_add_node) and tech note.

NetBackup 7 and onwards:

a)      Windows: Same as before. With the upcoming release, more validation before installation.

b)      UNIX: Installing on a node in cluster provides the option for add node to an existing resource group. Behaviorally similar to the windows case. Additionally more validation (cluster resource and dependencies) before installation starting 7.

c)       UNIX: enhanced logging (/usr/openv/netbackup/logs/cluster) with a configurable debug log level (part of the documentation).

Agents:

The VCS agent has been modified to add a few enhancements and logging capability, but the core functionality still remains the same.