Netbackup - Remote Admin Console EMM Errors

[SYMAJ]

I have a NBU installation with the master server on W2K8R2, 5230 Appliances as Media Servers - and am just adding a number of RAC's.

NBU Master is 7506, RAC is 7506.

I have installed the RAC software onto the Windows PC's, added the Master & Media server names to the RAC Server list, added the RAC PC names to the Master Server 'additional server' lists - but am getting EMM 92 and EMM 77 errors.

I can start the RAC on the PC and all seems to connect and startup OK.  When I select the Activity Monitor screen it hangs for a while and then I get the EMM 92 error.  It then goes on to display the jobs OK but no Topology picture.  If I select Device Monitor, again it hangs then I get the EMM 77 error.

I can run BPCLNTCMD from the Master to the RAC OK, and vice versa.  In the nbemm log on the master server I am seeing a number of potential issues:

1. it states that peeraddr a.b.c.d is not a Netbackup Server  (a.b.c.d is IP address of the RAC PC)

2. it states that 'ServiceContex credentials not found'

I am launching the RAC 'as administrator' on the RAC, I have the NBU services on the RAC running as the same account as the services on the Master server.

I noted from an old technote that there is a registry entry for EMMSERVER (although it may be outdated now), and this wasn't there on my RAC PC.  I added this with the Master Server name as the entry but no difference.

I am really confused (and frustrated) at this point.........

Any pointers ?

AJ.

[Marianne]

RAC pc's do not have registry entry for EMM server. 

My experience has always been that bpdbm needs to be restarted on master server to recognize new Server entries.

[Mark_Solutions]

As Marianne says you need a service re-start on the Master to register your PCs credentials

You also should not need the NetBackup services on your PC setting to anything other than the local system account

Finally, you do need to be a local admin on your PC for it to work correctly

Hope this helps

[SYMAJ]

Marianne,

Thanks for that info on the EMMSERVER key.  It is my experience also that a service restart is required on the Master after adding servers, and I did this after adding the new RAC servers.

Is there a case sensitivity issue here ?  Just in case I did add the RAC servers with FQDN and shortname, in both upper and lower case - but still no joy !

I am concerned about the entry in nbemm stating that my RAC server IP address is not a Netbackup Server.

AJ

[Mark_Solutions]

There can be a FQDN issue - worth adding your PC in both short and FQDN to the Master Servers Hosts file and vice versa (Master in your PCs Host file)

[Marianne]

Use 'bpclntcmd -ip 'RAC-pc-IP' to see how the master is resolving incoming request from PC.

Server entry needs to match resolved hostname exactly. 
NBU is case sensitive - even on Windows.

[SYMAJ]

One more thing to throw into the mix here is that the RAC PC has two interfaces - one on the 1.2.x.x network (which is where the Netbackup servers are) and one on the 172.22.x.x network (which is where all the production servers and PC's are).  We have a firewall between the two networks which allows ALL traffic from the netbackup servers to go anywhere, and a rule which allows specifc addresses to get back to the netbackup servers.

When I run BPCLNTCMD from the RAC PC to the master server I see the master server OK on the 1.2.x.x network, for both -hn and -ip.

When I run BPCLNTCMD from the master server to the RAC PC I see the RAC PC on the 172.22.x.x network.

I can see the client properties of the RAC PC (and amend them) from the master server OK, and as a test I ran a backup of the RAC PC and that is all working OK.

When I run the BPCLNTCMD from the master to the RAC PC the FQDN comes back all in lower case - this is included in the additional servers list on the master server and ALL services have been restarted.

I am puzzled that the RAC seems to 'sort of work' - once I get the hanging and the EMM errors out of the way I appear to be able to do everything excepting I get no topology picture on the Activity Monitor screen.

I have NO host file entries anywhere, and am relying solely on DNS (which appears to be resolving OK).

Still confused and no further........

AJ

[Marianne]

Are there different hostnames on DNS for the 2 interfaces on the RAC pc's?

Each IP address should have a different hostname. 
 

[Mark_Solutions · Accepted solution]

Sounds like Hosts files should sort this out for you but try a ping -a against the 1.2.x.x. address of the PC to see what name it resolves as on that network - you may need to add that as a valid server instead

[SYMAJ]

OK - update as follows:

If I disable the 1.2.x.x network card in the RAC PC (and the firewall is allowing any traffic from the RAC PC to reach the NBU servers) all works fine !

I then put a host file entry into the Master server telling it that the RAC PC (both FQDN and Shortname) is on the 1.2.x.x address (as opposed to the DNS address for this machine which is 172.22.x.x), enabled the 1.2.x.x card again in the RAC PC - and back to square one - failing.  It appears that the DNS

My thoughts were that DNS is telling the RAC PC that the master server is on 1.2.x.x, so it is routing traffic down the 1.2.x.x interface card - so if I could get the master to talk to the RAC PC on the same card (as opposed to the DNS entry of 172.22.x.x) it should all work.  But I must be missing soomething here.

I also then put a host file entry into the master server for the RAC PC, specifying FQDN & Shortname to the 1.2.x.x address, followed by a BPCLNTCMD -clear_host_cache.  I thought this should force all the traffic from netbackup to the RAC PC through the 1.2.x.x address - but I still get the failure.

My current thoughts are relating to the PREFERRED NETWORK parameter.  Will this help me out here ?  If I set NBU (either on the client or the master properties) to have the ONLY setting for the source / target addresses will this work and force the traffic down the specific interface ?

I am not sure whether I specify this on the master or the client properties, and what entries I should have in the source and target fields.  Can anyone advise ?

Thanks,

AJ.

[Mark_Solutions]

As your Master also has 2 interfaces i assume that also has two different host names

You need to follow this route ...

1. Look at what you master server name is as far a NetBackup is concerned - nbemmcmd -listhosts

2. That will decide which network you should use - make sure that is the master server name used by your PC

3. See what the name of your PC is on that network and make sure your PC's NetBackup client name uses that name (as it may have been detected by the installer as the other name), if not rename it and re-start its services

4. Add that name to the server list on the Master

At this point the PC should resolve, via DNS, the Master across the correct network using the correct name and vice versa and the PC will also have it own name correct, as will the Master

Hope this makes sense

[Marianne]

Please remember to do 'bpclntcmd -clear_host_cache' on master and RAC pc's when any network-related change is made. 

Host cache is normally the reason for failures shortly after changes. NBU refreshes only once per hour.

[SYMAJ]

Mark,

The Master has only 1 interface - the 1.2.x.x

RAC PC has 2 interfaces, 172.22.x.x and 1.2.x.x

Firewall routes traffic between the 2 networks, allowing any of the Netbackup Servers to go anywhere on the 172.22.x.x network and specific servers on the 172.22.x.x to go to the Netbackup servers.

DNS only has 1 entry for the RAC PC, and that is its 172.22.x.x address.

DNS only has 1 entry for the Master server (as it only has one address anyhow), and that is the 1.2.x.x address.

Marianne,

I am doing a -clear_host_cache on the master each time, but unsure as to how often I have done it on the RAC PC.  I will ensure it is done at both ends now and try again.  Just to be sure I will also restart NBU services everywhere again at the same time.

I am assuming no interaction needed with the Media Servers at this stage ?

Thanks for all the continued input.

AJ.

[SYMAJ · Accepted solution]

OK - problem now resolved.  (fell at the last hurdle due to fat-fingers when  editing the host file on the master server).

Summary:

Add the 1.2.x.x address for each of the RAC PC's to the host file on the master server with both the FQDN and the Shortname as returned by the ping -a 1.2.x.x command issued against each of the RAC PC's to override DNS and force the traffic down the 1.2.x.x interface of the RAC PC's. 

Add the name (as returned by the ping -a 1.2.x.x command issued against each of the RAC PC's) to the Additional Servers on the Master Server Properties.

Removed the EMMSERVER entry from the registry on the RAC PC (added for PD)

Reverted the NBU services on the RAC PC to run under Local Service Account

All works OK.

Many thanks for all the input.

AJ

[Mark_Solutions]

Good stuff - glad you are sorted