Error Code:37 - operation requested by an invalid server

[liverpool2507]

Hello All,

I am trying to run a NFS share backup but it fails with Error code 37: Operation requested by an invalid server.

In the storage unit created, i select media server as my Linux host and when i click on browse button, i can browse the destination in the Linux server (which is the mount point of the filesystem/share created). View properties in the stoarge unit shows proper connectivity.

In the policy, i select, policy type as Standard and On the RHS, i select Foolow NFS. In the Client Selection i select the LInux server(media server) and for backup selection, i browse and select the local data from the same Linux server.

Troubleshooting steps tried:

1. Checked if the media server info is available in the EMM database

….\Veritas\NetBackup\bin\admincmd\nbemmcmd -listhost –verbose

Media server name was available there.

2. In the host properties of the master server,  the media server name is added in Media Servers box.

3. Restarted NBU services.

4. Checked /etc/hosts files and /etc/resolv.conf files and the entries are available and proper over there.

5. Tried to clear up the corrupt lock files (if any) by using the folloing steps:

•  Stopped  all NetBackup daemons/services on the master server
•  While all NetBackup daemons/services on the master server are stopped, rename the bprd.ddirectory to bprd.d_old
•  Recreate a new, empty bprd.d directory on the master server
•  Start the master server NetBackup daemons/services

All the above steps tried did not give any good results and the backup still fails with the same error code.

NBU MAster: Windows 2008 

NBU Version: 7.6.0.1

Media Server: Linux 

Any leads will be appreciated.

Thanks in advance!!

Cheers!!!!

[liverpool2507]

Host entries are proper in both media and master server (entries for self and for each other).

No firewall is running on media server.

is it bpclntcmd or bptestbpcd??

Here is the o/p for bptestbpcd :

C:\Program Files\Veritas\NetBackup\bin\admincmd>bptestbpcd -host d2dww4.usa.hp.com -verbose -debug
06:46:06.875 [21992.15044] <2> bptestbpcd: VERBOSE = 0
06:46:06.882 [21992.15044] <2> vnet_pbxConnect: pbxConnectEx Succeeded
06:46:06.884 [21992.15044] <2> logconnections: BPCD CONNECT FROM 16.78.5.117.56676 TO 16.78.7.145.15
56 fd = 560
06:46:06.896 [21992.15044] <2> vnet_pbxConnect: pbxConnectEx Succeeded
06:46:06.898 [21992.15044] <8> do_pbx_service: [vnet_connect.c:2152] via PBX VNETD CONNECT FROM 16.7
8.5.117.56677 TO 16.78.7.145.1556 fd = 588
06:46:06.899 [21992.15044] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:443] VN_REQUES
T_CONNECT_FORWARD_SOCKET 10 0xa
06:46:06.939 [21992.15044] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:460] ipc_strin
g /usr/openv/var/tmp/vnet-30306428064883528901000000026-jJQ6Xv
06:46:07.536 [21992.15044] <2> bpcr_get_version_rqst: bpcd version: 07600000
1 1 1
16.78.5.117:56676 -> 16.78.7.145:1556
16.78.5.117:56677 -> 16.78.7.145:1556
06:46:07.736 [21992.15044] <2> bpcr_get_peername_rqst: Server peername length = 19
06:46:07.936 [21992.15044] <2> bpcr_get_hostname_rqst: Server hostname length = 17
06:46:08.136 [21992.15044] <2> bpcr_get_clientname_rqst: Server clientname length = 17
06:46:08.336 [21992.15044] <2> bpcr_get_version_rqst: bpcd version: 07600000
06:46:08.536 [21992.15044] <2> bpcr_get_platform_rqst: Server platform length = 17
06:46:08.736 [21992.15044] <2> bpcr_get_version_rqst: bpcd version: 07600000
06:46:08.937 [21992.15044] <2> bpcr_patch_version_rqst: theRest == > <
06:46:08.938 [21992.15044] <2> bpcr_get_version_rqst: bpcd version: 07600000
06:46:09.177 [21992.15044] <2> bpcr_patch_version_rqst: theRest == > <
06:46:09.178 [21992.15044] <2> bpcr_get_version_rqst: bpcd version: 07600000
PEER_NAME = ssdcl498.usa.hp.com
HOST_NAME = d2dww4.usa.hp.com
CLIENT_NAME = d2dww4.usa.hp.com
VERSION = 0x07600000
PLATFORM = linuxR_x86_2.6.18
PATCH_VERSION = 7.6.0.1
SERVER_PATCH_VERSION = 7.6.0.1
MASTER_SERVER = ssdcl498.usa.hp.com
EMM_SERVER = ssdcl498.usa.hp.com
NB_MACHINE_TYPE = MEDIA_SERVER
<2>bptestbpcd: EXIT status = 0
06:46:09.386 [21992.15044] <2> bptestbpcd: EXIT status = 0

here is the o/p of bpps:

[root@d2dww4 bin]# ./bpps -x
NB Processes
------------
root       418     1  0 Apr01 ?        00:00:00 /usr/openv/pdde/pdag/bin/mtstrmd
root       443     1  0 Apr01 ?        00:00:00 /usr/openv/netbackup/bin/bpcompatd
root       457     1  0 Apr01 ?        00:00:02 /usr/openv/netbackup/bin/nbrmms
root       483     1  0 Apr01 ?        00:01:20 /usr/openv/netbackup/bin/nbsl
root       576     1  0 Apr01 ?        00:00:00 /usr/openv/netbackup/bin/nbcssc -a NetBackup
root       591     1  0 Apr01 ?        00:00:01 /usr/openv/netbackup/bin/nbsvcmon
root     30409 32719  0 06:53 ?        00:00:00 /usr/openv/netbackup/bin/bpcd -standalone
root     32716     1  0 Apr01 ?        00:00:00 /usr/openv/netbackup/bin/vnetd -standalone
root     32719     1  0 Apr01 ?        00:00:00 /usr/openv/netbackup/bin/bpcd -standalone
root     32739     1  0 Apr01 ?        00:00:00 /usr/openv/netbackup/bin/nbdisco

MM Processes
------------
root       430     1  0 Apr01 ?        00:00:11 /usr/openv/volmgr/bin/ltid
root       436     1  0 Apr01 ?        00:00:11 vmd

Shared Symantec Processes
-------------------------
root      6885     1  0 Feb12 ?        00:00:05 /opt/VRTSpbx/bin/pbx_exchange

[Marianne]

Yes, bptestbpcd. :) (Long weekend here, so my brain is in holiday mode...) After this there MUST be something in media server's bpcd log. Have you checked? All processes seem right except for Media Manager. If no tape devices configured, only vmd should be running. With tapes, ltid should start more processes. Still something wrong with comms between master and media server. . More output please... On master: nbemmcmd -listhosts -verbose nbemmcmd -getemmserver

[liverpool2507]

No logs are generated in bpcd. is it a file or a folder? well i tried both but no luck :(

O/p of  nbemmcmd -listhosts -verbose:

C:\Program Files\Veritas\NetBackup\bin\admincmd>nbemmcmd -listhosts -verbose
NBEMMCMD, Version: 7.6.0.1
The following hosts were found:
ssdcl498
        MachineName = "ssdcl498"
        FQName = "ssdcl498.usa.hp.com"
        MachineDescription = ""
        MachineNbuType = server (6)
ssdcl498
        ClusterName = ""
        MachineName = "ssdcl498"
        FQName = "ssdcl498.usa.hp.com"
        GlobalDriveSeed = "VEND:#.:PROD:#.:IDX"
        LocalDriveSeed = ""
        MachineDescription = ""
        MachineFlags = 0x17
        MachineNbuType = master (3)
        MachineState = active for tape and disk jobs (14)
        NetBackupVersion = 7.6.0.1 (760100)
        OperatingSystem = windows (11)
        ScanAbility = 5

 d2dww4.usa.hp.com
        ClusterName = ""
        MachineName = "d2dww4.usa.hp.com"
        FQName = "d2dww4.usa.hp.com"
        LocalDriveSeed = ""
        MachineDescription = ""
        MachineFlags = 0x77
        MachineNbuType = media (1)
        MachineState = active for disk jobs (12)
        MasterServerName = "ssdcl498"
        NetBackupVersion = 7.6.0.1 (760100)
        OperatingSystem = linux (16)
        ScanAbility = 5
Command completed successfully.

P.S: have removed the rest of the hosts

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

O/P of nbemmcmd -getemmserver:

C:\Program Files\Veritas\NetBackup\bin\admincmd>nbemmcmd -getemmserver
NBEMMCMD, Version: 7.6.0.1
These hosts were found in this domain: d2dww4.usa.hp.com, ibc01a.usa.hp.com, ibc01b.usa.hp.com, sali
n03.usa.hp.com, ssdcl498, xwin01.usa.hp.com

Checking with the host "d2dww4.usa.hp.com"...
Checking with the host "ibc01a.usa.hp.com"...
Checking with the host "ibc01b.usa.hp.com"...
Checking with the host "salin03.usa.hp.com"...
Checking with the host "ssdcl498"...
Checking with the host "xwin01.usa.hp.com"...

Server Type    Host Version        Host Name                     EMM Server
MEDIA          7.6                 d2dww4.usa.hp.com             ssdcl498.usa.hp.com
MEDIA          7.6                 ibc01a.usa.hp.com             ssdcl498.usa.hp.com
MEDIA          7.6                 ibc01b.usa.hp.com             ssdcl498.usa.hp.com
MEDIA          7.6                 salin03.usa.hp.com            ssdcl498.usa.hp.com
MASTER         7.6                 ssdcl498                      ssdcl498
MEDIA          7.6                 xwin01.usa.hp.com             ssdcl498.usa.hp.com

Command completed successfully.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

if you see my master is showing ssdcl498 instead of ssdcl498.usa.hp.com. Do u think that could be an issue. I really dont think so because the backups are running fine with other media servers.( i have only tried with windows media server though)

[sdo]

The short name should not be a problem - if you have correct DNS suffix config, or hosts file entries with both long and short names.

I'd like to see output of:

bptestnetconn -asp -v6

...from master and media and client.  Can you post?

[Marianne]

"No logs are generated in bpcd. is it a file or a folder? " bpcd folder must be created under /usr/openv/netbackup/logs. Same for bpbrm. All of the output seems fine. We really need bpbrm and bpcd log on the media server after backup attempt.

[liverpool2507]

Hello Experts,

I have found the solution to the issue.:)

There were problems in the configuration.

The master server was responding with the same hostname on 2 subnets so this led me to look at the network properties of the master.

One of the network connections was configured to obtain an address using DHCP.

To attempt to correct this, I disabled DHCP connection and modified the values of the other NIC to reflect membership in *.usa.hp.com domain. 

It was at this point I realized d2dww4, also has two interfaces reporting back from the 16.78.* network.

since having multiple connections is confusing, I disabled one of them and restarted network services on d2dww4.(media server)

After this I tried resolving the master server from d2dww4.usa.hp.com again using bpclntcmd -pn command which returned back the correct name of the master server and the IP addresses of the media server.

Finally after all these changes, i ran a backup and it was successful too .:)

Thanks for all ur posts and suggestion. Much appreciated.

Cherrs!!

[liverpool2507]

Posts are not getting posted immediately.Going in moderation. Even after 5 hours, i still cant see my comment posted.

This is annoying! 

[Marianne]

When your posts are marked for moderation it means that the spam filter saw something in your post to cause this.
Only the Admins receive notification. Since all Admins are in the USA, it may even take longer for Admins to release these posts.
Trusted Advisors can release comments (not new posts).
So, since all of us are on different time zones, you can send a PM to one (or more of us) to notify us.

[Marianne]

Thanks for the feedback!

Log files would have helped us in troubleshooting this....
I can still not believe that no logs were created in log folders.

[liverpool2507]

Sure Marianne! 

will do that henceforth..thanx