HP UX File System Backup Failed with Error 49

[AQasir]

- My Netbackup Master server is running NBU 6.0 MP6 and Windows 2003 EE R2.

- One of my media servers is a HP UX 9000 RP 4440 running NBU 6.0 MB6, HP-UX 11.11 and Oracle 9i release II.

For several months we have had good backups and the setup/configuration has not changed. But this past weekend the HP-UX File system policy failed due to an "error 49 : The client failed to start up correctly"

I bounced the Netbackup services on the HP-UX as well as on the master server but the same error was generated when I kicked of the policy again.

Any ideas where I should start investigating this error. I am currently in the process of creating the bpcd and bpkar debug log directories and kicking off the job once more.

[AQasir]

The following is a sub-set of data from my bpcd log file. I did not include the entire log since it is too large and I hope this portion of the log file shows enough information to diagnose the problem.

Note that IP addresses and host names were changed to protect confidential data.

12:57:20.109 [28311] <2> bpcd main: BPCD_READ_USERS_CONFIG_RQST
12:57:20.328 [28311] <2> bpcd main: BPCD_GET_VERSION_RQST
12:57:20.547 [28311] <2> bpcd main: BPCD_READ_HOST_CONFIG_RQST
12:57:20.765 [28311] <2> bpcd main: BPCD_DISCONNECT_RQST
12:57:20.765 [28311] <2> bpcd exit_bpcd: exit status 0  ----------->exiting
12:57:21.790 [28318] <2> bpcd main: offset to GMT 18000
12:57:21.790 [28318] <2> logconnections: BPCD ACCEPT FROM 169.xxx.xxx.xxx.63016 TO 169.xxx.xxx.xxx.13724
12:57:21.790 [28318] <2> bpcd main: setup_sockopts complete
12:57:21.807 [28318] <2> print_arp_info: MediaServer.Short.Name (169.xxx.xxx.xxx) at 0:30:6e:ea:56:55 ether permanent published
12:57:21.808 [28318] <2> bpcd peer_hostname: Connection from host MediaServer.Short.Name (169.xxx.xxx.xxx) port 63016
12:57:21.816 [28318] <2> bpcd valid_server: comparing MasterServer.fully.qualified.domain.name and MediaServer.Short.Name
12:57:21.833 [28318] <2> bpcd valid_server: comparing MediaServer.Short.Name and MediaServer.Short.Name
12:57:21.833 [28318] <4> bpcd valid_server: hostname comparison succeeded
12:57:21.834 [28318] <2> bpcd main: output socket port number = 1
12:57:21.997 [28318] <2> bpcd main: Duplicated vnetd socket on stderr
12:57:21.997 [28318] <2> bpcd main: <---- NetBackup 6.0 0 ------------initiated
12:57:21.997 [28318] <2> bpcd main: Not using VxSS authentication with MediaServer.Short.Name
12:57:22.066 [28318] <2> bpcd main: BPCD_GET_STDOUT_HOST_SOCKET_RQST
12:57:22.135 [28318] <2> bpcd main: socket port number = 1
12:57:22.136 [28318] <2> bpcd main: Connected on output socket
12:57:22.137 [28318] <2> bpcd main: Skipping shutdown of send side of stdout.
12:57:22.137 [28318] <2> bpcd main: Duplicated socket on stdout
12:57:22.137 [28318] <2> bpcd main: BPCD_GET_VERSION_RQST
12:57:22.205 [28318] <2> bpcd main: BPCD_GET_PLATFORM_RQST
12:57:22.276 [28318] <2> bpcd main: BPCD_FORK_CMD_RQST
12:57:22.345 [28318] <2> bpcd main: fork cmd = /usr/openv/netbackup/bin/bpfilter bpfilter -r 2147483647 -ru root -dt 0 -to 0 -clnt MediaServer.Short.Name -class MediaServer.Short.Name_HPUX_FSBACKUP_FULL -sched WEEKLY_FULL -st FULL -bpstart_to 600 -bpend_to 960 -read_to 1200 -stream_count 28 -stream_number 28 -jobgrpid 117842 -use_otm -use_ofb -b MediaServer.Short.Name_1235411731 -fso -F encipher
12:57:22.345 [28318] <2> bpcd main: filter type = 0
12:57:22.345 [28318] <2> bpcd main: backuptime before fork = 0
12:57:22.347 [28318] <2> bpcd exit_bpcd: exit status 0  ----------->exiting
12:57:22.347 [28322] <2> bpcd main: Converting args string into an argv
12:57:22.347 [28322] <2> bpcd main: Done converting args string into an argv
12:57:22.347 [28322] <2> bpcd main: child_args[0] = /usr/openv/netbackup/bin/bpfilter
12:57:22.347 [28322] <2> bpcd main: child_args[1] = bpfilter
12:57:22.347 [28322] <2> bpcd main: child_args[2] = -r
12:57:22.347 [28322] <2> bpcd main: child_args[3] = 2147483647
12:57:22.347 [28322] <2> bpcd main: child_args[4] = -ru
12:57:22.347 [28322] <2> bpcd main: child_args[5] = root
12:57:22.347 [28322] <2> bpcd main: child_args[6] = -dt
12:57:22.347 [28322] <2> bpcd main: child_args[7] = 0
12:57:22.347 [28322] <2> bpcd main: child_args[8] = -to
12:57:22.347 [28322] <2> bpcd main: child_args[9] = 0
12:57:22.347 [28322] <2> bpcd main: child_args[10] = -clnt
12:57:22.347 [28322] <2> bpcd main: child_args[11] = MediaServer.Short.Name
12:57:22.347 [28322] <2> bpcd main: child_args[12] = -class
12:57:22.347 [28322] <2> bpcd main: child_args[13] = MediaServer.Short.Name_HPUX_FSBACKUP_FULL
12:57:22.347 [28322] <2> bpcd main: child_args[14] = -sched
12:57:22.347 [28322] <2> bpcd main: child_args[15] = WEEKLY_FULL
12:57:22.347 [28322] <2> bpcd main: child_args[16] = -st
12:57:22.347 [28322] <2> bpcd main: child_args[17] = FULL
12:57:22.347 [28322] <2> bpcd main: child_args[18] = -bpstart_to
12:57:22.347 [28322] <2> bpcd main: child_args[19] = 600
12:57:22.347 [28322] <2> bpcd main: child_args[20] = -bpend_to
12:57:22.347 [28322] <2> bpcd main: child_args[21] = 960
12:57:22.347 [28322] <2> bpcd main: child_args[22] = -read_to
12:57:22.347 [28322] <2> bpcd main: child_args[23] = 1200
12:57:22.347 [28322] <2> bpcd main: child_args[24] = -stream_count
12:57:22.347 [28322] <2> bpcd main: child_args[25] = 28
12:57:22.347 [28322] <2> bpcd main: child_args[26] = -stream_number
12:57:22.347 [28322] <2> bpcd main: child_args[27] = 28
12:57:22.347 [28322] <2> bpcd main: child_args[28] = -jobgrpid
12:57:22.348 [28322] <2> bpcd main: child_args[29] = 117842
12:57:22.348 [28322] <2> bpcd main: child_args[30] = -use_otm
12:57:22.348 [28322] <2> bpcd main: child_args[31] = -use_ofb
12:57:22.348 [28322] <2> bpcd main: child_args[32] = -b
12:57:22.348 [28322] <2> bpcd main: child_args[33] = MediaServer.Short.Name_1235411731
12:57:22.348 [28322] <2> bpcd main: child_args[34] = -fso
12:57:22.348 [28322] <2> bpcd main: child_args[35] = -F
12:57:22.348 [28322] <2> bpcd main: child_args[36] = encipher
12:57:22.348 [28322] <2> bpcd main: Before execvp of command
12:57:34.707 [28325] <2> bpcd main: offset to GMT 18000
12:57:34.707 [28325] <2> logconnections: BPCD ACCEPT FROM 169.xxx.xxx.xxx.1194 TO 169.xxx.xxx.xxx.13724
12:57:34.707 [28325] <2> bpcd main: setup_sockopts complete
12:57:34.771 [28325] <2> print_arp_info: MasterSerer.fully.qualified.domain.name (169.xxx.xxx.xxx) at 0:14:c2:c2:49:f3 ether
12:57:34.772 [28325] <2> bpcd peer_hostname: Connection from host MasterSerer.fully.qualified.domain.name (169.xxx.xxx.xxx) port 1194
12:57:34.780 [28325] <2> bpcd valid_server: comparing MasterServer.fully.qualified.domain.name and MasterSerer.fully.qualified.domain.name
12:57:34.805 [28325] <4> bpcd valid_server: hostname comparison succeeded
12:57:34.814 [28325] <2> bpcd main: output socket port number = 1
12:57:35.057 [28325] <2> bpcd main: Duplicated vnetd socket on stderr
12:57:35.057 [28325] <2> bpcd main: <---- NetBackup 6.0 0 ------------initiated
12:57:35.057 [28325] <2> bpcd main: Not using VxSS authentication with MasterSerer.fully.qualified.domain.name
12:57:35.203 [28325] <2> bpcd main: BPCD_GET_VERSION_RQST
12:57:35.422 [28325] <2> bpcd main: BPCD_GET_VERSION_RQST
12:57:35.640 [28325] <2> bpcd main: BPCD_READ_USERS_CONFIG_RQST
12:57:35.859 [28325] <2> bpcd main: BPCD_GET_VERSION_RQST
12:57:36.078 [28325] <2> bpcd main: BPCD_READ_HOST_CONFIG_RQST
12:57:36.406 [28325] <2> bpcd main: BPCD_DISCONNECT_RQST
12:57:36.406 [28325] <2> bpcd exit_bpcd: exit status 0  ----------->exiting
12:57:36.807 [28330] <2> bpcd main: offset to GMT 18000
12:57:36.808 [28330] <2> logconnections: BPCD ACCEPT FROM 169.xxx.xxx.xxx.1199 TO 169.xxx.xxx.xxx.13724
12:57:36.808 [28330] <2> bpcd main: setup_sockopts complete
12:57:36.868 [28330] <2> print_arp_info: MasterSerer.fully.qualified.domain.name (169.xxx.xxx.xxx) at 0:14:c2:c2:49:f3 ether
12:57:36.869 [28330] <2> bpcd peer_hostname: Connection from host MasterSerer.fully.qualified.domain.name (169.xxx.xxx.xxx) port 1199
12:57:36.877 [28330] <2> bpcd valid_server: comparing MasterServer.fully.qualified.domain.name and MasterSerer.fully.qualified.domain.name
12:57:36.903 [28330] <4> bpcd valid_server: hostname comparison succeeded
12:57:36.911 [28330] <2> bpcd main: output socket port number = 1
12:57:37.136 [28330] <2> bpcd main: Duplicated vnetd socket on stderr
12:57:37.137 [28330] <2> bpcd main: <---- NetBackup 6.0 0 ------------initiated
12:57:37.137 [28330] <2> bpcd main: Not using VxSS authentication with MasterSerer.fully.qualified.domain.name
12:57:37.281 [28330] <2> bpcd main: BPCD_GET_VERSION_RQST
12:57:37.500 [28330] <2> bpcd main: BPCD_GET_VERSION_RQST
12:57:37.719 [28330] <2> bpcd main: BPCD_READ_USERS_CONFIG_RQST
12:57:37.937 [28330] <2> bpcd main: BPCD_GET_VERSION_RQST
12:57:38.156 [28330] <2> bpcd main: BPCD_READ_HOST_CONFIG_RQST
12:57:38.375 [28330] <2> bpcd main: BPCD_DISCONNECT_RQST
12:57:38.375 [28330] <2> bpcd exit_bpcd: exit status 0  ----------->exiting
12:57:45.107 [28338] <2> bpcd main: offset to GMT 18000
12:57:45.107 [28338] <2> logconnections: BPCD ACCEPT FROM 169.xxx.xxx.xxx.1205 TO 169.xxx.xxx.xxx.13724
12:57:45.107 [28338] <2> bpcd main: setup_sockopts complete
12:57:45.167 [28338] <2> print_arp_info: MasterSerer.fully.qualified.domain.name (169.xxx.xxx.xxx) at 0:14:c2:c2:49:f3 ether
12:57:45.167 [28338] <2> bpcd peer_hostname: Connection from host MasterSerer.fully.qualified.domain.name (169.xxx.xxx.xxx) port 1205
12:57:45.175 [28338] <2> bpcd valid_server: comparing MasterServer.fully.qualified.domain.name and MasterSerer.fully.qualified.domain.name
12:57:45.202 [28338] <4> bpcd valid_server: hostname comparison succeeded
12:57:45.210 [28338] <2> bpcd main: output socket port number = 1
12:57:45.439 [28338] <2> bpcd main: Duplicated vnetd socket on stderr
12:57:45.439 [28338] <2> bpcd main: <---- NetBackup 6.0 0 ------------initiated
12:57:45.439 [28338] <2> bpcd main: Not using VxSS authentication with MasterSerer.fully.qualified.domain.name
12:57:45.598 [28338] <2> bpcd main: BPCD_FORK_CMD_RQST
12:57:45.668 [28338] <2> bpcd main: fork cmd = /usr/openv/netbackup/bin/bptm bptm -unload -dn MediaServer.Short.Name_tld1_hcart2_drive0 -dp /dev/rmt/c20t0d2BESTnb -dk 2000137 -m VMB722 -mk 4001519 -mds 8 -alocid 23156 -jobid -1235403471 -jm
12:57:45.668 [28338] <2> bpcd main: filter type = -1
12:57:45.668 [28338] <2> bpcd main: backuptime before fork = 0
12:57:45.669 [28338] <2> bpcd exit_bpcd: exit status 0  ----------->exiting
12:57:45.669 [28342] <2> bpcd main: Converting args string into an argv
12:57:45.669 [28342] <2> bpcd main: Done converting args string into an argv
12:57:45.670 [28342] <2> bpcd main: child_args[0] = /usr/openv/netbackup/bin/bptm
12:57:45.670 [28342] <2> bpcd main: child_args[1] = bptm
12:57:45.670 [28342] <2> bpcd main: child_args[2] = -unload
12:57:45.670 [28342] <2> bpcd main: child_args[3] = -dn
12:57:45.670 [28342] <2> bpcd main: child_args[4] = MediaServer.Short.Name_tld1_hcart2_drive0
12:57:45.670 [28342] <2> bpcd main: child_args[5] = -dp
12:57:45.670 [28342] <2> bpcd main: child_args[6] = /dev/rmt/c20t0d2BESTnb
12:57:45.670 [28342] <2> bpcd main: child_args[7] = -dk
12:57:45.670 [28342] <2> bpcd main: child_args[8] = 2000137
12:57:45.670 [28342] <2> bpcd main: child_args[9] = -m
12:57:45.670 [28342] <2> bpcd main: child_args[10] = VMB722
12:57:45.670 [28342] <2> bpcd main: child_args[11] = -mk
12:57:45.670 [28342] <2> bpcd main: child_args[12] = 4001519
12:57:45.670 [28342] <2> bpcd main: child_args[13] = -mds
12:57:45.670 [28342] <2> bpcd main: child_args[14] = 8
12:57:45.670 [28342] <2> bpcd main: child_args[15] = -alocid
12:57:45.670 [28342] <2> bpcd main: child_args[16] = 23156
12:57:45.670 [28342] <2> bpcd main: child_args[17] = -jobid
12:57:45.670 [28342] <2> bpcd main: child_args[18] = -1235403471
12:57:45.670 [28342] <2> bpcd main: child_args[19] = -jm
12:57:45.670 [28342] <2> bpcd main: Before execvp of command

[AQasir]

Heres more info related to this problem. It seems that each stream of the filse system backup is failing with error  "Error bpbrm(pid=908) from client csuxprd1: aCC runtime: Use of "-mt" must be consistent during both compilation and linking."

Does anyone know what causes this error. I have been searching the web and found that HP-UX patches could cause such a problem. Does anyone know specifically which patch?

Message Edited by AQasir on 02-23-2009 02:39 PM

[subhojit_pal]

Hi Qasir,

I faced the same problem with some HP-UX client Netbackup Version 6.5.1. I think, Unix admins have applied some C++ library patches which is hosing the bpbkar process. Check with the Unix admins if they have installed any patches and whether it can be uninstalled. Otherwise, open a case with Symantec support to provide fixed bpbkar . In our case, that was the solution.

Thanks,

Subhojit.

[AQasir]

I had come to the same conclusion and you are absolutely right. Patches were installed after last weekends backup and since I only take backup of this server once a week the issue did not surface until this weekend.

I found on various websites which patches could be causing this issue and I will work with the admin to remove the patches one by one and test the backup process. This way I will know which particular patches caused the problem. I will post my findings once testing is completed. 

[AQasir · Accepted solution]

As promised here are my findings and resolution to my problem :

In my QA environment I was able to replicate the error during the backup. I then uninstalled Unix patch PHSS_38154 and tested another backup and this time the backup worked. So for me the solution was to uninstall PHSS_38154 because I am running UNIX 11.11 !

This solution is posted else where on the internet so I cant claim any credit. Thanks to those that posted their experience with this type of problem.

Message Edited by AQasir on 02-25-2009 03:23 PM