Hi All bpbkar log file is

Jain_Thomas · ‎02-01-2015

Hi All

Status code 40 fails the backup soon after conection is established with the client.

Ping status from the master server to the client as well as from the clint to the master server does not show any packet drops.

What can be the possible cause for this failure? Your valuable suggestions and feedbacks will be most helpful.

02/01/2015 14:27:05 - Info bpbrm (pid=68353) qnsraapst01-bkp is the host to backup data from
02/01/2015 14:27:05 - Info bpbrm (pid=68353) reading file list for client
02/01/2015 14:27:05 - Info bpbrm (pid=68353) accelerator enabled
02/01/2015 14:27:27 - Info bpbrm (pid=68353) starting bpbkar on client
02/01/2015 14:27:27 - Info bpbkar (pid=0) Starting bpstart_notify script
02/01/2015 14:27:36 - Info bpbkar (pid=0) Finished bpstart_notify script
02/01/2015 14:27:36 - Info bpbkar (pid=10256) Backup started
02/01/2015 14:27:36 - Info bpbrm (pid=68353) bptm pid: 68435
02/01/2015 14:27:37 - Info bptm (pid=68435) start
02/01/2015 14:27:38 - Info bptm (pid=68435) using 262144 data buffer size
02/01/2015 14:27:38 - Info bptm (pid=68435) using 30 data buffers
02/01/2015 14:27:42 - Info bptm (pid=68435) start backup
02/01/2015 14:27:49 - Info bptm (pid=68435) backup child process is pid 68448
02/01/2015 14:27:49 - Info bptm (pid=68435) waited for full buffer 0 times, delayed 0 times
02/01/2015 14:27:50 - Info bptm (pid=68435) EXITING with status 90 <----------
02/01/2015 14:27:55 - Info qnlmv3nbuapl01 (pid=68435) StorageServer=PureDisk:qnlmv3nbuapl01; Report=PDDO Stats for (qnlmv3nbuapl01): scanned: 2 KB, CR sent: 0 KB, CR sent over FC: 0 KB, dedup: 100.0%, cache disabled
02/01/2015 14:40:12 - Info nbjm (pid=27869) starting backup job (jobid=640999) for client qnsraapst01-bkp, policy Unix_Filesystem, schedule Monthly_Full
02/01/2015 14:40:12 - Info nbjm (pid=27869) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=640999, request id:{155967A6-AA07-11E4-8876-0010E04041D6})
02/01/2015 14:40:12 - requesting resource stu_disk_qnlmv3nbuapl01
02/01/2015 14:40:12 - requesting resource qnlmv3nbusrv.NBU_CLIENT.MAXJOBS.qnsraapst01-bkp
02/01/2015 14:40:12 - requesting resource qnlmv3nbusrv.NBU_POLICY.MAXJOBS.Unix_Filesystem
02/01/2015 14:40:12 - granted resource qnlmv3nbusrv.NBU_CLIENT.MAXJOBS.qnsraapst01-bkp
02/01/2015 14:40:12 - granted resource qnlmv3nbusrv.NBU_POLICY.MAXJOBS.Unix_Filesystem
02/01/2015 14:40:12 - granted resource MediaID=@aaaan;DiskVolume=PureDiskVolume;DiskPool=dp_disk_qnlmv3nbuapl01;Path=PureDiskVolume;StorageServer=qnlmv3nbuapl01;MediaServer=qnlmv3nbuapl01
02/01/2015 14:40:12 - granted resource stu_disk_qnlmv3nbuapl01
02/01/2015 14:40:12 - estimated 63782054 kbytes needed
02/01/2015 14:40:12 - Info nbjm (pid=27869) started backup (backupid=qnsraapst01-bkp_1422790812) job for client qnsraapst01-bkp, policy Unix_Filesystem, schedule Monthly_Full on storage unit stu_disk_qnlmv3nbuapl01
02/01/2015 14:40:14 - started process bpbrm (pid=68353)
02/01/2015 14:40:35 - connecting
02/01/2015 14:40:35 - connected; connect time: 0:00:00
02/01/2015 14:40:58 - begin writing
network connection broken (40)

punbs · ‎02-01-2015

Hi..

Could you please cross verifty the selection detail and data availability in the perticular selection path.

Note: Error 90 always says that "The tape manager (bptm) or disk manager (bpdm) received no data when it performed a backup, archive, or duplication"

Please confirm, post that we will about the error 40

Rgds, punbs

Marianne · ‎02-01-2015

Also curious to know what is in Backup Selection for this policy. Have you created bpbkar log folder on the client? Please copy the log file to bpbkar.txt and upload as File attachment.

Handy NetBackup Links

watsons · ‎02-01-2015

i notice you use "qnsraapst01-bkp" as the client name, and likely a backup interface.

When you did the ping or traceroute test between master server to client, did you use the same backup interface as shown above?

If you're just testing production interface, that is not accurate - you will have to make sure backup interface does not experience network connection issues or packet drop as well.

Jain_Thomas · ‎02-02-2015

Hi All

bpbkar log file is attached.

Backup policy is set to take the backup of all local drives

root@qnsraapst01 # df -hk
Filesystem size used avail capacity Mounted on
rpool/ROOT/s10s_u10wos_17b
80G 13G 57G 19% /
/devices 0K 0K 0K 0% /devices
ctfs 0K 0K 0K 0% /system/contract
proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
swap 86G 496K 86G 1% /etc/svc/volatile
objfs 0K 0K 0K 0% /system/object
sharefs 0K 0K 0K 0% /etc/dfs/sharetab
/platform/sun4v/lib/libc_psr/libc_psr_hwcap3.so.1
70G 13G 57G 19% /platform/sun4v/lib/libc_psr.so.1
/platform/sun4v/lib/sparcv9/libc_psr/libc_psr_hwcap3.so.1
70G 13G 57G 19% /platform/sun4v/lib/sparcv9/libc_psr.so.1
fd 0K 0K 0K 0% /dev/fd
swap 86G 368K 86G 1% /tmp
swap 86G 64K 86G 1% /var/run
PAT3-iii 26G 1.1G 24G 5% /iii
PAT3-iiidb 305G 18G 287G 6% /iiidb
PAT3-iiidb-errlog 35G 480M 34G 2% /iiidb/errlog
PAT3-iiidb-software 119G 23G 97G 19% /iiidb/software
PAT3-temp 1.0T 31K 1.0T 1% /temp
root@qnsraapst01 #

Hostname to IP resolution of the client from master server is possible

-bash-4.1#
-bash-4.1# ping -s 10.1.228.83
PING 10.1.228.83: 56 data bytes
64 bytes from qnsraapst01-bkp (10.1.228.83): icmp_seq=0. time=0.695 ms
64 bytes from qnsraapst01-bkp (10.1.228.83): icmp_seq=1. time=0.725 ms
64 bytes from qnsraapst01-bkp (10.1.228.83): icmp_seq=2. time=0.670 ms
^C
----10.1.228.83 PING Statistics----
3 packets transmitted, 3 packets received, 0% packet loss
round-trip (ms) min/avg/max/stddev = 0.670/0.697/0.725/0.028
-bash-4.1#
-bash-4.1#
-bash-4.1#
-bash-4.1# ping qnsraapst01-bkp
qnsraapst01-bkp is alive
-bash-4.1#
-bash-4.1# ping -s qnsraapst01-bkp
PING qnsraapst01-bkp: 56 data bytes
64 bytes from qnsraapst01-bkp (10.1.228.83): icmp_seq=0. time=0.827 ms
64 bytes from qnsraapst01-bkp (10.1.228.83): icmp_seq=1. time=1.031 ms
64 bytes from qnsraapst01-bkp (10.1.228.83): icmp_seq=2. time=1.615 ms
64 bytes from qnsraapst01-bkp (10.1.228.83): icmp_seq=3. time=1.193 ms
^C
----qnsraapst01-bkp PING Statistics----
4 packets transmitted, 4 packets received, 0% packet loss
round-trip (ms) min/avg/max/stddev = 0.827/1.167/1.615/0.334
-bash-4.1#

Jain_Thomas · ‎02-02-2015

z

Marianne · ‎02-02-2015

Has backup ever worked for this client? I notice that accelerator is used with this policy. Have you tried without accelerator? Also try with 'Allow multiple data streams' in policy attributes. Strange errors in bpbkar that I've never seen before: <16> bpbkar: reached 12131

Handy NetBackup Links

Jain_Thomas · ‎02-03-2015

Hi Marianne

Yes, backup was fine till date.

Enabled "Allow multiple data streams" and found that it fails for one particular mount point /iii. And if I list the contents of the mountpoint it shows INNOVATIVE SYSTEM SOFTWARE UPDATE IN PROGRESS.

root@qnsraapst01 #
root@qnsraapst01 # df -hk
Filesystem size used avail capacity Mounted on
rpool/ROOT/s10s_u10wos_17b
80G 13G 57G 19% /
/devices 0K 0K 0K 0% /devices
ctfs 0K 0K 0K 0% /system/contract
proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
swap 85G 480K 85G 1% /etc/svc/volatile
objfs 0K 0K 0K 0% /system/object
sharefs 0K 0K 0K 0% /etc/dfs/sharetab
/platform/sun4v/lib/libc_psr/libc_psr_hwcap3.so.1
70G 13G 57G 19% /platform/sun4v/lib/libc_psr.so.1
/platform/sun4v/lib/sparcv9/libc_psr/libc_psr_hwcap3.so.1
70G 13G 57G 19% /platform/sun4v/lib/sparcv9/libc_psr.so.1
fd 0K 0K 0K 0% /dev/fd
swap 85G 368K 85G 1% /tmp
swap 85G 64K 85G 1% /var/run
PAT3-iii 26G 1.1G 24G 5% /iii
PAT3-iiidb 305G 18G 287G 6% /iiidb
PAT3-iiidb-errlog 35G 481M 34G 2% /iiidb/errlog
PAT3-iiidb-software 119G 23G 97G 19% /iiidb/software
PAT3-temp 1.0T 31K 1.0T 1% /temp
root@qnsraapst01 #
root@qnsraapst01 #
root@qnsraapst01 #
root@qnsraapst01 # cd /iii
root@qnsraapst01 # ls -Rl
.:
total 29
drwxr-x--- 7 iii iii 29 Feb 3 03:00 iiihome
drwxrwx--- 2 iii iii 2 Feb 1 2013 iiiweb
drwxrwx--- 21 iii iii 21 Jan 13 11:02 work

./iiihome:
total 1675
-rw-rw---- 1 iii iii 0 Jan 3 22:03 alex
-rwxrwx--- 1 iii iii 578668 Mar 6 2013 alignkw
drwxrwx--- 4 iii iii 4 Apr 18 2013 App not found. Exiting 1
-rw-rw---- 1 iii iii 96 Oct 7 00:22 bschk1.scr
-r-xr-x--- 1 iii iii 53567 Mar 6 2013 dbflip
-rw-rw---- 1 iii iii 4626 Mar 6 2013 dbflip.log
-r--r----- 1 iii iii 25556 Nov 28 2013 ifetch.py
-r--r--r-- 1 root sys 53165 Mar 6 2013 initpostgres
drwxrwx--- 4 iii iii 4 Apr 18 2013

INNOVATIVE SYSTEM SOFTWARE UPDATE IN PROGRESS

Please try again later....
-r--r----- 1 iii iii 14866 Nov 28 2013 ipull.py
-r--r----- 1 iii iii 328 Nov 28 2013 runpython
-rwxr-xr-x 1 iii iii 18576 Mar 29 2013 seedstatus
lrwxrwxrwx 1 root iii 28 Mar 29 2013 sierraconv -> /iiidb/data/cconv/sierraconv
-r-xr-x--- 1 iii iii 16575 Mar 30 2013 sierra_resync_dfile
-rw-rw-rw- 1 iii iii 184 Jan 28 13:44 texttohold.del

./iiihome/App not found. Exiting 1:
total 6
drwxrwx--- 2 iii iii 2 Apr 18 2013 conf
drwxrwx--- 2 iii iii 2 Apr 18 2013 logs

./iiihome/App not found. Exiting 1/conf:
total 0

./iiihome/App not found. Exiting 1/logs:
total 0

./iiihome/

INNOVATIVE SYSTEM SOFTWARE UPDATE IN PROGRESS

Please try again later....:
total 6
drwxrwx--- 2 iii iii 2 Apr 18 2013 conf
drwxrwx--- 2 iii iii 2 Apr 18 2013 logs

./iiihome/

INNOVATIVE SYSTEM SOFTWARE UPDATE IN PROGRESS

Please try again later..../conf:
total 0

./iiihome/

INNOVATIVE SYSTEM SOFTWARE UPDATE IN PROGRESS

Please try again later..../logs:
total 0

./iiiweb:
total 0

./work:
total 61
drwxrwx--- 2 iii iii 7 Oct 16 2013 achan
drwxrwx--- 2 iii iii 2 Jan 13 11:04 adixit
drwxrwx--- 2 iii iii 16 Mar 19 2014 amit
drwxrwx--- 2 iii iii 5 Sep 19 11:09 atautu
drwxrwx--- 2 iii iii 3 Jul 30 2014 davidg
drwxrwx--- 2 iii iii 10 Apr 2 2013 dhl
drwxrwx--- 2 iii iii 3 Oct 2 22:55 dialer_shellshock
drwxrwx--- 2 iii iii 5 Jun 12 2013 dli
drwxrwx--- 2 iii iii 2 Jun 2 2014 fflorian
drwxrwx--- 2 iii iii 7 Nov 21 08:46 her
drwxrwx--- 2 iii iii 3 Jun 6 2014 ipittas

Marianne · ‎02-06-2015

Seems you found the reason for the failure.

Any progress in the meantime?
Rest of mount points backing up fine?

Handy NetBackup Links

VOX

Info bptm (pid=68435) EXITING with status 90 <---------- network connection broken (40)