Getting error 13

[vijay5]

I can see this error,

04:14:11.922 [13401] <2> unix_daemonize: 0 = close(3) errno = 10 mode = 0x00008180
04:14:11.922 [13401] <2> unix_daemonize: 0 = close(4) errno = 10 mode = 0x0000c000
04:14:11.922 [13401] <2> unix_daemonize: 0 = close(5) errno = 10 mode = 0x0000c000
04:14:11.922 [13401] <2> unix_daemonize: 0 = close(6) errno = 10 mode = 0x0000c000
04:14:11.922 [13401] <2> unix_daemonize: 0 = close(7) errno = 10 mode = 0x0000c000
04:14:11.922 [13401] <2> unix_daemonize: 0 = close(8) errno = 10 mode = 0x0000c000
04:14:11.922 [13401] <2> unix_daemonize: 0 = close(9) errno = 10 mode = 0x0000c000
04:15:12.194 [20207] <8> do_long_siocgifconf: [vnet_addrinfo.c:5589] socket() failed 221 0xdd
04:15:12.194 [20207] <8> vnet_getifaddrs: [vnet_addrinfo.c:5385] do_long_siocgifconf1() failed 10 0xa
04:16:12.096 [20207] <8> do_long_siocgifconf: [vnet_addrinfo.c:5589] socket() failed 221 0xdd
04:16:12.096 [20207] <8> vnet_getifaddrs: [vnet_addrinfo.c:5385] do_long_siocgifconf1() failed 10 0xa
04:17:13.508 [20207] <8> do_long_siocgifconf: [vnet_addrinfo.c:5589] socket() failed 221 0xdd
04:17:13.508 [20207] <8> vnet_getifaddrs: [vnet_addrinfo.c:5385] do_long_siocgifconf1() failed 10 0xa
04:18:16.419 [20207] <8> do_long_siocgifconf: [vnet_addrinfo.c:5589] socket() failed 221 0xdd
04:18:16.419 [20207] <8> vnet_getifaddrs: [vnet_addrinfo.c:5385] do_long_siocgifconf1() failed 10 0xa
04:20:16.769 [20207] <8> do_long_siocgifconf: [vnet_addrinfo.c:5589] socket() failed 221 0xdd
04:20:16.769 [20207] <8> vnet_getifaddrs: [vnet_addrinfo.c:5385] do_long_siocgifconf1() failed 10 0xa
04:22:17.304 [20207] <8> do_long_siocgifconf: [vnet_addrinfo.c:5589] socket() failed 221 0xdd
04:22:17.344 [20207] <8> vnet_getifaddrs: [vnet_addrinfo.c:5385] do_long_siocgifconf1() failed 10 0xa
04:24:17.891 [20207] <8> do_long_siocgifconf: [vnet_addrinfo.c:5589] socket() failed 221 0xdd
04:24:17.936 [20207] <8> vnet_getifaddrs: [vnet_addrinfo.c:5385] do_long_siocgifconf1() failed 10 0xa
04:26:17.976 [20207] <8> do_long_siocgifconf: [vnet_addrinfo.c:5589] socket() failed 221 0xdd
04:26:17.977 [20207] <8> vnet_getifaddrs: [vnet_addrinfo.c:5385] do_long_siocgifconf1() failed 10 0xa
04:28:17.975 [20207] <8> do_long_siocgifconf: [vnet_addrinfo.c:5589] socket() failed 221 0xdd
04:28:17.975 [20207] <8> vnet_getifaddrs: [vnet_addrinfo.c:5385] do_long_siocgifconf1() failed 10 0xa

[vijay5]

I found this in bpcd logs

[Marianne]

Still not enough info....

Please share all relevant info:

NBU version and patch level on client, media server and master.

OS versions on client, media server and master.

Is only one client affected or more than one?

Details of backup selection ?

Can you show us all text in Details tab of failed job?

[vijay5]

@Marianne: PFB details

NBU version and patch level on client, media server and master.

Master : 7.6.1.2, Media server: 7.6.1.2, client: NetBackup-HP-UX11.11 7.5

OS versions on client, media server and master

Master:Linux 2.6.32, Media server: Linux 2.6.32, Client: Hp-UX

Is only one client affected or more than one? Only this client

Details of backup selection ?

/
/etc
/export/home
/local/data
/local/opt
/local/opt/CTMAGENT
/local/opt/SYSLOAD
/local/opt/TIVOLI
/opt
/opt/CTMAGENT
/opt/Monitoring
/stand
/u00?
/var
/local/home

Can you show us all text in Details tab of failed job?

12/23/2016 4:00:17 AM - Info nbjm(pid=122293) starting backup job (jobid=1503174) for client ftnis03, policy ux-p-sys-hp1111-ab-a29-std, schedule incr
12/23/2016 4:00:17 AM - Info nbjm(pid=122293) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=1503174, request id:{EF1F7500-C8BB-11E6-986F-6F79202B92E2})
12/23/2016 4:00:17 AM - requesting resource stu_disk_fr0-nbuapm88-p01
12/23/2016 4:00:17 AM - requesting resource nbumtr1.NBU_CLIENT.MAXJOBS.ftnis03
12/23/2016 4:00:17 AM - requesting resource nbumtr1.NBU_POLICY.MAXJOBS.ux-p-sys-hp1111-ab-a29-std
12/23/2016 4:00:30 AM - granted resource nbumtr1.NBU_CLIENT.MAXJOBS.ftnis03
12/23/2016 4:00:30 AM - granted resource nbumtr1.NBU_POLICY.MAXJOBS.ux-p-sys-hp1111-ab-a29-std
12/23/2016 4:00:30 AM - granted resource MediaID=@aaaah;DiskVolume=PureDiskVolume;DiskPool=dp_disk_fr0-nbuapm88-p01;Path=PureDiskVolume;StorageServer=fr0-nbuapm88-p01;MediaServer=fr0-nbuapm88-p01
12/23/2016 4:00:30 AM - granted resource stu_disk_fr0-nbuapm88-p01
12/23/2016 4:02:07 AM - estimated 772876 Kbytes needed
12/23/2016 4:02:07 AM - Info nbjm(pid=122293) started backup (backupid=ftnis03_1482462127) job for client ftnis03, policy ux-p-sys-hp1111-ab-a29-std, schedule incr on storage unit stu_disk_fr0-nbuapm88-p01
12/23/2016 4:02:07 AM - started process bpbrm (165641)
12/23/2016 4:02:08 AM - Info bpbrm(pid=165641) ftnis03 is the host to backup data from
12/23/2016 4:02:08 AM - Info bpbrm(pid=165641) reading file list for client
12/23/2016 4:02:08 AM - Info bpbrm(pid=165641) accelerator enabled
12/23/2016 4:02:11 AM - Info bpbrm(pid=165641) starting bpbkar on client
12/23/2016 4:02:11 AM - Warning bpbrm(pid=165641) from client ftnis03: WRN - NetBackup configuration flag IGNORE_XATTR set for backup operation.
12/23/2016 4:02:11 AM - Info bpbkar(pid=3336) Backup started
12/23/2016 4:02:11 AM - Info bpbrm(pid=165641) bptm pid: 165689
12/23/2016 4:02:11 AM - connecting
12/23/2016 4:02:11 AM - connected; connect time: 0:00:00
12/23/2016 4:02:12 AM - Info bptm(pid=165689) start
12/23/2016 4:02:12 AM - Info bptm(pid=165689) using 262144 data buffer size
12/23/2016 4:02:12 AM - Info bptm(pid=165689) using 30 data buffers
12/23/2016 4:02:13 AM - Info bptm(pid=165689) start backup
12/23/2016 4:02:14 AM - Info bptm(pid=165689) backup child process is pid 165744
12/23/2016 4:02:14 AM - begin writing
12/23/2016 4:02:17 AM - Info bpbkar(pid=3336) 5000 entries sent to bpdbm
12/23/2016 4:02:17 AM - Info bpbrm(pid=165641) from client ftnis03: TRV - [/dev/log.un] is a socket special file. Skipping
12/23/2016 5:02:18 AM - Error bpbrm(pid=165641) socket read failed: errno = 62 - Timer expired
12/23/2016 5:02:20 AM - Error bptm(pid=165689) media manager terminated by parent process
12/23/2016 5:02:24 AM - Info fr0-nbuapm88-p01(pid=165689) StorageServer=PureDisk:fr0-nbuapm88-p01; Report=PDDO Stats for (fr0-nbuapm88-p01): scanned: 3570 KB, CR sent: 156 KB, CR sent over FC: 0 KB, dedup: 95.6%, cache disabled
12/23/2016 5:02:29 AM - Info bpbkar(pid=3336) done. status: 13: file read failed
12/23/2016 5:02:29 AM - end writing; write time: 1:00:15
file read failed (13)

[Marianne]

Thanks for this info.

Based on the Backup Selection - another question or two:

Is 'Allow Multiple Streams' selected in Policy Attributes?
If so - can you tell us which stream is failing?
If not - can you please select this option so that you can see which stream is problematic?

We can see that bpbkar sent data to the media server up to this point:
12/23/2016 4:02:17 AM - Info bpbkar(pid=3336) 5000 entries sent to bpdbm

After skipping the socket special file, it seems bpbkar on the client got 'stuck'.

It seems that no more data from the client was received until bpbrm on the media server timed out an hour later and subsequently killed bptm:

12/23/2016 5:02:18 AM - Error bpbrm(pid=165641) socket read failed: errno = 62 - Timer expired
12/23/2016 5:02:20 AM - Error bptm(pid=165689) media manager terminated by parent process

This message errno = 62 - Timer expired is actually a Client Read Timeout.

I do not recommend an increase of Client Read Timeout.

Rather split the Backup Selection into separate streams (as explained above) and enable logs to see where exactly the client is getting 'stuck'. 

Logs - level 3 should be fine:
On client: bpbkar
On media server: bptm and bpbrm

[Marianne]

See this TN:

http://www.veritas.com/docs/000038599 

Problem

BUG REPORT: Backup fails with status 13 when bpbkar hangs processing /dev files on VxFS 5.0.

Error Message

socket read failed: errno = 62 - Timer expired

Solution

Bug: 1667080

Detail/Symptom(s):
When extended attributes are in use on VxFS 5.0 files, the NetBackup bpbkar process may hang indefinitely resulting in a timeout on the media server and subsequent failure of the backup job.

.....

Workaround:
In most instances, the problem can be prevented by adding the following line to the bp.conf file on the client host.

IGNORE_XATTR = YES

[vijay5]

After enabling Multistreaming,

12/23/2016 11:28:29 AM - Info nbjm(pid=122293) starting backup job (jobid=1503475) for client ftnis03, policy ux-p-sys-hp1111-ab-a29-std, schedule full
12/23/2016 11:28:29 AM - Info nbjm(pid=122293) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=1503475, request id:{8BEF5538-C8FA-11E6-B2F0-1B1D930794D1})
12/23/2016 11:28:29 AM - requesting resource stu_disk_fr0-nbuapm88-p01
12/23/2016 11:28:29 AM - requesting resource nbumtr1.NBU_CLIENT.MAXJOBS.ftnis03
12/23/2016 11:28:29 AM - requesting resource nbumtr1.NBU_POLICY.MAXJOBS.ux-p-sys-hp1111-ab-a29-std
12/23/2016 11:28:31 AM - granted resource nbumtr1.NBU_CLIENT.MAXJOBS.ftnis03
12/23/2016 11:28:31 AM - granted resource nbumtr1.NBU_POLICY.MAXJOBS.ux-p-sys-hp1111-ab-a29-std
12/23/2016 11:28:31 AM - granted resource MediaID=@aaaah;DiskVolume=PureDiskVolume;DiskPool=dp_disk_fr0-nbuapm88-p01;Path=PureDiskVolume;StorageServer=fr0-nbuapm88-p01;MediaServer=fr0-nbuapm88-p01
12/23/2016 11:28:31 AM - granted resource stu_disk_fr0-nbuapm88-p01
12/23/2016 11:28:36 AM - estimated 0 Kbytes needed
12/23/2016 11:28:36 AM - Info nbjm(pid=122293) started backup (backupid=ftnis03_1482488916) job for client ftnis03, policy ux-p-sys-hp1111-ab-a29-std, schedule full on storage unit stu_disk_fr0-nbuapm88-p01
12/23/2016 11:28:37 AM - Info bpbrm(pid=32242) ftnis03 is the host to backup data from
12/23/2016 11:28:37 AM - Info bpbrm(pid=32242) reading file list for client
12/23/2016 11:28:37 AM - started process bpbrm (32242)
12/23/2016 11:28:38 AM - Info bpbrm(pid=32242) accelerator enabled
12/23/2016 11:29:09 AM - connecting
12/23/2016 11:29:14 AM - Info bpbrm(pid=32242) starting bpbkar on client
12/23/2016 11:29:14 AM - connected; connect time: 0:00:05
12/23/2016 11:29:15 AM - Warning bpbrm(pid=32242) from client ftnis03: WRN - NetBackup configuration flag IGNORE_XATTR set for backup operation.
12/23/2016 11:29:15 AM - Info bpbkar(pid=4273) Backup started
12/23/2016 11:29:15 AM - Info bpbrm(pid=32242) bptm pid: 32308
12/23/2016 11:29:15 AM - Info bpbrm(pid=32242) from client ftnis03: TRV - Cannot process path [/local/opt/TIVOLI]: [No such file or directory]. Skipping
12/23/2016 11:29:15 AM - Info bpbkar(pid=4273) accelerator sent 0 bytes out of 0 bytes to server, optimization 0.0%
12/23/2016 11:29:15 AM - Info bptm(pid=32308) start
12/23/2016 11:29:16 AM - Info bptm(pid=32308) using 262144 data buffer size
12/23/2016 11:29:16 AM - Info bptm(pid=32308) using 30 data buffers
12/23/2016 11:29:21 AM - Info bptm(pid=32308) start backup
12/23/2016 11:29:26 AM - Info bptm(pid=32308) backup child process is pid 32369
12/23/2016 11:29:26 AM - Info bptm(pid=32308) waited for full buffer 0 times, delayed 0 times
12/23/2016 11:29:26 AM - Info bptm(pid=32308) EXITING with status 90 <----------
12/23/2016 11:29:26 AM - begin writing
12/23/2016 11:29:33 AM - Info fr0-nbuapm88-p01(pid=32308) StorageServer=PureDisk:fr0-nbuapm88-p01; Report=PDDO Stats for (fr0-nbuapm88-p01): scanned: 3 KB, CR sent: 0 KB, CR sent over FC: 0 KB, dedup: 100.0%, cache disabled
12/23/2016 11:29:35 AM - Info bpbkar(pid=4273) done. status: 71: None of the files mentioned in the file list exist or may not be accessible
12/23/2016 11:29:35 AM - end writing; write time: 0:00:09
None of the files mentioned in the file list exist or may not be accessible (71)

These are got failed with error 71,

/export/home
/local/data
/local/opt/CTMAGENT
/local/opt/TIVOLI
/opt/CTMAGENT
/opt/Monitoring
/u00?

[Marianne]

This is a different issue. Non-existent folders or filesystems will not show errors in single stream backup.

Are the rest of the streams still running? We need to see which one fails with status 13.

[vijay5]

You are right... Only mentioned folders are not exists...

Remaning are got success. Now no error 13.

[Marianne]

Did you make any changes other than multi-streaming?
Maybe the bp.conf entry?

[vijay5]

no only multistreaming enabled and checked for failed directories and inform Unix team to check on missing directories. I didnt change bp.conf

[vijay5]

What is the use of IGNORE_XATTR = YES ? Can you expain ?

[Marianne]

All explained in the TN that I've posted above.