Backups failing with error 10 and 13

[rraja]

The key message from the bpbkar log is:

03:09:35.608 [85] <4> bpbkar PrintFile: /bancsmicomsdata/XILogs/Trace1.log.2011-04-24-22

03:09:35.614 [85] <4> bpbkar PrintFile: /bancsmicomsdata/XILogs/Trace3.log

03:09:35.616 [85] <4> bpbkar PrintFile: /bancsmicomsdata/XILogs/Trace4.log

03:09:35.643 [85] <4> bpbkar PrintFile: /bancsmicomsdata/XILogs/Trace1.log.2011-04-24-12

03:29:39.592 [85] <32> bpbkar read_and_sort_dir_entries: FTL - dir_list_rec malloc failed. Errno = 11: Resource temporarily unavailable

03:29:39.593 [85] <16> bpbkar Exit: ERR - bpbkar FATAL exit status = 10: allocation failed

03:29:39.593 [85] <4> bpbkar Exit: INF - EXIT STATUS 10: allocation failed

The referenced ‘maloc’ is a system call to allocate memory to a process. There are a large number of files under the /bancsmicomsdata directory structure and it is likely that this is the cause of the failure. 

As per the Symantec tech note 76719, the possible solution is by increasing the memory for the /usr/openv/netbackup/bin/bpbkar process. But the tech note is for an error on AIX client, we are having issue on Solaris 10 client. Please check and confirm whether the same is applicable for Solaris 10 client and also advise if there are any other way to fix this issue. 

 http://www.symantec.com/docs/TECH76719

https://www-secure.symantec.com/connect/forums/backup-failure-19

[Yasuhisa_Ishika]

Resource management parameter "process.max-data-size" limits size of data segment of each process. And, "ulimit -d" also limits. # I'm not sure these limits let malloc() return EAGAIN. Try to devide your backup into some smaller streams to avoid massive memory allocation, for a diagnosis.

[Anton_Panyushki]

I wonder if you tried to ls the directory you want to backup. I'd make sure that OS can deal with this directory before taking any backups.

[rraja]

A scan of the /bancsmicomsdata directory structure reading the directory indexes took almost 30 minutes, a summary of the file count is:

/bancsmicomsdata/XILogs    6,445 files

/bancsmicomsdata/data      10,655 files

/bancsmicomsdata/datafiles 5,450,661 files

For a test purpose I have excluded the below these folders from the backup but still the backups are failing with error code 13. 

http://www.symantec.com/docs/TECH75332: Minimally, the application will require the number of file descriptors to be set to 8000 or greater.

• http://publib.boulder.ibm.com/infocenter/tivihelp/v2r1/index.jsp?topic=/com.ibm.itame3.doc_5.1/am51_...

nofiles(descriptors) option limits the number of file descriptors belonging to a single process. File descriptors includes not only files but also sockets for Internet communication.

The "nofiles" (descriptors) setting limits the number of concurrent connections to the server, because each connection requires an open socket descriptor. 

So I think we can try increase the nofiles value at client side. 

http://www.symantec.com/docs/TECH71821 : We have increased the client read timeout value at client side.