Problem with AIX 5.3 client backup with ALL_LOCAL_DRIVES selection

[Tauhid]

I have windows 2008 master server running net backup 7.0.1

and I tried to backup of AIX 5.3 client.

1st time I make a policy with single file and it backup sucessfully.

Then I make a policy with ALL_LOCAL_DRIVES selection for AIX client.

the backup is started sucessfully. But after writing 47 GB it failes with the following errors: 

12/30/2013 11:25:33 AM - Critical bpbrm(pid=9732) from client 10.38.61.50: FTL - dir_list_rec malloc failed. Errno = 12: Not enough space

12/30/2013 11:26:32 AM - Error bpbrm(pid=9732) cannot connect to 10.38.61.50 to send mail, The operation completed successfully.  (0)

12/30/2013 11:26:32 AM - end writing; write time: 00:54:21

Could anyone give me a solution?

[Amol_Nair]

Take a look at the following technote
http://www.symantec.com/docs/TECH76719

[Jaime_Vazquez]

The TECH article is not correct for your scenario. AIX 5.3 is a 32 or 64 bit OS, depending on how it was installed. NBU 7.X for AIX is a 64 bit application and as such NBU 7.X only supports 64 bit OS versions, so the memory access limit is not the issue. For NBU 6.5.X, NBU was a 32 bit application. You can verify that on the AIX client by running "file /usr/openv/netbackup/bin/*'.  32 bit linked modules will read "executable (RISC System/6000) or object module".  64 bit modules will read "64-bit XCOFF executable or object module". There is a known issue with 'bprd' for earlier versions of NBU which was improperly linked with the wrong 'maxDATA' value. See #1 below on how to determine if your executable has this problem, and if so, how to repair it.  This is only on AIX Master Servers.

Memory limits are hit by one of two posible scenarios.

1.  The loader section has a 'maxdata' setting that is not 0. Except for 'bprd' all of the supplied executables are set this way. You can verify the executable loader values with " dump -o -X 64 $module_name".  Look for the entry for 'maxDATA'. To set it to zero dynamially, you can run "ldedit -o maxdata:0 $executalbe_name". To be safe, make a copy of the executable ahead of time, just in case. The module needs to be reloaded/restarted for the change to take place.

2. The "ulimit' value for memory has been exceeded.  Run "ulimit -a" to see the values.  AIX kernel enforces the lower of the two values. The working values are found in "/etc/security/limits". Talk to your sysadmin on how to raise them on your server.

[Tauhid]

Thanks for sharing information.

TECH76719 is not my case as my system is 64 bit application as Jaime_Vazquezsaid.

But in "dump -o -X 64 $module_name" , what is the module name?

and in "ldedit -o maxdata:0 $executalbe_name" what is the executable name?

Ulimit shows the value unlimited.

Could you please give further advise to proceed

[Marianne]

Firstly, if you do not have 'allow multiple data streams' enabled in the policy, please do so.

This will tell us which filesystem is having a problem. Only job for problematic filesystem will fail.

Create bpbkar log folder on the client and increase logging level. Level 3 should give enough info but I guess our Symantec friends will insist on level 5.

Please tell us which filesystem and upload bpbkar log file as File attachment.

[Tauhid]

I allow the multiple data stream.

I craete a bpbkar log in "/usr/openv/netbackup/logs/bpbkar"

But I can't set the loging level using "vxlogcfg" command.

Present setting on "usr/openv/netbackup/nblog.conf" is :

***********************************************************************

###############################################################################
# !!! DO NOT EDIT THIS FILE MANUALLY. USE vxlogcfg INSTEAD !!!
#     Also do not edit the L10nResource related settings
###############################################################################

###############################################################################
#  Default Settings for all NetBackup Originator Identifiers (OIDs)
###############################################################################

Default.LogDirectory=/usr/openv/logs
Default.LogFilePermissions=664
Default.L10nLib=/usr/openv/lib/libvxexticu
Default.L10nResourceDir=/usr/openv/resources
Default.AppMsgLogging=ON
Default.DiagnosticLevel=6
Default.DebugLevel=1
Default.RolloverMode=FileSize|LocalTime
Default.RolloverAtLocalTime=00.00
Default.MaxLogFileSizeKB=51200
Default.NumberOfLogFiles=3
Default.LogRecycle=false
Default.LogToStdout=false
Default.LogToStderr=false
Default.LogToOslog=false
Default.SyslogOpt=0
Default.SyslogFacility=LOG_LOCAL5
Default.SyslogIdent=VRTS-NB

###############################################################################
#  These individual OID settings override the OID defaults
#
#  LogDirectory is only specified for shared OID's (i.e. OID's with log files)
###############################################################################

# 111. Enterprise Media Manager
111.L10nResource=nbemm
111.OIDNames=nbemm
111.LogDirectory=nbemm

# 116. Policy Execution Manager
116.L10nResource=nbpem
116.OIDNames=nbpem
116.LogDirectory=nbpem

# 117. Job Manager
117.L10nResource=nbjm
117.OIDNames=nbjm
117.LogDirectory=nbjm

# 118. Resource Broker
118.L10nResource=nbrb

******************************************************

I tried to change the debugging level by the command

#cd usr/openv/netbackup/bin

#vxlogcfg -a --prodid 51216 --orgid ALL -s
    DebugLevel=0 -s DiagnosticLevel=1

But "vxlogcfg" not working in my system.

*****************************************************

Another

        I tried to backup different file system one by one. Then I found that only "/Proc" is a problem with

file write failed Error(Error 14) at the beginning of the backup and details are below :

1/8/2014 4:18:09 PM - Warning bpbrm(pid=9260) from client 10.38.61.50: WRN - Could not get ACL information for /proc/sys/. Errno = 22: A system call received a parameter that is not valid.
1/8/2014 4:18:09 PM - Error bpbrm(pid=9260) from client 10.38.61.50: ERR - Cannot write to tir_info_file, fd is NULL.
1/8/2014 4:18:10 PM - mounting 1106L3
1/8/2014 4:18:13 PM - end writing
file write failed(14)

**********************************************************

Need further assistant.

[Andy_Welburn · Accepted solution]

You shouldn't be backing up /proc - is excluded by default on all but AIX systems:

proc (UNIX platforms)

Does not exclude automatically for AIX, so /proc must be added manually to the exclude list. If /proc is not added manually, partially successful backups may result with the ALL_LOCAL_DRIVES directive on AIX.

Files that are excluded from backups by default on UNIX/Linux
http://www.symantec.com/business/support/index?page=content&id=HOWTO67237

..... altho' not sure whether this will solve your initial problem.

[Marianne]

bpbkar is a legacy log - meaning that vxlogcfg is not applicable.

Simply add 
VERBOSE = 3
to /usr/openv/netbackup/bp.conf file on the client.

(Legacy and Unified logs are explained in Troubleshooting Guide.)

As per Andy's post - add /proc to exclude_list on client.