Backup of databases on 1 Exchange 2010 node fail with error 69

[ZacP]

Hi, I'm hoping someone can help out with a Netbackup issue we've been having for the past couple of days;

Environment; Netbackup v 7.5.0.3, Exchange 2010 DAG

We're backing up all of our Exchange 2010 DAG databases via a single policy in Netbackup, which had been working correctly, but for the past couple of days backups from 1 of the DAG nodes is failing with error 69. Telnet & bpclntcmd from the Exhcange node seems to be working correctly. It doesn't look liek a bpfis log has been created on that node since the backup last ran successfully.

The databases themselves are mounted and healthy in the exchange console.

I've sanitised some of the host names etc in the log.

From the activity log;

Parent job;

09/04/2012 09:40:45 - Info nbjm (pid=3364) starting backup job (jobid=99936) for client DAG1.xxxxxx.com, policy xxxxxx_Exchange, schedule Incremental
09/04/2012 09:40:45 - Info nbjm (pid=3364) requesting MEDIA_SERVER_ONLY resources from RB for backup job (jobid=99936, request id:{953BF2BC-75A8-4E32-BB26-07F02392ED87})
09/04/2012 09:40:45 - requesting resource BKM03_SU_A
09/04/2012 09:40:45 - requesting resource dcsp-bk01.cloud.internal.NBU_CLIENT.MAXJOBS.DAG1.xxxxxx.com
09/04/2012 09:40:45 - requesting resource dcsp-bk01.cloud.internal.NBU_POLICY.MAXJOBS.xxxxxx_Exchange
09/04/2012 09:40:45 - requesting resource EXCHANGE_RESOLVER.dcsp-bk01.cloud.internal.xxxxxx_Exchange.DAG1.xxxxxx.com
09/04/2012 09:40:46 - granted resource  dcsp-bk01.cloud.internal.NBU_CLIENT.MAXJOBS.DAG1.xxxxxx.com
09/04/2012 09:40:46 - granted resource  dcsp-bk01.cloud.internal.NBU_POLICY.MAXJOBS.xxxxxx_Exchange
09/04/2012 09:40:46 - granted resource  EXCHANGE_RESOLVER.dcsp-bk01.cloud.internal.xxxxxx_Exchange.DAG1.xxxxxx.com
09/04/2012 09:40:46 - estimated 1473592 kbytes needed
09/04/2012 09:40:46 - begin Parent Job
09/04/2012 09:40:46 - Info RUNCMD (pid=1756) started
09/04/2012 09:40:46 - Info RUNCMD (pid=1756) exiting with status: 0
Operation Status: 0
09/04/2012 09:40:46 - end Parent Job; elapsed time 0:00:00
Operation Status: 0
Operation Status: 0
09/04/2012 09:40:46 - started process bpbrm (pid=1724)
09/04/2012 09:40:56 - Info bpbrm (pid=1724) DAG1.sucrogen.com is the host to restore to
09/04/2012 09:40:56 - Info bpbrm (pid=1724) reading file list from client
09/04/2012 09:41:05 - Info bpbrm (pid=1724) client_pid=8968
09/04/2012 09:41:05 - Info bpbrm (pid=1724) from client DAG1.xxxxxx.com: TRV - BPRESOLVER has executed on server (WNSTLEX07)
Operation Status: 0
Operation Status: 0
Operation Status: 69
Operation Status: 0
09/04/2012 09:44:31 - Info RUNCMD (pid=6744) started
09/04/2012 09:44:31 - Info RUNCMD (pid=6744) exiting with status: 0
Operation Status: 0
Operation Status: 69
invalid filelist specification  (69)

From the child job;

09/04/2012 09:43:13 - Info nbjm (pid=3364) starting backup job (jobid=99937) for client xxxxxx, policy xxxxxx_Exchange, schedule Incremental
09/04/2012 09:43:13 - Info nbjm (pid=3364) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=99937, request id:{AF16E7CA-77F9-48A6-897C-F18582868C89})
09/04/2012 09:43:13 - requesting resource BKM03_SU_A
09/04/2012 09:43:13 - requesting resource dcsp-bk01.cloud.internal.NBU_CLIENT.MAXJOBS.xxxxxx
09/04/2012 09:43:13 - requesting resource dcsp-bk01.cloud.internal.NBU_POLICY.MAXJOBS.xxxxxx_Exchange
09/04/2012 09:43:13 - granted resource  dcsp-bk01.cloud.internal.NBU_CLIENT.MAXJOBS.xxxxxx
09/04/2012 09:43:13 - granted resource  dcsp-bk01.cloud.internal.NBU_POLICY.MAXJOBS.Sucrogen_Exchange
09/04/2012 09:43:13 - granted resource  MediaID=@aaac_;DiskVolume=M:\;DiskPool=BKM03_PoolA;Path=M:\;StorageServer=dcsp-bkm03.cloud.internal;MediaServer=dcsp-bkm03.cloud.internal
09/04/2012 09:43:13 - granted resource  BKM03_SU_A
09/04/2012 09:43:13 - estimated 53133334 kbytes needed
09/04/2012 09:43:13 - begin Parent Job
Operation Status: 0
09/04/2012 09:43:13 - end Parent Job; elapsed time 0:00:00
Operation Status: 0
09/04/2012 09:43:15 - started process bpbrm (pid=3636)
09/04/2012 09:43:21 - Info bpbrm (pid=3636) xxxxxx is the host to backup data from
09/04/2012 09:43:21 - Info bpbrm (pid=3636) reading file list from client
09/04/2012 09:43:21 - Info bpbrm (pid=3636) start bpfis on client
09/04/2012 09:43:21 - Info bpbrm (pid=3636) Starting create snapshot processing
09/04/2012 09:43:22 - Info bpfis (pid=8600) Backup started
09/04/2012 09:43:46 - Warning bpbrm (pid=3636) from client xxxxxx: WRN - snapshot preparation successful, with the following condition: Exchange database <xxxxxx_Service_Account> is not mounted and cannot be backed up.
09/04/2012 09:43:51 - end writing
Operation Status: 1
Operation Status: 0
09/04/2012 09:43:52 - started process bpbrm (pid=2924)
09/04/2012 09:43:58 - Info bpbrm (pid=2924) Starting delete snapshot processing
09/04/2012 09:43:58 - Info bpfis (pid=0) Snapshot will not be deleted
09/04/2012 09:44:04 - Error bpbrm (pid=2924) from client WNRYDEX03: Get bpfis state from dcsp-bk01.cloud.internal failed. status = 48
09/04/2012 09:44:04 - Info bpfis (pid=7448) Backup started
09/04/2012 09:44:14 - Error bpbrm (pid=2924) from client WNRYDEX03: Delete bpfis state from dcsp-bk01.cloud.internal failed. status = 48
09/04/2012 09:44:31 - end writing
Operation Status: 0
Operation Status: 69
invalid filelist specification  (69)

From what I understand error 48 generally means there is a name resolution problem, but not sure where to go next.

Thanks,

Zac

[watsons]

There are some technotes on resolving error 48: http://www.symantec.com/docs/TECH59149

Using those bpclntcmd, bptestbpcd, nslookup should help identify where the name resolution is.

Becauase of the error 48,   "Exchange database <xxxxxx_Service_Account> is not mounted and cannot be backed up", hence the error 69.

[V4]

Schedule seems to be incremental as name suggests it? Could you confirm is it working fine with Full Schedule?

[jim_dalton]

Zac,

Check the status of the DBs on your nodes, check that all the nodes in the dag are defined within the preferred server list and that the NB software is installed and configged correctly on those preferred servers. Also have a look at the state of the snapshots on the nodes. If you search the forum theres an extensive article on bpfis and snapshots via vssadmin.

Jim