cancel
Showing results for 
Search instead for 
Did you mean: 

Exchange 2010 DAG backups failing with 69 and 130

rjrumfelt
Level 6

Hello All,

Sorry, it's been a while since I've posted on here, and things have changed around quite a bit.  I wasn't able to search to see if someone else has asked a similar question, and the KB wasn't very helpful with my questions.

I'm backing up an Exchange 2010 DAG with 4 members, and on one member I'm getting an exit status 69, and on another an exit status 130.  For the 69, I've ensured that the DB's on that particular member are mounted and healthy, and that the NBU client is running as the correct user (not local system, but one we specifically created for Exchange).  For the 130, all of the vss writers appear healthy, and I'm curious as to where to go next.  I currently can't log into the box with the 130, as it is offline, so I'd like to focus on the 69 error if possible.The bpresolver log doesn't really tell me much, and we don't even have one from the latest backup.

 

Thoughts?

1 ACCEPTED SOLUTION

Accepted Solutions

Mark_Solutions
Level 6
Partner Accredited Certified

I am sure it is but just check that the account you are using is also specifically added as a local admin on the exchange servers / CAS servers

You will also need to set the Legacy NetBackup Client Network Service to use the same account

View solution in original post

14 REPLIES 14

Tim_Hansen
Level 5
Certified

are you backing up passive or active copies? If passive, can your mailbox servers communicate with the active copy without issues? Is circular logging enabled, and if not, how many log files are there for the failing databases? are there any logs generated in the windows application event logs that coincide with the time of the backups (corrupt or missing log files, network issues, consistency checks failing, etc)?

rjrumfelt
Level 6

Communication has been verified to all DAG members to/from master and media servers, as well as between DAG members.  We're backing up the passive copies and then failing over to the active if the passive doesn't work.  Circular logging is not enabled.  No event logs unfortunately.

I can get you the number of log files, but just out of curiosity, why do you ask about the number of log files?

Nicolai
Moderator
Moderator
Partner    VIP   

 

Do you have a mix of DAG and non-DAG information stores ?
 
Of some sort strange to me specifying "Database Availability Groups:\" will fail if you have non-dag information stores. You need to specify the entire DAG path as in "Database Availability Groups:\DAG-USR1". If you have a "clean" setup there is no issue.
 
Br Nicolai

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Welcome back Jason!

Please tell us which NBU version on Master and Exchange nodes?

GRT backups?

Can you possibly share text in Details tab of status 69 and 130 jobs? We will ask for logs based on what we see in Job details.

Ensure bpfis and bpbkar logs exist on on all nodes.

rjrumfelt
Level 6

All db's are configured within the DAG.  Also, we are enumerating the databases, rather than just using the directive.  I realize that the directive is there for a reason, and its a more robust solution to use the directive rather than list out the db's in the backup target list, however due to certain limitations in our environment, we want the ability to, in the event of a failure, rerun the stream for an individual database.

rjrumfelt
Level 6

Thanks Marianne.  bpfis and bpbkar are both enabled, however for the backup failing 130, I can't get to bpfis right now b/c the system is currenlty unavailable.  We're not using GRT, just your basic DB backup through the DAG.  The detail logs from the activity monitor are below.  The "Unable to backup Exchange database. It may not be mounted., status 69" comment is a bit misleading b/c all DB's are mounted, online and healthy, on both passive and active nodes.

69:

 

04/13/2012 02:00:53 - Info nbjm (pid=1368) starting backup job (jobid=2544318) for client exchclient, policy Microsoft_Exchange_MB_SG, schedule Incremental

04/13/2012 02:00:53 - Info nbjm (pid=1368) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=2544318, request id:{0739DD0A-852E-11E1-ADF0-00144FAADA8E})

04/13/2012 02:00:53 - requesting resource storgeunit-hcart3-robot-tld-4-DR

04/13/2012 02:00:53 - requesting resource master_server.NBU_CLIENT.MAXJOBS. exchclient

04/13/2012 02:00:53 - requesting resource master_server.NBU_POLICY.MAXJOBS. Microsoft_Exchange_MB_SG

04/13/2012 02:01:52 - granted resource  master_server.NBU_CLIENT.MAXJOBS. exchclient

04/13/2012 02:01:52 - granted resource  master_server.NBU_POLICY.MAXJOBS. Microsoft_Exchange_MB_SG

04/13/2012 02:01:52 - granted resource  WA0031

04/13/2012 02:01:52 - granted resource  IBM.ULTRIUM-TD3.148

04/13/2012 02:01:52 - granted resource  storageunit-hcart3-robot-tld-4-DR

04/13/2012 02:01:55 - estimated 40339694 kbytes needed

04/13/2012 02:01:55 - begin Parent Job

Operation Status: 0

04/13/2012 02:01:55 - end Parent Job; elapsed time 0:00:00

Operation Status: 0

04/13/2012 02:01:55 - started process bpbrm (pid=648)

04/13/2012 02:01:59 - Info bpbrm (pid=648) exchclient is the host to backup data from

04/13/2012 02:01:59 - Info bpbrm (pid=648) reading file list from client

04/13/2012 02:01:59 - Info bpbrm (pid=648) start bpfis on client

04/13/2012 02:01:59 - begin Create Snapshot

04/13/2012 02:02:00 - Info bpfis (pid=8984) Backup started

04/13/2012 02:05:18 - Critical bpbrm (pid=648) from client exchclient: FTL - snapshot preparation failed - Unable to backup Exchange database. It may not be mounted., status 69

04/13/2012 02:05:18 - Info bpbrm (pid=648) DB_BACKUP_STATUS is 103

04/13/2012 02:05:19 - Critical bpbrm (pid=648) from client exchclient: FTL - snapshot preparation failed, status 69

04/13/2012 02:05:19 - Info bpfis (pid=8984) done. status: 69

04/13/2012 02:05:19 - end Create Snapshot; elapsed time 0:03:20

04/13/2012 02:05:20 - Info bpfis (pid=0) done. status: 69: invalid filelist specification

04/13/2012 02:05:20 - end writing

Operation Status: 69

Operation Status: 0

04/13/2012 02:05:21 - started process bpbrm (pid=921)

04/13/2012 02:05:27 - end writing

Operation Status: 0

Operation Status: 69

invalid filelist specification  (69)

 

130:

 

04/13/2012 02:00:10 - Info nbjm (pid=1368) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=2544317, request id:{ED882434-852D-11E1-9579-00144FAADA8E})

04/13/2012 02:00:10 - requesting resource storageunit -hcart3-robot-tld-4-DR

04/13/2012 02:00:10 - requesting resource master_server.NBU_CLIENT.MAXJOBS. exchclient2

04/13/2012 02:00:10 - requesting resource master_server.NBU_POLICY.MAXJOBS. Microsoft_Exchange_MB_SG

04/13/2012 02:01:51 - granted resource  master_server.NBU_CLIENT.MAXJOBS.D2ASEPRSH122

04/13/2012 02:01:51 - granted resource  master_server.NBU_POLICY.MAXJOBS. Microsoft_Exchange_MB_SG

04/13/2012 02:01:51 - granted resource  WA0022

04/13/2012 02:01:51 - granted resource  IBM.ULTRIUM-TD3.131

04/13/2012 02:01:51 - granted resource  storageunit -hcart3-robot-tld-4-DR

04/13/2012 02:01:54 - estimated 40339694 kbytes needed

04/13/2012 02:01:54 - begin Parent Job

Operation Status: 0

04/13/2012 02:01:54 - end Parent Job; elapsed time 0:00:00

Operation Status: 0

04/13/2012 02:01:55 - started process bpbrm (pid=645)

04/13/2012 02:01:58 - Info bpbrm (pid=645) exchclient2 is the host to backup data from

04/13/2012 02:01:58 - Info bpbrm (pid=645) reading file list from client

04/13/2012 02:01:58 - Info bpbrm (pid=645) start bpfis on client

04/13/2012 02:01:58 - begin Create Snapshot

04/13/2012 02:01:59 - Info bpfis (pid=15496) Backup started

04/13/2012 02:04:43 - Critical bpbrm (pid=645) from client exchclient2: FTL - snapshot creation failed - SNAPSHOT_NOTIFICATION::SnapshotPrepare failed., status 130

04/13/2012 02:04:43 - Info bpbrm (pid=645) DB_BACKUP_STATUS is 156

04/13/2012 02:04:43 - Info bpbrm (pid=645) DB_BACKUP_STATUS is 156

04/13/2012 02:04:44 - Info bpbrm (pid=645) DB_BACKUP_STATUS is 156

04/13/2012 02:04:44 - Info bpbrm (pid=645) DB_BACKUP_STATUS is 156

04/13/2012 02:04:45 - Info bpbrm (pid=645) DB_BACKUP_STATUS is 156

04/13/2012 02:04:45 - Info bpbrm (pid=645) DB_BACKUP_STATUS is 156

04/13/2012 02:04:46 - Info bpbrm (pid=645) DB_BACKUP_STATUS is 156

04/13/2012 02:04:46 - Info bpbrm (pid=645) DB_BACKUP_STATUS is 156

04/13/2012 02:04:47 - Info bpbrm (pid=645) DB_BACKUP_STATUS is 156

04/13/2012 02:04:47 - Info bpbrm (pid=645) DB_BACKUP_STATUS is 156

04/13/2012 02:04:47 - Info bpbrm (pid=645) DB_BACKUP_STATUS is 156

04/13/2012 02:04:48 - Info bpbrm (pid=645) DB_BACKUP_STATUS is 156

04/13/2012 02:04:48 - Info bpbrm (pid=645) DB_BACKUP_STATUS is 156

04/13/2012 02:04:49 - Critical bpbrm (pid=645) from client exchclient2: FTL - snapshot creation failed, status 130

04/13/2012 02:04:49 - Warning bpbrm (pid=645) from client exchclient2: WRN - NEW_STREAM0 is not frozen

04/13/2012 02:04:49 - Warning bpbrm (pid=645) from client exchclient2: WRN - Microsoft Information Store:\DAG-DB03 is not frozen

04/13/2012 02:04:49 - Warning bpbrm (pid=645) from client exchclient2: WRN - NEW_STREAM1 is not frozen

04/13/2012 02:04:49 - Warning bpbrm (pid=645) from client exchclient2: WRN - Microsoft Information Store:\ DAG -DB11 is not frozen

04/13/2012 02:04:49 - Warning bpbrm (pid=645) from client exchclient2: WRN - NEW_STREAM2 is not frozen

04/13/2012 02:04:49 - Warning bpbrm (pid=645) from client exchclient2: WRN - Microsoft Information Store:\DAG-DB14 is not frozen

04/13/2012 02:04:49 - Warning bpbrm (pid=645) from client exchclient2: WRN - NEW_STREAM3 is not frozen

04/13/2012 02:04:49 - Warning bpbrm (pid=645) from client exchclient2: WRN - Microsoft Information Store:\DAG-DB15 is not frozen

04/13/2012 02:04:49 - Warning bpbrm (pid=645) from client exchclient2: WRN - NEW_STREAM4 is not frozen

04/13/2012 02:04:49 - Warning bpbrm (pid=645) from client exchclient2: WRN - Microsoft Information Store:\DAG-DB23 is not frozen

04/13/2012 02:04:49 - Warning bpbrm (pid=645) from client exchclient2: WRN - NEW_STREAM5 is not frozen

04/13/2012 02:04:50 - Warning bpbrm (pid=645) from client exchclient2: WRN - Microsoft Information Store:\DAG-DB26 is not frozen

04/13/2012 02:04:50 - Warning bpbrm (pid=645) from client exchclient2: WRN - NEW_STREAM6 is not frozen

04/13/2012 02:04:50 - Warning bpbrm (pid=645) from client exchclient2: WRN - Microsoft Information Store:\DAG-DB27 is not frozen

04/13/2012 02:04:50 - Warning bpbrm (pid=645) from client exchclient2: WRN - NEW_STREAM7 is not frozen

04/13/2012 02:04:50 - Warning bpbrm (pid=645) from client exchclient2: WRN - Microsoft Information Store:\DAG-DB35 is not frozen

04/13/2012 02:04:50 - Warning bpbrm (pid=645) from client exchclient2: WRN - NEW_STREAM8 is not frozen

04/13/2012 02:04:50 - Warning bpbrm (pid=645) from client exchclient2: WRN - Microsoft Information Store:\DAG-DB38 is not frozen

04/13/2012 02:04:50 - Warning bpbrm (pid=645) from client exchclient2: WRN - NEW_STREAM9 is not frozen

04/13/2012 02:04:50 - Warning bpbrm (pid=645) from client exchclient2: WRN - Microsoft Information Store:\DAG-DB47 is not frozen

04/13/2012 02:04:50 - Warning bpbrm (pid=645) from client exchclient2: WRN - NEW_STREAM10 is not frozen

04/13/2012 02:04:50 - Warning bpbrm (pid=645) from client exchclient2: WRN - Microsoft Information Store:\DAG-DB50 is not frozen

04/13/2012 02:04:50 - Warning bpbrm (pid=645) from client exchclient2: WRN - NEW_STREAM11 is not frozen

04/13/2012 02:04:50 - Warning bpbrm (pid=645) from client exchclient2: WRN - Microsoft Information Store:\DAG-DB59 is not frozen

04/13/2012 02:04:50 - Warning bpbrm (pid=645) from client exchclient2: WRN - NEW_STREAM12 is not frozen

04/13/2012 02:04:50 - Warning bpbrm (pid=645) from client exchclient2: WRN - Microsoft Information Store:\DAG-DB02 is not frozen

04/13/2012 02:04:50 - Info bpfis (pid=15496) done. status: 130

04/13/2012 02:04:50 - end Create Snapshot; elapsed time 0:02:52

04/13/2012 02:04:51 - Info bpfis (pid=0) done. status: 130: system error occurred

04/13/2012 02:04:51 - end writing

Operation Status: 130

Operation Status: 0

04/13/2012 02:04:52 - started process bpbrm (pid=901)

04/13/2012 02:04:59 - end writing

Operation Status: 0

Operation Status: 130

04/13/2012 02:14:59 - Info nbjm (pid=1368) starting backup job (jobid=2544317) for client exchclient2, policy Microsoft_Exchange_MB_SG, schedule Incremental

system error occurred  (130)

 

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

I have noticed the Status 103 'hiding' in the Job Details.

We don't know your NBU version yet? Unpatched version 7.0 had a bug (fixed in 7.0.1): http://www.symantec.com/docs/TECH125427

I guess you have already ruled out any possibility of typo error in Backup Selection?
Another possibility could be permissions - check NBU Client Service logon account. Try to use Exchange Admin account.

rjrumfelt
Level 6

Sorry, I forgot the version.  The master, media and clients are all running 7.1.0.3.  All members of the DAG are running the NBU account as an Exchange admin account.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Looks like we will need logs then.......

Mark_Solutions
Level 6
Partner Accredited Certified

I am sure it is but just check that the account you are using is also specifically added as a local admin on the exchange servers / CAS servers

You will also need to set the Legacy NetBackup Client Network Service to use the same account

rjrumfelt
Level 6

Mark,  That is one thing that we have not yet done.  I will make sure this has been set for the exit status 69.  I'll be working on sanitizing the bpfis logs for posting.

 

One note though, there are quite a few other DAGs in our environment, and they are set up the same way - with the regular NBU service running with an elevated privilage acct, and the legacy service running as local system.  This is the only member of all DAGs with this issue.

Nufane
Level 3

I'm getting this exact same error, and it appears to have started after my upgrade to Exchange 2010 SP2.  The only difference is that I'm running Public folders on my servers, so my backup is set with the full DAG paths, but I still get the 69 and 130 errors.

olumayyowa
Level 2
Partner Accredited Certified
I am getting this same error. The environment is exchange 2010 DAG with 5 members. The netbackup version is 7.1.0.4. Four members were backed up successfullly while the remaining one always return with: FTL - snapshot preparation failed - Unable to backup Exchange database. It may not be mounted., status 69. I have checked to be sure that the database is mounted. Also, All the DAG members are of the same server configuration. I need to resolve this issue soonest because the customer is becoming impatient. Any way out?

Maurice_Byrd
Level 4

Has anyone found an answer to this issue?  My environment is as follows.

NBU v7.1.0.4

2010 DAG with three members - circular logging disabled on mail databases and enabled for Journal

All servers are running Windows Server 2008 R2 - 64bit

Full backups run successfully; incrementals fail on two members of the DAG. On the servers where the backup fails the same two databases are passive on both. In the Exchange mgmt console all databases report healthy.

VSS writers on all Exchange servers report stable: no error

Here's a sample of the job details:

 

6/21/2012 9:48:11 AM - Info bpfis(pid=9444) Backup started          

6/21/2012 9:48:50 AM - Critical bpbrm(pid=6452) from client DDPEXCMBX01: FTL - snapshot preparation failed - Error attempting to find volumes to snap., status 130

6/21/2012 9:48:50 AM - Info bpbrm(pid=6452) DB_BACKUP_STATUS is 103         

6/21/2012 9:48:50 AM - Critical bpbrm(pid=6452) from client DDPEXCMBX01: FTL - snapshot preparation failed, status 130  

 

Thoughts?