Netbackup 7.0.1 DAG error 26 and exchangesnapshot error 156

[elamarr]

Attempting to backup exchange 2010. Initially created VM servers and started backing up the servers via snapshots. All went well.

2 CAS & 2 Mailbox servers. Created the DAG on mail servers and errors started (156).

Configured permissions and new backup account. Followed referenced articles and checked possible registry keys.

Installed client NBU software and configured new policy to perform DAG backup (ref: guide and video below).

Error 26 appeared and attempted various settings (ex: VSS, MS_Windows, Exchange, etc.)

I feel I am missing something simple that I just can not grasp.

Admin guide:

http://www.symantec.com/business/support/index?page=content&id=TECH127056

Ref:

http://www.symantec.com/business/support/index?page=content&pmv=print&impressions=&viewlocale=&id=TE...

Video:

http://www.symantec.com/tv/products/details.jsp?vid=736743545001

Errors:

3/13/2012 4:47:14 AM - requesting resource uspaexnbu01-dlt-robot-tld-0
3/13/2012 4:47:14 AM - requesting resource uspaexnbu01.NBU_CLIENT.MAXJOBS.USPAEXEX03
3/13/2012 4:47:14 AM - requesting resource uspaexnbu01.NBU_POLICY.MAXJOBS.Snapshot-ExchangeServers
3/13/2012 4:47:14 AM - awaiting resource uspaexnbu01-dlt-robot-tld-0 - No drives are available
3/13/2012 4:48:53 AM - granted resource uspaexnbu01.NBU_CLIENT.MAXJOBS.USPAEXEX03
3/13/2012 4:48:53 AM - granted resource uspaexnbu01.NBU_POLICY.MAXJOBS.Snapshot-ExchangeServers
3/13/2012 4:48:53 AM - granted resource 001269
3/13/2012 4:48:53 AM - granted resource QUANTUM.SDLT600.002
3/13/2012 4:48:53 AM - granted resource uspaexnbu01-dlt-robot-tld-0
3/13/2012 4:48:53 AM - started
3/13/2012 4:48:54 AM - estimated 42102159 Kbytes needed
3/13/2012 4:48:54 AM - begin Parent Job
3/13/2012 4:48:54 AM - begin Flash Backup Windows, Start Notify Script
3/13/2012 4:48:54 AM - started process RUNCMD (7728)
3/13/2012 4:48:54 AM - ended process 0 (7728)
Status 0
3/13/2012 4:48:54 AM - end Flash Backup Windows, Start Notify Script; elapsed time: 00:00:00
3/13/2012 4:48:54 AM - begin Flash Backup Windows, Step By Condition
Status 0
3/13/2012 4:48:54 AM - end Flash Backup Windows, Step By Condition; elapsed time: 00:00:00
3/13/2012 4:48:54 AM - begin Flash Backup Windows, Read File List
Status 0
3/13/2012 4:48:54 AM - end Flash Backup Windows, Read File List; elapsed time: 00:00:00
3/13/2012 4:48:54 AM - begin Flash Backup Windows, Create Snapshot
3/13/2012 4:48:55 AM - started process bpbrm (7860)
3/13/2012 4:48:56 AM - begin Create Snapshot
3/13/2012 4:49:01 AM - snapshot backup of client USPAEXEX03 using method VMware
3/13/2012 4:50:53 AM - Critical bpbrm(pid=7860) from client USPAEXEX03: FTL - snapshot creation failed, status 156  
3/13/2012 4:50:53 AM - Warning bpbrm(pid=7860) from client USPAEXEX03: WRN - ALL_LOCAL_DRIVES is not frozen   
3/13/2012 4:50:55 AM - end Create Snapshot; elapsed time: 00:01:59
3/13/2012 4:50:56 AM - end writing
Status 156
3/13/2012 4:50:56 AM - end Flash Backup Windows, Create Snapshot; elapsed time: 00:02:02
3/13/2012 4:50:56 AM - begin Flash Backup Windows, Stop On Error
Status 0
3/13/2012 4:50:56 AM - end Flash Backup Windows, Stop On Error; elapsed time: 00:00:00
3/13/2012 4:50:56 AM - begin Flash Backup Windows, Delete Snapshot On Exit
3/13/2012 4:50:56 AM - begin Flash Backup Windows, Delete Snapshot On Exit
3/13/2012 4:50:57 AM - started process bpbrm (2316)
3/13/2012 4:51:03 AM - end writing
Status 0
3/13/2012 4:51:03 AM - end Flash Backup Windows, Delete Snapshot On Exit; elapsed time: 00:00:07
3/13/2012 4:51:03 AM - begin Flash Backup Windows, End Notify Script
3/13/2012 4:51:04 AM - started process RUNCMD (7504)
3/13/2012 4:51:04 AM - ended process 0 (7504)
Status 0
3/13/2012 4:51:04 AM - end Flash Backup Windows, End Notify Script; elapsed time: 00:00:01
Status 156
3/13/2012 4:51:04 AM - end Flash Backup Windows, Delete Snapshot On Exit; elapsed time: 00:00:08
snapshot error encountered(156)

-------------------***************************-------------------------------****************-----

3/13/2012 1:34:51 PM - requesting resource uspaexnbu01-dlt-robot-tld-0
3/13/2012 1:34:51 PM - requesting resource uspaexnbu01.NBU_CLIENT.MAXJOBS.WAIDAG
3/13/2012 1:34:51 PM - requesting resource uspaexnbu01.NBU_POLICY.MAXJOBS.2010Exchange-DAG
3/13/2012 1:34:51 PM - requesting resource EXCHANGE_RESOLVER.uspaexnbu01.2010Exchange-DAG.WAIDAG
3/13/2012 1:34:51 PM - granted resource uspaexnbu01.NBU_CLIENT.MAXJOBS.WAIDAG
3/13/2012 1:34:51 PM - granted resource uspaexnbu01.NBU_POLICY.MAXJOBS.2010Exchange-DAG
3/13/2012 1:34:51 PM - granted resource EXCHANGE_RESOLVER.uspaexnbu01.2010Exchange-DAG.WAIDAG
3/13/2012 1:34:51 PM - estimated 0 Kbytes needed
3/13/2012 1:34:51 PM - begin Parent Job
3/13/2012 1:34:51 PM - begin Unknown Type, Start Notify Script
3/13/2012 1:34:52 PM - started process RUNCMD (7400)
3/13/2012 1:34:52 PM - ended process 0 (7400)
Status 0
3/13/2012 1:34:52 PM - end Unknown Type, Start Notify Script; elapsed time: 00:00:01
3/13/2012 1:34:52 PM - begin Unknown Type, Step By Condition
Status 0
3/13/2012 1:34:52 PM - end Unknown Type, Step By Condition; elapsed time: 00:00:00
3/13/2012 1:34:52 PM - begin Unknown Type, Read File List
Status 0
3/13/2012 1:34:52 PM - end Unknown Type, Read File List; elapsed time: 00:00:00
3/13/2012 1:34:52 PM - begin Unknown Type, Resolver Discovery
3/13/2012 1:34:53 PM - started process bpbrm (4212)
3/13/2012 1:35:01 PM - Error bpbrm(pid=4212) This type of backup is not supported on this version of Exchange
Status 26
3/13/2012 1:35:06 PM - end Unknown Type, Resolver Discovery; elapsed time: 00:00:14
3/13/2012 1:35:06 PM - begin Unknown Type, Stop On Error
Status 0
3/13/2012 1:35:06 PM - end Unknown Type, Stop On Error; elapsed time: 00:00:00
3/13/2012 1:35:06 PM - begin Unknown Type, End Notify Script
3/13/2012 1:35:07 PM - started process RUNCMD (7396)
3/13/2012 1:35:07 PM - ended process 0 (7396)
Status 0
3/13/2012 1:35:07 PM - end Unknown Type, End Notify Script; elapsed time: 00:00:01
Status 26
3/13/2012 1:35:07 PM - end Parent Job; elapsed time: 00:00:16
client/server handshaking failed(26)

[J_H_Is_gone]

I know you have a vm backup of the exchange server

and then you installed the client software on the sever to do the dag backup

I think you are seeing just normal 'cannot talk' issues.

create a test windows policy to just backup the c drive of the exchange server.

if that fails then you need to do basic communication issue troubleshooting -  so for not take the dag out of the  picture and just try to see if you can backup the c drive as a normal windows policy.

 Once you can backup just the C drive - then disable the test policy and try the dag again.

[elamarr]

A normal C drive backup works fine.

If I try Snapshot I still get error.

Allso on the 2 CAS servers, I have another issue.

CAS server config:

Platform: VMWare VSphere 5

C drive: OS, Exchange software

D drive: Pagesys file.

The current snapshot works for just the C drive. How can we get the D drive or do we need to re-config the page file back on to C (moved for optimization and expected load))?

[Marianne]

'This type of backup is not supported on this version of Exchange'

Can you confirm that 7.0.1 patch was installed on Exchange nodes? This TN describes a bug causing this error. Fixed by 7.0.1. http://www.symantec.com/docs/TECH125427

[elamarr]

Getting better I reinstalled the client software again with patches, Restarted domain controllers and netbackup master server.

I still get a 156 error for C drive snapshot of mailbox servers.

I now can run a DAG backup and get a error 13

It creates 6 actual running jobs for the DAG

I do have 3 mailstores which are fairly empty. I am researching the read failure (13).

Thank you Marianne which helped me to back track and restart everything.

Still need help/suggestions.

Thank you in advance.

3/13/2012 11:34:02 PM - requesting resource nbu01-dlt-robot-tld-0
3/13/2012 11:34:02 PM - requesting resource nbu01.NBU_CLIENT.MAXJOBS.ex04
3/13/2012 11:34:02 PM - requesting resource nbu01.NBU_POLICY.MAXJOBS.2010Exchange-DAG
3/13/2012 11:34:02 PM - granted resource nbu01.NBU_CLIENT.MAXJOBS.ex04
3/13/2012 11:34:02 PM - granted resource nbu01.NBU_POLICY.MAXJOBS.2010Exchange-DAG
3/13/2012 11:34:02 PM - granted resource 001012
3/13/2012 11:34:02 PM - granted resource QUANTUM.SDLT600.005
3/13/2012 11:34:02 PM - granted resource nbu01-dlt-robot-tld-0
3/13/2012 11:34:02 PM - estimated 0 Kbytes needed
3/13/2012 11:34:02 PM - begin Parent Job
3/13/2012 11:34:02 PM - begin Unknown Type, Step By Condition
Status 0
3/13/2012 11:34:02 PM - end Unknown Type, Step By Condition; elapsed time: 00:00:00
3/13/2012 11:34:02 PM - begin Unknown Type, Read File List
Status 0
3/13/2012 11:34:02 PM - end Unknown Type, Read File List; elapsed time: 00:00:00
3/13/2012 11:34:02 PM - begin Unknown Type, Create Snapshot
3/13/2012 11:34:02 PM - started process bpbrm (7096)
3/13/2012 11:34:02 PM - started
3/13/2012 11:34:07 PM - begin Create Snapshot
3/13/2012 11:34:44 PM - Critical bpbrm(pid=7096) from client ex04: FTL - snapshot preparation failed - No Exchange databases were found during the enumeration of 2010MaiStore3, status 13
3/13/2012 11:34:44 PM - Critical bpbrm(pid=7096) from client ex04: FTL - snapshot preparation failed, status 13  
3/13/2012 11:34:45 PM - end Create Snapshot; elapsed time: 00:00:38
3/13/2012 11:34:47 PM - end writing
Status 13
3/13/2012 11:34:47 PM - end Unknown Type, Create Snapshot; elapsed time: 00:00:45
3/13/2012 11:34:47 PM - begin Unknown Type, Stop On Error
Status 0
3/13/2012 11:34:47 PM - end Unknown Type, Stop On Error; elapsed time: 00:00:00
3/13/2012 11:34:47 PM - begin Unknown Type, Delete Snapshot On Exit
3/13/2012 11:34:47 PM - begin Unknown Type, Delete Snapshot On Exit
3/13/2012 11:34:47 PM - started process bpbrm (6080)
3/13/2012 11:34:59 PM - end writing
Status 0
3/13/2012 11:34:59 PM - end Unknown Type, Delete Snapshot On Exit; elapsed time: 00:00:12
Status 13
3/13/2012 11:34:59 PM - end Unknown Type, Delete Snapshot On Exit; elapsed time: 00:00:12
file read failed(13)

[Marianne]

You need to troubleshoot the 2 backup failures separately.

For the C-drive status 156, please post your policy config. On master, run this from cmd:
<install-path>\veritas\netbackup\bin\admincmd\bppllist <policy-name> -L

For DAG backup, you need to troubleshoot this:

"No Exchange databases were found during the enumeration of 2010MaiStore3"

Please logon to ex04 as the same user starting the NBU Client Service, open the BAR GUI and see if you can browse mailstores and databases.

[elamarr]

Was out sick for a few days. Again thank you.

Here is the policy config.

Microsoft Windows [Version 6.0.6002]
Copyright (c) 2006 Microsoft Corporation.  All rights reserved.

C:\Program Files\Veritas\NetBackup\bin\admincmd>bppllist snapshot-exchangeserver
s -l

CLASS snapshot-exchangeservers *NULL* 0 701000 0 *NULL*
NAMES
INFO 29 0 0 0 *NULL* 0 0 2 0 0 0 0 0 0 1 0 1 0 1286807066 614976B03D2D4995AE0200
7101688892 1 0 0 0 0 1 0 0 2 0 0 0 0 1 0 -1 62
KEY *NULL*
BCMD *NULL*
RCMD *NULL*
RES uspaexnbu01-dlt-robot-tld-0 *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL*
 *NULL* *NULL*
POOL NetBackup NetBackup NetBackup NetBackup NetBackup NetBackup NetBackup NetBa
ckup NetBackup NetBackup
FOE 0 0 0 0 0 0 0 0 0 0
SHAREGROUP *ANY*
DATACLASSIFICATION *NULL*
ACN uspaexnbu01
SSM VMware
SSMARGCOUNT 7
SSMARG nameuse 1
SSMARG Virtual_machine_backup 2
SSMARG trantype 6
SSMARG disable_quiesce 0
SSMARG file_system_optimization 1
SSMARG snapact 2
SSMARG vmmono 0
CLIENT USPAEXEX01 vmx-07 winLonghorn64Guest 0 0 0 0 *NULL*
CLIENT USPAEXEX02 vmx-07 winLonghorn64Guest 0 0 0 0 *NULL*
CLIENT USPAEXEX03 vmx-07 winLonghorn64Guest 0 0 0 0 *NULL*
CLIENT USPAEXEX04 vmx-07 winLonghorn64Guest 0 0 0 0 *NULL*
INCLUDE ALL_LOCAL_DRIVES
SCHED Full 0 1 604800 0 0 0 0 *NULL* 0 1 0 0 0 0 -1 0
SCHEDCALENDAR
SCHEDCALDAYOWEEK 1,1;2,1;3,1;4,1;5,1;6,1;7,1;1,2;2,2;3,2;4,2;5,2;6,2;7,2;1,3;2,3
;3,3;4,3;5,3;6,3;7,3;1,4;2,4;3,4;4,4;5,4;6,4;7,4;1,5;2,5;3,5;4,5;5,5;6,5;7,5
SCHEDWIN 0 18000 0 54000 0 18000 0 18000 0 18000 0 18000 0 72000
SCHEDRES *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL*
SCHEDPOOL *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL*
SCHEDRL 0 0 0 0 0 0 0 0 0 0
SCHEDFOE 0 0 0 0 0 0 0 0 0 0
SCHEDSG *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL*

C:\Program Files\Veritas\NetBackup\bin\admincmd>

[Marianne]

Unformatted (-l) output is difficult to read.

Please post formatted output with big -L:

bppllist snapshot-exchangeservers -L

[elamarr]

Big L

C:\Program Files\Veritas\NetBackup\bin\admincmd>bppllist snapshot-exchangeserver
s -L

Policy Name:       snapshot-exchangeservers
Options:           0x0
template:          FALSE
c_unused1:         ?
Names:             (none)
Policy Type:       FlashBackup-Windows (29)
Active:            yes
Effective date:    10/11/2010 10:24:26
File Restore Raw:  yes
Mult. Data Stream: no
Perform Snapshot Backup:   yes
Snapshot Method:           VMware
Snapshot Method Arguments: nameuse=1,Virtual_machine_backup=2,trantype=6,disable
_quiesce=0,file_system_optimization=1,snapact=2,vmmono=0
Perform Offhost Backup:    yes
Backup Copy:               0
Use Data Mover:            no
Data Mover Type:           2
Use Alternate Client:      no
Alternate Client Name:     uspaexnbu01
Use Virtual Machine:      1
Hyper-V Server Name:     (none)
Enable Instant Recovery:   no
Policy Priority:   0
Max Jobs/Policy:   2
Disaster Recovery: 0
Collect BMR Info:  no
Keyword:           (none specified)
Data Classification:       -
Residence is Storage Lifecycle Policy:    no
Client Encrypt:    no
Checkpoint:        no
Residence:         uspaexnbu01-dlt-robot-tld-0
Volume Pool:       NetBackup
Server Group:      *ANY*
Granular Restore Info:  no
Exchange Source attributes:              no
Exchange 2010 Preferred Server: (none defined)
Generation:      62
Ignore Client Direct:  no
Client/HW/OS/Pri:  USPAEXEX01 vmx-07 winLonghorn64Guest 0 0 0 0 ?
Client/HW/OS/Pri:  USPAEXEX02 vmx-07 winLonghorn64Guest 0 0 0 0 ?
Client/HW/OS/Pri:  USPAEXEX03 vmx-07 winLonghorn64Guest 0 0 0 0 ?
Client/HW/OS/Pri:  USPAEXEX04 vmx-07 winLonghorn64Guest 0 0 0 0 ?
Include:           ALL_LOCAL_DRIVES
Schedule:          Full
  Type:            FULL (0)
  Calendar sched: Enabled
    Sunday, Week 1
    Monday, Week 1
    Tuesday, Week 1
    Wednesday, Week 1
    Thursday, Week 1
    Friday, Week 1
    Saturday, Week 1
    Sunday, Week 2
    Monday, Week 2
    Tuesday, Week 2
    Wednesday, Week 2
    Thursday, Week 2
    Friday, Week 2
    Saturday, Week 2
    Sunday, Week 3
    Monday, Week 3
    Tuesday, Week 3
    Wednesday, Week 3
    Thursday, Week 3
    Friday, Week 3
    Saturday, Week 3
    Sunday, Week 4
    Monday, Week 4
    Tuesday, Week 4
    Wednesday, Week 4
    Thursday, Week 4
    Friday, Week 4
    Saturday, Week 4
    Sunday, Week 5
    Monday, Week 5
    Tuesday, Week 5
    Wednesday, Week 5
    Thursday, Week 5
    Friday, Week 5
    Saturday, Week 5
  Maximum MPX:     1
  Synthetic:       0
  PFI Recovery:    0
  Retention Level: 0 (1 week)
  u-wind/o/d:      0 0
  Incr Type:       DELTA (0)
  Alt Read Host:   (none defined)
  Max Frag Size:   0 MB
  Number Copies:   1
  Fail on Error:   0
  Residence:       (specific storage unit not required)
  Volume Pool:     (same as policy volume pool)
  Server Group:    (same as specified for policy)
  Residence is Storage Lifecycle Policy:     0
  Daily Windows:
   Day         Open       Close       W-Open     W-Close
   Sunday      000:00:00  005:00:00   000:00:00  005:00:00
   Monday      000:00:00  015:00:00   024:00:00  039:00:00
   Tuesday     000:00:00  005:00:00   048:00:00  053:00:00
   Wednesday   000:00:00  005:00:00   072:00:00  077:00:00
   Thursday    000:00:00  005:00:00   096:00:00  101:00:00
   Friday      000:00:00  005:00:00   120:00:00  125:00:00
   Saturday    000:00:00  020:00:00   144:00:00  164:00:00

C:\Program Files\Veritas\NetBackup\bin\admincmd>

[Marianne]

Your policy shows 4 client names - are all of them failing with 156?

Do you have bpfis log folder on uspaexnbu01?

If so, pleas post log as attachment.

[elamarr]

Info near bottom. Just ran the failing at 16:24.

Only 2 of the 4 servers failing.

Starting to create seperate policies for the failing servers.

[elamarr]

Quick question,

VMWare - would having the same datastore location cause and be a culprit?

Disk:

C - VMFS

Logs & Mailstore - RDM

CAS servers (2) - on different datastores - ex: 1- on store #3  & 1- on store #6

Mailbox (2 servers *** ones failing ***)  - ex: 1- on store #3 & 1- on same store #3

*** Recap  -

History:

Servers created : 1-29-2012

Snapshot – ExchangeServers policy created 2-1-2012

Created new mailbox roles  on two servers – EX03 & EX04 - : 3-5-2012

156 errors started for EX03 & EX04 servers: 3-6-2012

Configured DAG policy -  2010Exchange-DAG    -: 3-8-2012

13 errors for DAG

***

[Marianne · Accepted solution]

The problem is with the RDM disks. NetBackup cannot back up the RDM by means of a VMware backup host.

See http://www.symantec.com/docs/HOWTO44434

http://www.symantec.com/docs/HOWTO70901

[elamarr]

Thanks to Marianne, we took a closer look and went with an upgrade to 7.5 and changed the virtual disk type.