Exchange 2010 DAG Backup Issue

[sriramrane]

Hi Firends,

I have some issue with Exchange 2010 DAG database backup, not sure what could be the reason. to brief about my setup, below are the details

exch1.abc.net (DAG node1) 

exch2.abc.net (DAG node2)

exch3.abc.net (HUB/CAS node1)

exch4.abc.net (HUB/CAS node2)

exch-DAG (dag virtual name)

exchange-mail (exchange front-end virtual name)

I have NBU 7.5.0.6 client installed on all physical servers and services are running on domain admin account (service account) with required all previleges. 

In NBU Admin Console -> host properties -> Clients  I have modfied hte account to use (service account) and other paramters in checkbox and the exchange-mail in proxy host settings

In NBU Admin Console -> Host Porperties -> Distributed Application Restore Mapping the below are the entries added

Application Host       Component Host 

exch-DAG                  exch1.abc.net

exch-DAG                  exch2.abc.net

exchange-mail            exch3.abc.net

exchange-mail            exch4.abc.net

We have the GRT enabled backup for Exchange 2010 DAG databases. 

ISSUE : We have the successful backup once we restart the services. when we try to restart the backup the Exchange GRT backup wil fail with the STATUS CODE (2)  with following message "none of the requested files were backed up(2)"

After the job fail, I have tried to browse the database from selection list within backup job polices and found that I could not able to browse. Again once I restart the netbackup services on database server and HUB/CAS server, then once backup will go fine and agian have the same issue of failure of jobs with the status code 2. Appreciate your help on this.

Thanks 

Rane 

[RamNagalla]

so you have GRT enabled backup..

but you did not say about the NFS services in your setup, just wanted to make sure if that is forget to specify here and taken care in the setup.

if NFS is not configured.make sure you did that before another backup try

http://www.symantec.com/business/support/index?page=content&id=TECH145156

if yes...

please post the detail of the

bppllist <policyname> -L

and also failed job detail status.

does the job creating the child backup jobs.. or its failing at the parent job level only...

[sriramrane]

The all required pre-requisites are configured. the NFS Server on Media Server and the NFS Client on on Exchange nodes are enabled. 

As I have mentioned, I have taken a backup several time. But the problem here is, once I have the full backup the next time backup with exit with status code (2). Both GRT and Non-GRT backup work fine at once, but next time when I try to run they fail. 

Have tried to connect the Information  store from the backup selection list and I couldnt able to browse. But once I restart the services, I am able to browse in selection list and able to backup. the backups are consistent.

[Marianne]

Please show us all text in Job Details of failed as well as successful backup.

Job Details will probably tell us where to start with troubleshooting.

[Yogesh9881]

Plz verify

1. Netbackup Services is configured with exchange / Domain Admin rights on all DAG members.

2. Test file level backup 3-4 times without restarting NBU services (all nodes)

3. Do you observed anything on OS level logs (System / Application) ??

[sriramrane]

Hi Marianne,

Below is the failed job log for Exchange DAG

Microsoft Exchange Database Availability Groups:\MOI-DAG.moi.ae\Microsoft Information Store\PFDB

Microsoft Exchange Database Availability Groups:\MOI-DAG.moi.ae\Microsoft Information Store\PFDB2

Microsoft Exchange Database Availability Groups:\MOI-DAG.moi.ae\Microsoft Information Store\MBDB-ITSERVICES

Microsoft Exchange Database Availability Groups:\MOI-DAG.moi.ae\Microsoft Information Store\MBDB-SECTIONS

Microsoft Exchange Database Availability Groups:\MOI-DAG.moi.ae\Microsoft Information Store\MBDB-MANAGERS

Microsoft Exchange Database Availability Groups:\MOI-DAG.moi.ae\Microsoft Information Store\MBDB-USERS

PREFERRED_SERVER:srv110042

PREFERRED_SERVER:srv110043

24/01/2014 01:36:05 ? - Info nbjm(pid=5512) starting backup job (jobid=6991) for client MOI-DAG, policy moi_exchange_monthly, schedule Montly_Full  

24/01/2014 01:36:05 ? - Info nbjm(pid=5512) requesting MEDIA_SERVER_ONLY resources from RB for backup job (jobid=6991, request id:{449A9F70-FE73-4016-91C4-CCD7C638E802})  

24/01/2014 01:36:05 ? - requesting resource stu_disk_srv110161

24/01/2014 01:36:05 ? - requesting resource SVC-BUKUP.moi.ae.NBU_CLIENT.MAXJOBS.MOI-DAG

24/01/2014 01:36:05 ? - requesting resource SVC-BUKUP.moi.ae.NBU_POLICY.MAXJOBS.moi_exchange_monthly

24/01/2014 01:36:05 ? - requesting resource EXCHANGE_RESOLVER.SVC-BUKUP.moi.ae.moi_exchange_monthly.MOI-DAG

24/01/2014 01:36:05 ? - granted resource SVC-BUKUP.moi.ae.NBU_CLIENT.MAXJOBS.MOI-DAG

24/01/2014 01:36:05 ? - granted resource SVC-BUKUP.moi.ae.NBU_POLICY.MAXJOBS.moi_exchange_monthly

24/01/2014 01:36:05 ? - granted resource EXCHANGE_RESOLVER.SVC-BUKUP.moi.ae.moi_exchange_monthly.MOI-DAG

24/01/2014 01:36:05 ? - estimated 0 Kbytes needed

24/01/2014 01:36:05 ? - begin Parent Job

24/01/2014 01:36:05 ? - begin Exchange 14 Resolver, Start Notify Script

24/01/2014 01:36:05 ? - Info RUNCMD(pid=7512) started            

24/01/2014 01:36:05 ? - Info RUNCMD(pid=7512) exiting with status: 0         

Status 0

24/01/2014 01:36:05 ? - end Exchange 14 Resolver, Start Notify Script; elapsed time: 00:00:00

24/01/2014 01:36:05 ? - begin Exchange 14 Resolver, Step By Condition

Status 0

24/01/2014 01:36:05 ? - end Exchange 14 Resolver, Step By Condition; elapsed time: 00:00:00

24/01/2014 01:36:05 ? - begin Exchange 14 Resolver, Read File List

Status 0

24/01/2014 01:36:05 ? - end Exchange 14 Resolver, Read File List; elapsed time: 00:00:00

24/01/2014 01:36:05 ? - begin Exchange 14 Resolver, Resolver Discovery

24/01/2014 01:36:05 ? - started process bpbrm (4292)

24/01/2014 01:36:20 ? - Info bpbrm(pid=4292) MOI-DAG is the host to restore to      

24/01/2014 01:36:20 ? - Info bpbrm(pid=4292) reading file list from client        

24/01/2014 01:37:14 ? - Info bpbrm(pid=4292) client_pid=14972            

24/01/2014 01:37:15 ? - Info bpbrm(pid=4292) from client MOI-DAG: TRV - BPRESOLVER has executed on server (SRV110042)  

24/01/2014 01:38:33 ? - Warning bpbrm(pid=4292) from client MOI-DAG: WRN - Exchange 2010 item <Microsoft Exchange Database Availability Groups:\PFDB> not backed up.  See bpresolver log.

24/01/2014 01:38:33 ? - Warning bpbrm(pid=4292) from client MOI-DAG: WRN - Exchange 2010 item <Microsoft Exchange Database Availability Groups:\PFDB2> not backed up.  See bpresolver log.

24/01/2014 01:38:33 ? - Warning bpbrm(pid=4292) from client MOI-DAG: WRN - Exchange 2010 item <Microsoft Exchange Database Availability Groups:\MBDB-ITSERVICES> not backed up.  See bpresolver log.

24/01/2014 01:38:33 ? - Warning bpbrm(pid=4292) from client MOI-DAG: WRN - Exchange 2010 item <Microsoft Exchange Database Availability Groups:\MBDB-SECTIONS> not backed up.  See bpresolver log.

24/01/2014 01:38:33 ? - Warning bpbrm(pid=4292) from client MOI-DAG: WRN - Exchange 2010 item <Microsoft Exchange Database Availability Groups:\MBDB-MANAGERS> not backed up.  See bpresolver log.

24/01/2014 01:38:33 ? - Warning bpbrm(pid=4292) from client MOI-DAG: WRN - Exchange 2010 item <Microsoft Exchange Database Availability Groups:\MBDB-USERS> not backed up.  See bpresolver log.

24/01/2014 01:38:33 ? - Info bpresolver(pid=14972) done.  status: 2         

24/01/2014 01:38:33 ? - Error bpbrm(pid=4292) no worklist items returned from bpresolver on client  MOI-DAG   

24/01/2014 01:38:33 ? - Info bpresolver(pid=14972) done. status: 2: none of the requested files were backed up  

Status 2

24/01/2014 01:38:33 ? - end Exchange 14 Resolver, Resolver Discovery; elapsed time: 00:02:28

24/01/2014 01:38:33 ? - begin Exchange 14 Resolver, Stop On Error

Status 0

24/01/2014 01:38:33 ? - end Exchange 14 Resolver, Stop On Error; elapsed time: 00:00:00

24/01/2014 01:38:33 ? - begin Exchange 14 Resolver, End Notify Script

24/01/2014 01:38:33 ? - Info RUNCMD(pid=2284) started            

24/01/2014 01:38:34 ? - Info RUNCMD(pid=2284) exiting with status: 0         

Status 0

24/01/2014 01:38:34 ? - end Exchange 14 Resolver, End Notify Script; elapsed time: 00:00:01

Status 2

24/01/2014 01:38:34 ? - end Parent Job; elapsed time: 00:02:29

none of the requested files were backed up(2)

[sriramrane]

Hi Yogesh,

1. Netbackup services running with Domain Admin account and which is having Exchange Enterprise Admin role as well on all the DAG members. Also I have added the account in local admin group on all the exchange nodes too. 

2. File system backup works fine without any issues. Tried couple of times.

3. Have checked Event Logs (System, Secuirty and Application Logs), but there not warning, info or the failed alert listed.

[sriramrane]

Hi Marianne,

Attched failed exchange 2010 DAG backup logs.

[Yogesh9881]

Hi 

Just for testing purpose can we downgrade NBU version on client system ? if possible make it 7.1

[sriramrane]

Not really. All Servers should be running on same version. Only upgrade is possible and downgrade is bit of doubt. Since the Compliance Manager might not agree for the downgrade and Change Management Team will too have issues related to downgrading unless there is a specific reason from Symantec directly stating there is an issue with 7.5.0.6 clients for Exchange 2010 GRT backups. These all are 24/7 production systems, and as per compliance all systems should be running on same level. 

[Mark_Solutions]

You say the NBU services are running with the exchange account .. but do you have the NetBackup Client Service AND the NetBackup Legacy Network Service running using that account? - you need both.

What is in the bpresolver log? - could we see that please - should be on a CAS Server

[sriramrane]

We have NetBackup Client Service, NetBackup Legacy Client Service and NetBackup Legacy Network Service running on Domain Amin account and which has all the required pre-requisites. We have the successful Exchange GRT backup once when we restart the services, and again we will have the same issue of jobs failing with Status code (2) when we run the backup again after first successful backup. But one thing I am thinking which might have gone wrong. 

1) We have NFS Server component enabled on NBU Media/Master Server 

2) We have NFS Client enebled on all the Exchange DAG and HUB/CAS Servers.

But one thing which we havent did, where we do need to have the restart of servers, that is running this command  "sc config portmap start= auto" and then restart the servers. Will this cause the abnormal behaviour of failing the backups after first backups?

[Mark_Solutions]

It could well do - if anything caused it to stop then it would not start up again

You dont need a reboot after issuing the command you can just use sc start portmap to start the service

If the Client for NFS service stops that can also be an issue to get it running again - if that ocurrs you can try the following:

net stop nfsrdr

net start nfsrdr

The Client for NFS service should then be able to start

If this does happen make sure nothing is interferring with it such as Anti Virus or Access Protection - especially if using McAfee

Hope this helps

[Yogesh9881]

As he said., I am wondering about 3rd point.

"3. Have checked Event Logs (System, Secuirty and Application Logs), but there not warning, info or the failed alert listed."

[sriramrane]

I meant there is no any event log/s related to the failed backup jobs in any of the Exchange servers.  In fact I was wondering how I am not able to connect to the database from the selection list at second time once the first full backup is completed with GRT. Its abnormal and not consitent after the first backup. There is no logging related to this activity within Event Logs

[RamNagalla]

DO you have the bpresolver log enabled in the DAG node.. if yes.. please post that log..

if not please enable it and tyr the backup again and post the log.