Exchange backups are failing with error code 26

[premchand_logan]

Hello,

The exchange backups (DAG) are failing with error code 26

Exchange 2010 sp1

netbackup master : 7.0.1 , Media server and both the exchange nodes are : 7.0.1

Job log info:

25-May-2012 10:20:24 - requesting resource UK_WDR_WDR-PD
25-May-2012 10:20:24 - requesting resource xpback.NBU_CLIENT.MAXJOBS.dag02
25-May-2012 10:20:24 - requesting resource xpback.NBU_POLICY.MAXJOBS.OneIS_DB_PD_WDR_EXCH_DAG
25-May-2012 10:20:24 - requesting resource EXCHANGE_RESOLVER.xpback.OneIS_DB_PD_WDR_EXCH_DAG.dag02
25-May-2012 10:20:24 - granted resource  xpback.NBU_CLIENT.MAXJOBS.dag02
25-May-2012 10:20:24 - granted resource  xpback.NBU_POLICY.MAXJOBS.OneIS_DB_PD_WDR_EXCH_DAG
25-May-2012 10:20:24 - granted resource  EXCHANGE_RESOLVER.xpback.OneIS_DB_PD_WDR_EXCH_DAG.dag02
25-May-2012 10:20:24 - estimated 0 kbytes needed
25-May-2012 10:20:24 - begin Parent Job
25-May-2012 10:20:24 - started process RUNCMD (pid=27615)
25-May-2012 10:20:24 - ended process 0 (pid=27615)
Operation Status: 0
25-May-2012 10:20:24 - end Parent Job; elapsed time 0:00:00
Operation Status: 0
Operation Status: 0
25-May-2012 10:20:26 - Info bpbrm (pid=27625) from client dag02: TRV - BPRESOLVER has executed on server (UK-EX023)
25-May-2012 10:20:24 - started process bpbrm (pid=27625)
25-May-2012 10:20:36 - Error bpbrm (pid=27625) socket read failed: errno = 104 - Connection reset by peer
Operation Status: 26
Operation Status: 0
25-May-2012 10:20:36 - started process RUNCMD (pid=27947)
25-May-2012 10:20:36 - ended process 0 (pid=27947)
Operation Status: 0
Operation Status: 26
client/server handshaking failed  (26)

[Yogesh9881]

Is it new implementation or was it working before  ??

have you tested file level backup for all DAG member hosts ??

[rizwan84tx]

The client name must be DAG virtual name.

Are you backing up the Active or passive nodes?

Make sure that all Mailbox servers in the DAG is added in the preffered server list.

Can you please comment about this host "UK-EX023" ?

[Marianne]

100% sure that all DAG nodes are on 7.0.1?

Problem supposed to be solved with 7.0.1 : http://www.symantec.com/docs/TECH125427

[V4]

agree with marianne... as from logs yo posted..   BAR is showing at 7.0GA..

please verify you have installed 7.0.1 patch on client...

If patch is successfully deployed two additional services would have been added under NBU

NetBackup Legacy client

NetBackup Letgacy Network

pls Check

[premchandgeetha]

Hello,

Yes. Both the DAG nodes are in 7.0.1 . The main thing is that some times the exchange backups for both the nodes(uk-ex023 and uk-ex022) are getting successful and sometimes they are not  ( faiing with EC 26) and the OS backups also same. Sometimes both the nodes' OS backups are getting succesful, sometimes only uk-ex023's OS backups are getting failed. ( ensured that both the services " Netbackup Legacy client service / Netbackup Legacy Network service are available)

and as per your tech note http://www.symantec.com/docs/TECH125427

I am not getting the below error as mentioned in the tech note

For an Exchange 2010 DAG backup attempt, the following messages are logged in the Activity Monitor. (See the detailed status for the backup job.):

Error bpbrm(pid=20332) This type of backup is not supported on this version of Exchange
Error bpbrm(pid=20332) cannot run bpresolver or it failed on client DAG2010001 Status 26

I am getting the below error in the activity monitor :(highlighted below)

05-Sep-2012 19:20:26 - requesting resource UK_WDR_WDR-PD
05-Sep-2012 19:20:26 - requesting resource xpback.NBU_CLIENT.MAXJOBS.dag02
05-Sep-2012 19:20:26 - requesting resource xpback.NBU_POLICY.MAXJOBS.OneIS_DB_PD_WDR_EXCH_DAG
05-Sep-2012 19:20:26 - requesting resource EXCHANGE_RESOLVER.xpback.OneIS_DB_PD_WDR_EXCH_DAG.dag02
05-Sep-2012 19:20:26 - granted resource  xpback.NBU_CLIENT.MAXJOBS.dag02
05-Sep-2012 19:20:26 - granted resource  xpback.NBU_POLICY.MAXJOBS.OneIS_DB_PD_WDR_EXCH_DAG
05-Sep-2012 19:20:26 - granted resource  EXCHANGE_RESOLVER.xpback.OneIS_DB_PD_WDR_EXCH_DAG.dag02
05-Sep-2012 19:20:26 - estimated 49299642 kbytes needed
05-Sep-2012 19:20:26 - begin Parent Job
05-Sep-2012 19:20:26 - started process RUNCMD (pid=12845)
05-Sep-2012 19:20:26 - ended process 0 (pid=12845)
Operation Status: 0
05-Sep-2012 19:20:26 - end Parent Job; elapsed time 0:00:00
Operation Status: 0
Operation Status: 0
05-Sep-2012 19:20:26 - started process bpbrm (pid=12854)
05-Sep-2012 19:20:46 - Error bpbrm (pid=12854) socket read failed: errno = 104 - Connection reset by peer
Operation Status: 26
Operation Status: 0
05-Sep-2012 19:20:46 - started process RUNCMD (pid=13257)
05-Sep-2012 19:20:46 - ended process 0 (pid=13257)
Operation Status: 0
Operation Status: 26
client/server handshaking failed  (26)

sometimes when i tried to open the BAR on DAG02 ( virtual name of uk-ex023), it says the error

like : " Netbackup has stopped working" ( pls refer the attached screen shot for the same)

also when i find the application logs, i get the Application error as below in DAG02 ( virtual name of uk-ex023)

--------------------------------------------------------------------------------------------------------------------------------------

Faulting application name: nbwin.exe, version: 7.0.2010.707, time stamp: 0x4c35851f

Faulting module name: MSVCP80.dll, version: 8.0.50727.4940, time stamp: 0x4ca2b550

Exception code: 0xc0000005

Fault offset: 0x000000000002730a

Faulting process id: 0x2a04

Faulting application start time: 0x01cd8cdb76bf32bd

Faulting application path: C:\Program Files\Veritas\NetBackup\bin\nbwin.exe

Faulting module path: C:\Windows\WinSxS\amd64_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4940_none_88df89932faf0bf6\MSVCP80.dll

Report Id: bef71dbb-f8ce-11e1-9beb-78e7d1e1aabc

--------------------------------------------------------------------------------------------------------------------------------------------

Faulting application name: bpresolver.exe, version: 7.0.2010.707, time stamp: 0x4c3584ca

Faulting module name: MSVCP80.dll, version: 8.0.50727.4940, time stamp: 0x4ca2b550

Exception code: 0xc0000005

Fault offset: 0x000000000002730a

Faulting process id: 0x27fc

Faulting application start time: 0x01cd8cdac0201c4d

Faulting application path: C:\Program Files\Veritas\NetBackup\bin\bpresolver.exe

Faulting module path: C:\Windows\WinSxS\amd64_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4940_none_88df89932faf0bf6\MSVCP80.dll

Report Id: 050639a1-f8ce-11e1-9beb-78e7d1e1aabc

    -----------------------------------------------------------------------------------------------------------------------------------------------

Faulting application name: bpbkar32.exe, version: 7.0.2010.707, time stamp: 0x4c475004

Faulting module name: MSVCP80.dll, version: 8.0.50727.4940, time stamp: 0x4ca2b550

Exception code: 0xc0000005

Fault offset: 0x000000000002730a

Faulting process id: 0x1c8

Faulting application start time: 0x01cd8cdc156a5f0d

Faulting application path: C:\Program Files\Veritas\NetBackup\bin\bpbkar32.exe

Faulting module path: C:\Windows\WinSxS\amd64_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4940_none_88df89932faf0bf6\MSVCP80.dll

Report Id: 59eee3af-f8cf-11e1-9beb-78e7d1e1aabc

-------------------------------------------------------------------------------------------------------------------------------------------------

Let me know if you need any other logs / info to resolve this issue.

[premchandgeetha]

Forgot to mention that some times the OS backups for UK-EX023 are getting failed with EC 13. Also Netbackup client version has been reinstalled on DAG02 . still I am facing the same issue.

as below

06-Sep-2012 00:58:00 - requesting resource UK_WDR_WDR-PD
06-Sep-2012 00:58:00 - requesting resource xpback.NBU_CLIENT.MAXJOBS.uk-ex023
06-Sep-2012 00:58:00 - requesting resource xpback.NBU_POLICY.MAXJOBS.OneIS_OS_PD_BDR_EXCH
06-Sep-2012 00:58:02 - granted resource  xpback.NBU_CLIENT.MAXJOBS.uk-ex023
06-Sep-2012 00:58:02 - granted resource  xpback.NBU_POLICY.MAXJOBS.OneIS_OS_PD_BDR_EXCH
06-Sep-2012 00:58:02 - granted resource  MediaID=@aaaa9;DiskVolume=PureDiskVolume;DiskPool=UK-WDR1-PD_Pool;Path=PureDiskVolume;StorageServer=...
06-Sep-2012 00:58:02 - granted resource  UK_WDR_WDR-PD
06-Sep-2012 00:58:02 - estimated 15301700 kbytes needed
06-Sep-2012 00:58:03 - started process bpbrm (pid=1236)
06-Sep-2012 00:58:06 - connecting
06-Sep-2012 00:58:08 - connected; connect time: 0:00:00
06-Sep-2012 01:00:38 - Error bpbrm (pid=1236) socket read failed, An existing connection was forcibly closed by the remote host.  (10054)
06-Sep-2012 00:58:17 - end writing
file read failed  (13)
 

[V4]

Not sure if it helps, But worth give try

https://www-secure.symantec.com/connect/videos/exchange-2010-sp2-grt-netbackup-75

Refer the transcript from above silent video, though it's for NBU 7.5 and ex2k10 but still applies to NBU 7.x family

Make sure you get it through

i had recently encountered similar of yours, where everything was LOOKING AS IT SHOULD BE (same as mentioned under transcript) but still backups  were failing with status 13. everylogs was still pointing to same reason access issue though had all the required privileges. Issue solved after weekend and turned out it was AD replication which took time to update the changes (privilege updates)

There after we tried changing account to local system account that too did worked.

during entire backup process this simply slipped of us that customer had 8 Domain controllers which must be at same level hence no choice till wait to get it replicated. Check if it's ur case too

[premchandgeetha]

Hello Speedfreak,

 Actually this issue started 3 months before. Symantec created a work around policy , where in the policy they mentioned : Database backup source: "Active copy only" and the server UK-EX023 ( problematic) is included in the preferred server list. refer the attached screen shot ( uk-ex022 exchange backups were working fine at that time, 3 dbs are mounted in ex022 and the rest of 3 dbs are mounted in ex023). so the exchange backups was getting successful for ex022 from the original policy and the exchange backups were getting successful for ex023 from the work around policy, which was created  by symantec.

Now some two weeks before there was three security patches were installed on both the nodes, from that day even the work around policy also started to failed on both the nodes.

I changed the account to local system and tried still backups are failing with EC 13 for OS and EC 26 for exchange backups.

Let me know if you need any other info / logs

[V4]

did you tried isolating it , by removing thos security patches.. were thee patches tested in UAT before prod ?

Try if it helps

[premchandgeetha]

No. We did not try isolating it, since it needs so many procedure ( reverting the security patch changes), also did not test in UAt before prod.  Any EEB available for this issue ?

[V4]

can u reconfirm with your messaging team on SP level on exchange , is it SP1 or SP2.....