05-25-2012 03:45 AM
Hello,
The exchange backups (DAG) are failing with error code 26
Exchange 2010 sp1
netbackup master : 7.0.1 , Media server and both the exchange nodes are : 7.0.1
Job log info:
25-May-2012 10:20:24 - requesting resource UK_WDR_WDR-PD
25-May-2012 10:20:24 - requesting resource xpback.NBU_CLIENT.MAXJOBS.dag02
25-May-2012 10:20:24 - requesting resource xpback.NBU_POLICY.MAXJOBS.OneIS_DB_PD_WDR_EXCH_DAG
25-May-2012 10:20:24 - requesting resource EXCHANGE_RESOLVER.xpback.OneIS_DB_PD_WDR_EXCH_DAG.dag02
25-May-2012 10:20:24 - granted resource xpback.NBU_CLIENT.MAXJOBS.dag02
25-May-2012 10:20:24 - granted resource xpback.NBU_POLICY.MAXJOBS.OneIS_DB_PD_WDR_EXCH_DAG
25-May-2012 10:20:24 - granted resource EXCHANGE_RESOLVER.xpback.OneIS_DB_PD_WDR_EXCH_DAG.dag02
25-May-2012 10:20:24 - estimated 0 kbytes needed
25-May-2012 10:20:24 - begin Parent Job
25-May-2012 10:20:24 - started process RUNCMD (pid=27615)
25-May-2012 10:20:24 - ended process 0 (pid=27615)
Operation Status: 0
25-May-2012 10:20:24 - end Parent Job; elapsed time 0:00:00
Operation Status: 0
Operation Status: 0
25-May-2012 10:20:26 - Info bpbrm (pid=27625) from client dag02: TRV - BPRESOLVER has executed on server (UK-EX023)
25-May-2012 10:20:24 - started process bpbrm (pid=27625)
25-May-2012 10:20:36 - Error bpbrm (pid=27625) socket read failed: errno = 104 - Connection reset by peer
Operation Status: 26
Operation Status: 0
25-May-2012 10:20:36 - started process RUNCMD (pid=27947)
25-May-2012 10:20:36 - ended process 0 (pid=27947)
Operation Status: 0
Operation Status: 26
client/server handshaking failed (26)
05-25-2012 05:15 AM
Is it new implementation or was it working before ??
have you tested file level backup for all DAG member hosts ??
05-25-2012 05:54 AM
The client name must be DAG virtual name.
Are you backing up the Active or passive nodes?
Make sure that all Mailbox servers in the DAG is added in the preffered server list.
Can you please comment about this host "UK-EX023" ?
05-25-2012 11:54 AM
100% sure that all DAG nodes are on 7.0.1?
Problem supposed to be solved with 7.0.1 : http://www.symantec.com/docs/TECH125427
05-26-2012 08:54 AM
agree with marianne... as from logs yo posted.. BAR is showing at 7.0GA..
please verify you have installed 7.0.1 patch on client...
If patch is successfully deployed two additional services would have been added under NBU
NetBackup Legacy client
NetBackup Letgacy Network
pls Check
09-07-2012 04:37 AM
Hello,
Yes. Both the DAG nodes are in 7.0.1 . The main thing is that some times the exchange backups for both the nodes(uk-ex023 and uk-ex022) are getting successful and sometimes they are not ( faiing with EC 26) and the OS backups also same. Sometimes both the nodes' OS backups are getting succesful, sometimes only uk-ex023's OS backups are getting failed. ( ensured that both the services " Netbackup Legacy client service / Netbackup Legacy Network service are available)
and as per your tech note http://www.symantec.com/docs/TECH125427
I am not getting the below error as mentioned in the tech note
For an Exchange 2010 DAG backup attempt, the following messages are logged in the Activity Monitor. (See the detailed status for the backup job.):
Error bpbrm(pid=20332) This type of backup is not supported on this version of Exchange
Error bpbrm(pid=20332) cannot run bpresolver or it failed on client DAG2010001 Status 26
I am getting the below error in the activity monitor :(highlighted below)
05-Sep-2012 19:20:26 - requesting resource UK_WDR_WDR-PD
05-Sep-2012 19:20:26 - requesting resource xpback.NBU_CLIENT.MAXJOBS.dag02
05-Sep-2012 19:20:26 - requesting resource xpback.NBU_POLICY.MAXJOBS.OneIS_DB_PD_WDR_EXCH_DAG
05-Sep-2012 19:20:26 - requesting resource EXCHANGE_RESOLVER.xpback.OneIS_DB_PD_WDR_EXCH_DAG.dag02
05-Sep-2012 19:20:26 - granted resource xpback.NBU_CLIENT.MAXJOBS.dag02
05-Sep-2012 19:20:26 - granted resource xpback.NBU_POLICY.MAXJOBS.OneIS_DB_PD_WDR_EXCH_DAG
05-Sep-2012 19:20:26 - granted resource EXCHANGE_RESOLVER.xpback.OneIS_DB_PD_WDR_EXCH_DAG.dag02
05-Sep-2012 19:20:26 - estimated 49299642 kbytes needed
05-Sep-2012 19:20:26 - begin Parent Job
05-Sep-2012 19:20:26 - started process RUNCMD (pid=12845)
05-Sep-2012 19:20:26 - ended process 0 (pid=12845)
Operation Status: 0
05-Sep-2012 19:20:26 - end Parent Job; elapsed time 0:00:00
Operation Status: 0
Operation Status: 0
05-Sep-2012 19:20:26 - started process bpbrm (pid=12854)
05-Sep-2012 19:20:46 - Error bpbrm (pid=12854) socket read failed: errno = 104 - Connection reset by peer
Operation Status: 26
Operation Status: 0
05-Sep-2012 19:20:46 - started process RUNCMD (pid=13257)
05-Sep-2012 19:20:46 - ended process 0 (pid=13257)
Operation Status: 0
Operation Status: 26
client/server handshaking failed (26)
sometimes when i tried to open the BAR on DAG02 ( virtual name of uk-ex023), it says the error
like : " Netbackup has stopped working" ( pls refer the attached screen shot for the same)
also when i find the application logs, i get the Application error as below in DAG02 ( virtual name of uk-ex023)
--------------------------------------------------------------------------------------------------------------------------------------
Faulting application name: nbwin.exe, version: 7.0.2010.707, time stamp: 0x4c35851f
Faulting module name: MSVCP80.dll, version: 8.0.50727.4940, time stamp: 0x4ca2b550
Exception code: 0xc0000005
Fault offset: 0x000000000002730a
Faulting process id: 0x2a04
Faulting application start time: 0x01cd8cdb76bf32bd
Faulting application path: C:\Program Files\Veritas\NetBackup\bin\nbwin.exe
Faulting module path: C:\Windows\WinSxS\amd64_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4940_none_88df89932faf0bf6\MSVCP80.dll
Report Id: bef71dbb-f8ce-11e1-9beb-78e7d1e1aabc
--------------------------------------------------------------------------------------------------------------------------------------------
Faulting application name: bpresolver.exe, version: 7.0.2010.707, time stamp: 0x4c3584ca
Faulting module name: MSVCP80.dll, version: 8.0.50727.4940, time stamp: 0x4ca2b550
Exception code: 0xc0000005
Fault offset: 0x000000000002730a
Faulting process id: 0x27fc
Faulting application start time: 0x01cd8cdac0201c4d
Faulting application path: C:\Program Files\Veritas\NetBackup\bin\bpresolver.exe
Faulting module path: C:\Windows\WinSxS\amd64_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4940_none_88df89932faf0bf6\MSVCP80.dll
Report Id: 050639a1-f8ce-11e1-9beb-78e7d1e1aabc
-----------------------------------------------------------------------------------------------------------------------------------------------
Faulting application name: bpbkar32.exe, version: 7.0.2010.707, time stamp: 0x4c475004
Faulting module name: MSVCP80.dll, version: 8.0.50727.4940, time stamp: 0x4ca2b550
Exception code: 0xc0000005
Fault offset: 0x000000000002730a
Faulting process id: 0x1c8
Faulting application start time: 0x01cd8cdc156a5f0d
Faulting application path: C:\Program Files\Veritas\NetBackup\bin\bpbkar32.exe
Faulting module path: C:\Windows\WinSxS\amd64_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4940_none_88df89932faf0bf6\MSVCP80.dll
Report Id: 59eee3af-f8cf-11e1-9beb-78e7d1e1aabc
-------------------------------------------------------------------------------------------------------------------------------------------------
Let me know if you need any other logs / info to resolve this issue.
09-07-2012 04:43 AM
Forgot to mention that some times the OS backups for UK-EX023 are getting failed with EC 13. Also Netbackup client version has been reinstalled on DAG02 . still I am facing the same issue.
as below
06-Sep-2012 00:58:00 - requesting resource UK_WDR_WDR-PD
06-Sep-2012 00:58:00 - requesting resource xpback.NBU_CLIENT.MAXJOBS.uk-ex023
06-Sep-2012 00:58:00 - requesting resource xpback.NBU_POLICY.MAXJOBS.OneIS_OS_PD_BDR_EXCH
06-Sep-2012 00:58:02 - granted resource xpback.NBU_CLIENT.MAXJOBS.uk-ex023
06-Sep-2012 00:58:02 - granted resource xpback.NBU_POLICY.MAXJOBS.OneIS_OS_PD_BDR_EXCH
06-Sep-2012 00:58:02 - granted resource MediaID=@aaaa9;DiskVolume=PureDiskVolume;DiskPool=UK-WDR1-PD_Pool;Path=PureDiskVolume;StorageServer=...
06-Sep-2012 00:58:02 - granted resource UK_WDR_WDR-PD
06-Sep-2012 00:58:02 - estimated 15301700 kbytes needed
06-Sep-2012 00:58:03 - started process bpbrm (pid=1236)
06-Sep-2012 00:58:06 - connecting
06-Sep-2012 00:58:08 - connected; connect time: 0:00:00
06-Sep-2012 01:00:38 - Error bpbrm (pid=1236) socket read failed, An existing connection was forcibly closed by the remote host. (10054)
06-Sep-2012 00:58:17 - end writing
file read failed (13)
09-07-2012 08:55 AM
Not sure if it helps, But worth give try
https://www-secure.symantec.com/connect/videos/exchange-2010-sp2-grt-netbackup-75
Refer the transcript from above silent video, though it's for NBU 7.5 and ex2k10 but still applies to NBU 7.x family
Make sure you get it through
i had recently encountered similar of yours, where everything was LOOKING AS IT SHOULD BE (same as mentioned under transcript) but still backups were failing with status 13. everylogs was still pointing to same reason access issue though had all the required privileges. Issue solved after weekend and turned out it was AD replication which took time to update the changes (privilege updates)
There after we tried changing account to local system account that too did worked.
during entire backup process this simply slipped of us that customer had 8 Domain controllers which must be at same level hence no choice till wait to get it replicated. Check if it's ur case too
09-07-2012 09:55 AM
Hello Speedfreak,
Actually this issue started 3 months before. Symantec created a work around policy , where in the policy they mentioned : Database backup source: "Active copy only" and the server UK-EX023 ( problematic) is included in the preferred server list. refer the attached screen shot ( uk-ex022 exchange backups were working fine at that time, 3 dbs are mounted in ex022 and the rest of 3 dbs are mounted in ex023). so the exchange backups was getting successful for ex022 from the original policy and the exchange backups were getting successful for ex023 from the work around policy, which was created by symantec.
Now some two weeks before there was three security patches were installed on both the nodes, from that day even the work around policy also started to failed on both the nodes.
I changed the account to local system and tried still backups are failing with EC 13 for OS and EC 26 for exchange backups.
Let me know if you need any other info / logs
09-07-2012 10:05 AM
did you tried isolating it , by removing thos security patches.. were thee patches tested in UAT before prod ?
Try if it helps
09-07-2012 10:10 AM
No. We did not try isolating it, since it needs so many procedure ( reverting the security patch changes), also did not test in UAt before prod. Any EEB available for this issue ?
09-07-2012 08:35 PM
can u reconfirm with your messaging team on SP level on exchange , is it SP1 or SP2.....