Exchange mailbox backups hang

[David_Oswell]

Looking to see if there was any resolution to; http://seer.support.veritas.com/docs/266744.htm

For some reason this problem has started for our exchange mailbox backups on 10d:

Remote Agent path: C:\Program Files\VERITAS\Backup Exec\RANT\

File name Modify date & time Size ProdVer FileVer Description
----------------------------- -------------------- ------- -------------- -------------- -----------
bediag.exe 9/21/05 8:16:14 PM 723520 10.1.5629.0 10.1.5629.0 Backup Exec Diagnostic Utility
begather.exe 9/20/05 9:56:40 PM 371264 10.1.5629.0 10.1.5629.0 Backup Exec Gather Utility
belnapi.exe 9/23/05 9:41:42 AM 55360 10.1.5629.0 10.1.5629.0 Backup Exec Notes Agent Process
beremote.exe 9/23/05 9:42:30 AM 512064 10.1.5629.0 10.1.5629.0 Backup Exec Remote Agent for Windows NT/2000
spswrapper.exe 9/21/05 8:10:50 PM 174080 10.1.5629.0 10.1.5629.0 Backup Exec SharePoint Portal Server
vxmon.exe 9/21/05 8:12:06 PM 367680 10.1.5629.0 10.1.5629.0 Remote Agent Monitor
atl71.dll 6/29/05 1:48:58 PM 89088 7.10.3077.0 7.10.3077.0 ATL Module for Windows (Unicode)
bebsdu.dll 9/23/05 9:41:30 AM 138816 10.1.5629.0 10.1.5629.0 Backup Exec Matcher
beclass.dll 9/21/05 8:09:46 PM 522304 10.1.5629.0 10.1.5629.0 Backup Exec Class Library
bedisk.dll 9/23/05 9:42:22 AM 254528 10.1.5629.0 10.1.5629.0 BEDisk
bediskfind.dll 9/21/05 8:11:22 PM 38976 10.1.5629.0 10.1.5629.0 BEDiskFind
bediskfinds.dll 9/21/05 8:11:20 PM 39488 10.1.5629.0 10.1.5629.0 BEDiskFind
bedscomn.dll 9/23/05 9:41:52 AM 70720 10.1.5629.0 10.1.5629.0 Backup Exec Generic Store
bedscps.dll 9/23/05 9:42:10 AM 158784 10.1.5629.0 10.1.5629.0 Backup Exec CPS Agent
bedsdpm.dll 9/23/05 9:42:08 AM 255040 10.1.5629.0 10.1.5629.0 Backup Exec DPM Agent
bedsmbox.dll 9/23/05 9:41:36 AM 250944 10.1.5629.0 10.1.5629.0 Backup Exec Mailbox Store
bedsmdoc.dll 9/23/05 9:41:38 AM 65600 10.1.5629.0 10.1.5629.0 Backup Exec MS Document Store
bedsnote.dll 9/23/05 9:41:40 AM 280128 10.1.5629.0 10.1.5629.0 Backup Exec Notes Store
bedsnt5.dll 9/23/05 9:41:46 AM 629824 10.1.5629.0 10.1.5629.0 Backup Exec NTFS Store
bedsoffhost.dll 9/23/05 9:42:12 AM 237120 10.1.5629.0 10.1.5629.0 Backup Exec off host Data Store
bedsra.dll 9/23/05 9:42:06 AM 20544 10.1.5629.0 10.1.5629.0 Backup Exec Remote Agent Store
bedsshadow.dll 9/23/05 9:41:50 AM 665152 10.1.5629.0 10.1.5629.0 Backup Exec Shadow Copy Store
bedssms.dll 9/23/05 9:41:54 AM 62016 10.1.5629.0 10.1.5629.0 Backup Exec SMS Store
bedssmsp.dll 9/23/05 9:41:56 AM 18496 10.1.5629.0 10.1.5629.0 Backup Exec SIDF Parser
bedssps2.dll 9/23/05 9:41:58 AM 226880 10.1.5629.0 10.1.5629.0 Backup Exec SharePoint V2
bedssql2.dll 9/23/05 9:42:00 AM 296000 10.1.5629.0 10.1.5629.0 Backup Exec SQL Store
bedsupfs.dll 9/23/05 9:42:04 AM 36416 10.1.5629.0 10.1.5629.0 Backup Exec Utility Partition Store
bedsxchg.dll 9/23/05 9:42:02 AM 57920 10.1.5629.0 10.1.5629.0 Backup Exec Exchange Store
bedsxese.dll 9/23/05 9:41:32 AM 92224 10.1.5629.0 10.1.5629.0 Backup Exec Platinum Store
beerrors_de.dll 9/21/05 6:45:38 PM 118848 10.1.5629.0 10.1.5629.0 Backup Exec Error Text Library
beerrors_en.dll 9/21/05 6:45:34 PM 138816 10.1.5629.0 10.1.5629.0 Backup Exec Error Text Library
beerrors_es.dll 9/21/05 6:45:40 PM 115264 10.1.5629.0 10.1.5629.0 Backup Exec Error Text Library
beerrors_fr.dll 9/21/05 6:45:40 PM 117312 10.1.5629.0 10.1.5629.0 Backup Exec Error Text Library
beerrors_it.dll 9/21/05 6:45:42 PM 114752 10.1.5629.0 10.1.5629.0 Backup Exec Error Text Library
beerrors_jp.dll 9/21/05 6:45:46 PM 118336 10.1.5629.0 10.1.5629.0 Backup Exec Error Text Library
beerrors_ko.dll 9/21/05 6:45:44 PM 114240 10.1.5629.0 10.1.5629.0 Backup Exec Error Text Library
beerrors_zh.dll 9/21/05 6:45:36 PM 81984 10.1.5629.0 10.1.5629.0 Backup Exec Error Text Library
beevents_de.dll 9/20/05 9:47:46 PM 81984 10.1.5629.0 10.1.5629.0 Backup Exec Error Text Library
beevents_en.dll 9/20/05 9:47:42 PM 73280 10.1.5629.0 10.1.5629.0 Backup Exec Event Text Library
beevents_es.dll 9/20/05 9:47:48 PM 83008 10.1.5629.0 10.1.5629.0 Backup Exec Error Text Library
beevents_fr.dll 9/20/05 9:47:48 PM 82496 10.1.5629.0 10.1.5629.0 Backup Exec Error Text Library
beevents_it.dll 9/20/05 9:47:50 PM 83008 10.1.5629.0 10.1.5629.0 Backup Exec Error Text Library
beevents_jp.dll 9/20/05 9:47:54 PM 98880 10.1.5629.0 10.1.5629.0 Backup Exec Error Text Library
beevents_ko.dll 9/20/05 9:47:52 PM 98368 10.1.5629.0 10.1.5629.0 Backup Exec Error Text Library
beevents_zh.dll 9/20/05 9:47:44 PM 79936 10.1.5629.0 10.1.5629.0 Backup Exec Error Text Library
bemsdk.dll 9/21/05 8:08:54 PM 1649728 10.1.5629.0 10.1.5629.0 Backup Exec SDK Class Library
benetapi.dll 9/21/05 8:11:24 PM 20544 10.1.5629.0 10.1.5629.0 Backup Exec Communications API
benetutl.dll 9/21/05 8:11:28 PM 17472 10.1.5629.0 10.1.5629.0 Backup Exec Network Communications Utilities
bestdutl.dll 9/21/05 8:11:58 PM 88128 10.1.5629.0 10.1.5629.0 Backup Exec Utilities
bexml.dll 9/21/05 8:09:48 PM 62528 10.1.5629.0 10.1.5629.0 Backup Exec XML Class Library
engine_de.dll 9/21/05 8:11:44 PM 310848 10.1.5629.0 10.1.5629.0 Backup Exec Job Engine
engine_en.dll 9/21/05 8:11:44 PM 265792 10.1.5629.0 10.1.5629.0 Backup Exec Job Engine
engine_es.dll 9/21/05 8:11:46 PM 309312 10.1.5629.0 10.1.5629.0 Backup Exec Job Engine
engine_fr.dll 9/21/05 8:11:48 PM 316992 10.1.5629.0 10.1.5629.0 Backup Exec Job Engine
engine_it.dll 9/21/05 8:11:50 PM 307264 10.1.5629.0 10.1.5629.0 Backup Exec Job Engine
engine_jp.dll 9/21/05 8:11:52 PM 187968 10.1.5629.0 10.1.5629.0 Backup Exec Job Engine
engine_ko.dll 9/21/05 8:11:52 PM 181824 10.1.5629.0 10.1.5629.0 Backup Exec Job Engine
engine_zh.dll 9/21/05 8:11:54 PM 138816 10.1.5629.0 10.1.5629.0 Backup Exec Job Engine
intranw.dll 9/21/05 8:10:22 PM 168000 10.1.5629.0 10.1.5629.0 Backup Exec IntraNW Client Interface
mfc71.dll 6/29/05 1:49:00 PM 1060864 7.10.3077.0 7.10.3077.0 MFCDLL Shared Library - Retail Version
mfc71chs.dll 6/29/05 1:49:00 PM 40960 7.10.3077.0 7.10.3077.0 MFC Language Specific Resources
mfc71cht.dll 6/29/05 1:49:00 PM 45056 7.10.3077.0 7.10.3077.0 MFC Language Specific Resources
mfc71deu.dll 6/29/05 1:49:00 PM 65536 7.10.3077.0 7.10.3077.0 MFC Language Specific Resources
mfc71enu.dll 6/29/05 1:49:00 PM 57344 7.10.3077.0 7.10.3077.0 MFC Language Specific Resources
mfc71esp.dll 6/29/05 1:49:00 PM 61440 7.10.3077.0 7.10.3077.0 MFC Language Specific Resources
mfc71fra.dll 6/29/05 1:49:00 PM 61440 7.10.3077.0 7.10.3077.0 MFC Language Specific Resources
mfc71ita.dll 6/29/05 1:49:00 PM 61440 7.10.3077.0 7.10.3077.0 MFC Language Specific Resources
mfc71jpn.dll 6/29/05 1:49:00 PM 49152 7.10.3077.0 7.10.3077.0 MFC Language Specific Resources
mfc71kor.dll 6/29/05 1:49:00 PM 49152 7.10.3077.0 7.10.3077.0 MFC Language Specific Resources
mfc71u.dll 6/29/05 1:49:00 PM 1047552 7.10.3077.0 7.10.3077.0 MFCDLL Shared Library - Retail Version
msvcp50.dll 6/29/05 1:48:58 PM 565760 5.0.0.7022 5.0.0.7022 Microsoft (R) C++ Runtime Library
msvcp71.dll 6/29/05 1:49:00 PM 499712 7.10.3077.0 7.10.3077.0 Microsoft? C++ Runtime Library
msvcr71.dll 6/29/05 1:49:00 PM 348160 7.10.3052.4 7.10.3052.4 Microsoft? C Runtime Library
ndmpcomm.dll 9/23/05 9:42:14 AM 109632 10.1.5629.0 10.1.5629.0 Backup Exec NDMP Interface Library
ndmpsrvr.dll 9/23/05 9:42:18 AM 1280576 10.1.5629.0 10.1.5629.0 Backup Exec NDMP Server
sigcomp230.dll 7/29/04 3:34:16 PM 51296 2.30.17.0 2.30.17.0 Component Library
vspapi.dll 7/07/05 1:12:10 AM 26184 1.6.0.0 1.6.0.0 Volume Snapshot Provider Interface
vxace.dll 3/27/04 4:44:38 PM 1209496 2.2.0.3 2.2.0.3 ACE
vxmon_de.dll 9/20/05 9:50:32 PM 113728 10.1.5629.0 10.1.5629.0 Remote Agent Monitor User Interface Resources
vxmon_en.dll 9/20/05 9:50:30 PM 112704 10.1.5629.0 10.1.5629.0 Remote Agent Monitor User Interface Resources
vxmon_es.dll 9/20/05 9:50:32 PM 113728 10.1.5629.0 10.1.5629.0 Remote Agent Monitor User Interface Resources
vxmon_fr.dll 9/20/05 9:50:34 PM 113728 10.1.5629.0 10.1.5629.0 Remote Agent Monitor User Interface Resources
vxmon_it.dll 9/20/05 9:50:36 PM 113216 10.1.5629.0 10.1.5629.0 Remote Agent Monitor User Interface Resources
vxmon_jp.dll 9/20/05 9:50:38 PM 111680 10.1.5629.0 10.1.5629.0 Remote Agent Monitor User Interface Resources
vxmon_ko.dll 9/20/05 9:50:38 PM 111168 10.1.5629.0 10.1.5629.0 Remote Agent Monitor User Interface Resources
vxmon_zh.dll 9/20/05 9:50:40 PM 110656 10.1.5629.0 10.1.5629.0 Remote Agent Monitor User Interface Resources

(After last mailbox is backed up)
05/19/06 19:33:14 TF xfer time = 4406 seconds.
05/19/06 19:33:14 WRITE: tpreceive_fail_count = 106
05/19/06 19:33:14 WRITE: waiting_on_buffers_count = 97
05/19/06 19:33:14 WRITE: buffers_written_count = 24994
05/19/06 19:33:14 TF_CloseSet()
05/19/06 19:33:14 FreeFormatEnv( cur_fmt=0 )
05/19/06 19:33:14 Detach from \\TFT-EXCHANGE\Microsoft Exchange Mailboxes

Never completes. There are also no waiting alarts to be responded.

[STeve_O]

Where is you Global Catalog DC server located? Is it at the same site ad the exchange server?

[David_Oswell]

Yep everything is in the same site.

[Steve_TYiybg]

Typically when it hang like that it having issus contacting the DC in you directory access tab in exchange.
I assuming you have Backup Exec 10.1 with SP1 and you have enabled th Bypassmbox properties registry key.

[shweta_rege]

Hello,


As per the information provided, we understand that mailbox backups hang. To fix this problem we suggest you to perform the following.

1. Stop the Removable Storage Media service

2. Check whether all the Backup Exec services on Backup Exec Media server
and remote Exchange Server are running

3. If any antivirus applications are running please stop that application and try performing backup

4. Perform Backup Of mailboxes as a separate backup job and check if you get any errors are problems

Have you selected "Advanced Open File Option" in mailbox backup? If yes, then uncheck this option then run the same backup job again and verify the result.


NOTE : If we do not receive your reply within two business days, this post would be marked assumed answered
and would be moved to answered questions pool.

*************************************************
************************************************

[David_Oswell]

Thanks for the replies,

Steve -
SP1 wasn't installed, going to do the install today and hopefully that fixes things up. What's the bypassmbox reg key do?

shweta-
RSM is running on both the backup & exchange servers, I'll stop it on the exchange server. Not sure if it is needed on the backup server or not? (We don't have it setup to use it for the library)

All services have been running fine. I've been using vxmon on the exchange server to monitor the backups and to log them. No errors occur with the services.

We do have McAfee GroupShield on the exchange server, and have been beating that up due to slow backup speeds. But I believe that is alright now.

The job was setup to backup Mailboxes & public folders in the same job, we'll split them into separate jobs and see if either one completes.

AOFO is not selected.

Will have an update later today.
Thanks,

[Shilpa_pawar_2]

Hi,

On exchange server change the following registry key to 1 and test a new backup:

HKey_Local_Machine > Software > VERITAS > Backup Exec > Engine > Exchange > Bypass MBox properties

NOTE : If we do not receive your reply within two business days, this post would be marked "assumed answered" and would be moved to "answered questions" pool.

[David_Oswell]

We've installed SP1, and disabled the RSM services. No change, still hangs on at the end of the mailbox backup.

Will give the reg edit a try. What information will that exclude?

Also; we have this problem on two exchange servers, one is 2000 and the other 2003.

[David_Oswell]

The reg change seems to work, we got a successful backup of 2 mailboxes. Will see how it does when run for all mailboxes.

Thanks,

[David_Oswell]

The full backup still hangs at the end, the only difference in the jobs that I can see (other then the number of mailboxes) is our full included the public folders. I've removed that from the selection and trying again.

Prior to the last test I had turned the debugging level up to medium;

05/25/06 17:34:39 Detach from \\TFT-EXCHANGE\Microsoft Exchange Mailboxes
(I think the following messages appeared after cancel was selected)
05/25/06 17:34:39 MailBoxDS::Detach: fsh->file_hand deleted
05/25/06 17:34:39 ndmpdBEWSBrowseScanEnd: ERROR: NDMP_NO_ERR
05/25/06 17:34:39 ndmpSendReply: message:0xf317
05/25/06 17:34:39 ndmpFreeMessage: message:0xf317
05/25/06 17:34:58 ndmpProcessRequests
05/25/06 17:34:58 ndmp_recv_msg
05/25/06 17:34:58 ndmpdConnectClose: called.
05/25/06 17:34:58 ndmpClose
05/25/06 17:34:58 ndmpFreeMessage: message:0x902
05/25/06 17:34:58 ndmpFreeMessage: message:0x902
05/25/06 17:34:58 DS_CloseSelectionService
05/25/06 17:34:58 MailBoxDS::RemoveDrive done.
05/25/06 17:34:58 MailBoxDS::RemoveDrive done.
05/25/06 17:34:58 ndmpDestroyConnectionMessage was edited by:
David Oswell

[Tom_Ashton]

Hi,

My mailbox backups are extremely slow. Will this reg key entry speed them up?

The reg key was not present on my Exchange 2003 server. I have created it. Is this OK?

Regards

TomMessage was edited by:
Tom Ashton

[David_Oswell]

So far no success with backing up all the mailboxes. It still hangs after the last mailbox. vxmon shows a backup job is running, but doesn't show any connected media server (under the status tab). It does show all the info while the mailboxes are being processed.

Could the problem be occuring if too many mailboxes are selected to backup?

Any other suggestions?

05/25/06 19:05:19 TF xfer time = 4900 seconds.
05/25/06 19:05:19 WRITE: tpreceive_fail_count = 300
05/25/06 19:05:19 WRITE: waiting_on_buffers_count = 291
05/25/06 19:05:19 WRITE: buffers_written_count = 75865
05/25/06 19:05:19 ndmpdApiLog: version:3
05/25/06 19:05:19 ndmpdApiLog: message_id = 12, log_type = 0, entry_len=17
05/25/06 19:05:19 ndmpSendRequest: message:0xf204
05/25/06 19:05:19 sent seq#: 85180 msg: 0xf204
05/25/06 19:05:19 ndmpdApiLog: version:3
05/25/06 19:05:19 ndmpdApiLog: message_id = 7, log_type = 0, entry_len=4
05/25/06 19:05:19 ndmpSendRequest: message:0xf204
05/25/06 19:05:19 sent seq#: 85181 msg: 0xf204
05/25/06 19:05:19 TF_CloseSet()
05/25/06 19:05:19 FreeFormatEnv( cur_fmt=0 )
05/25/06 19:05:19 Detach from \\TFT-EXCHANGE\Microsoft Exchange Mailboxes

[shweta_rege]

Hello,


- How many mailboxes are selected to backup?




NOTE : If we do not receive your reply within two business days, this post would be marked assumed answered
and would be moved to answered questions pool.

*************************************************
************************************************

[David_Oswell]

Around 100 on the 2000 exchange server (system ones are deselected),
60 on the 2003 exchange server.

The log posts are from the 2000 exchange server.

I did notice with the 2 mailbox backup test it ran much faster then the full. The test ran at about 60-70MB/min, the fulls run at around 10-20mb/min. They start at these speeds and didn't change much. Even when the tested mailboxes were backed up the speed stayed consistent. Not sure if that is any help, or if it is a separate issue.Message was edited by:
David Oswell

[Amruta_Bhide]

Hello David,

- Please try the following Registry change on the exchnage Server as well as the Media Server.

How to enable support for "Restrict Anonymous" when using Backup Exec 8.6, 9.x, and 10.x for Windows Servers
http://support.veritas.com/docs/238618


- You could also try seprating the Backup job.

******************************************************************
*****************************************************************

Note : If we do not receive your reply within two business days, this post would be marked �assumed answered� and would be moved to �answered questions� pool.


Thanks.

[David_Oswell]

Amruta,

We will give the reg change a try.

We have thought about splitting the mailbox job into small chunks to see if that works, but that's a last resort idea at the moment.

As a work around we've been restarting the backup exec agent in the morning when we know the job has hung. This seems to kick the job back into action and completes with the verify (although shows the disconnecting as being a failure condition)

Dave,

[Deepali_Badave]

Hello,

Please keep us updating on this issue.

NOTE : If we do not receive your reply within two business days, this post would be marked assumed answered and would be moved to answered questions pool.

[David_Oswell]

The reg change didn't do anything to solve the problem.
We've scheduled a restart of the backup exec agent, this has allowed the mailbox backups to at least complete with information and not hang everything up.

The 2003 exchange server seems to have kicked in a while ago, not sure what was changed to get that one running. But since we are moving to that server over the next few months and the workaround appears to be working well, I'm not too worried about this now.

Thanks for your help

[priya_khire]

Thanks for your update. Let us know if the issue recurs.

Regards.