Visual C++ Runtime error in beremote.exe on Exchange Server

[cypherx]

Hello, I had this problem before, there might be an old thread about it but we never came up with a solution since the problem is so intermittent it's hard to find a reason for what causes it to happen.

Situation:

On our Windows 2003 server we have Backup Exec 10d.  It is scheduled to do three backups.

1st, using the exchange add-on it does brick level backups of our MS Exchange 2003 server.

2nd, it does a full system backup of our Windows 2003 server.

Last, it does a full system backup of our Websense/Fax server.

Now each job has been monitored to see how long it takes, so we set time limits for each one.  For example, if the Exchange server takes longer than 4 hours to complete, cancel the job.  We don't want to run out of time because when you schedule jobs you can only provide a 'window of opportunity' as I like to call it.  There's no option to 'start this job after previous job regardless of time'.  Very limited for software that costs over $1000 don't you think?

Anyway here's the error that we see on our exchange server in the morning...

The title bar says: Microsoft Visual C++ Runtime Library

There's a red X, and it says Runtime Error!

Program: C:\Program Files\VERITAS\BackupExec\RANT\beremote.exe

This application has requested the Runtime to terminate it in an unusual way.  Please contact the applications support team for more information.

[OK]

Now on the Backup Exec 10d server console we see:

Optiofax Full System Save   Missed    Elapsed Time 0:00:00 Byte Count: 0

Full System Save(Nightly)    Missed    Elapsed Time 0:00:00 Byte Count: 0

Exchange Save     HP1         Backup   Failed   Elapsed Time: 13:38:24  Byte Count: 16,002,008,824

On a good day we should see:

Optiofax Full System Save   Completed Elapsed Time 5:21:20 Byte Count: 16,520,617,250

Full System Save(Nightly)    Completed Elapsed Time 5:31:15 Byte Count: 87,370,362,037

Exchange Save                    Completed Elapsed Time: 3:26:56  Byte Count: 40,696,217,511

So this Visual C++ error in beremote.exe is causing all of our backups to FAIL.  This is NOT ACCEPTABLE.  It doesn’t matter that I have Exchange set to cancel after 4 hours.  Look at the log….  13:38:24!  That’s WAY more than 4 hours.  I don’t know what kind of math Symantec is using to determine job run times. 

So to recap…

We come into work in the morning, see that the Exchange save is still running, but no byte count is happening, and we go over to the Exchange server and find this visual c++ error in beremote.exe.  We click OK on this error, then back on the BackupExec console we see the Exchange save fail “Communication Error”, and our next two backups go into a missed status, since were way beyond the time frame for them to run.

Can someone please help with this situation?  I greatly appreciate your efforts.  I hope someone out there is up to this challenge.

Details for our Exchange save:

Resource order:

First Storage Group

Microsoft Exchange Public Folders

Microsoft Exchange Mailboxes

Microsoft SQL Server

Resource Credentials:

All Successful

Priority and Availability:

Medium

Cancel backup job if not completed within 4 hours

Device and Media:

All Devices

Media Set 1

Overwrite Media

General Settings:

Exchange Save

Verify after backup completes

Hardware Compression (if available, otherwise none)

Advanced:

Back up files and directories by following junction points

Set Remote Agent priority: Normal Priority

Open file backup when Advanced Open File Option is not used: With a lock

Microsoft Exchange:

Exchange Server backup method: Full – Database & Logs (flush committed logs)

Mailbox backup method: Copy – Back up messages

Schedule: Run according to schedule.

S, M, T, W, Th, Fr

Start no earlier than: 4:45 PM

And no later than 5:00 PM

Do not delete the job

Error event logs on our Exchange Server:

Only error I see is once we hit the OK on the C++ runtime error.

Under System

Information 7/12/2007    5:12:22 PM  Application Popup   Event 26

Application popup: Microsoft Visual C++ Runtime Library: Runtime Error!

Program: C:\Program Files\VERITAS\Backup Exec\RANT\beremote.exe

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

Error 7/13/2007    6:09:31 AM  Service Control Manager  Event 7031

The Backup Exec Remote Agent for Windows Servers service terminated unexpectedly.  It has done this 1 time.  The following corrective action will be taken in 60000 milliseconds.  Restart the service.

[cypherx]

Any help on this issue please?  It just happened again today.  All of our backups were MISSED because the Exchange agent had an error.

"Microsoft Visual C++ Runtime Library - Runtime Error!  Program C:\Program Files\Veritas\Backup Exec\RANT\beremote.exe"

Once you click OK on this, our primary server with Backup Exec 10d shows Communication error for this backup, and then the two other backups scheduled after this fail because they are 'missed'.

Since my last post I've upgraded to Backup Exec 10d Service Pack 3, applied Hotfix 48 and 49, and we've purchased the AOFO option (don't use it on Exchange though).  After all of this I re-installed the Backup Exec remote agent to our Exchange server and Websense server.

Please help, me and my boss are at our wits end.  We both cannot imagine for the insane cost of this software that it does not have 100% reliability. 

I now have to run at least or Domain Controller data and Exchange backups now, which hogs up the tape drive.  Hopefully we won't need to restore a file for a user today.

It is very important for us since we are a production facility to have 100% accurate backups.

[cypherx]

Hey, does anyone even read these forums???

This issue just happend again today.  Someone on another site suggested redownloading the Microsoft Visual C++ redistributable package, but then another user said that was a bad idea.  The problem is with Symantec and their inability to create reliable software.

I'm about ready to rip out everything symantec in this company and never use it again in my lifetime.

Way to go symantec, way to step up, be a man and assist a user with your unreliable software.

[cypherx]

This C++ error happened again today on the Exchange save.  Doesn't Symantec hire competent C++ programmers?  Don't they look over the C++ syntax to ensure it is correct before releasing a program to the public?

The fact that Symantec has given me no help at all regarding this issue does not make them look good in my eyes.

I will NEVER use another symantec application again, unless they can resolve this.  I will guarantee that.

ARE YOU LISTENING???

Come on, prove me wrong Symantec.  Show me that you really do care about the quality of your products.  I dare you.

[Ben_L_]

Cypherx,

First off I have to mention that the forums are peer-to-peer help, these forums are not meant to an enterprise support solution. With a problem such as this one, you may want to open a support case so that we can look into the problem.  A support person such as myself, will post here from time to time, but that is only when we are available to do so.   Here is the link for our support contacts. http://www.symantec.com/enterprise/services/category.jsp?pcid=support_services

Now back to the problem at hand with the C++ errors you are getting. Can you give me the version number from the properties of beremote.exe on the exchange server?

How far into the job does it get before you receive this error?  Since I noticed you said you were backing up the Information Store, Mailboxes, Public Folders and SQL, I need to know where it is in the job when it crashes.

Also can you check to see if a crash dump is generated on the server?  I really doubt one is, but it could prove helpful.

Regards,

[cypherx]

Thanks Ben for the information.

The file version for beremote.exe on the Exchange server is 10.1.5629

The Exchange backup started at 3:28 AM and at 3:55 AM I see an "Application Popup" event in the System log on the Exchange server.
Event ID:26
Application popup: Microsoft Visual C++ Runtime Library : Runtime Error!

Program: C:\Program Files\VERITAS\Backup Exec\RANT\beremote.exe


This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

Now the person comes in around 5:30 ~ 6 AM but in this case he didn't go back to check on the servers until 6:55 where he saw the error message on the exchange server.  As soon as he clicks OK the following shows in the System event viewer:
Event ID: 7031
The Backup Exec Remote Agent for Windows Servers service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 60000 milliseconds: Restart the service.

Then at 6 :56 AM The Backup Exec Remote Agent for Windows Servers service entered the running state and on our master server we have an error stating the backup failed due to a communications problem.

This is what happens every time.  It's completly random.  This year it happend on the following dates (from latest to oldest occurance) 8/29, 8/16, 8/7, 7/25, 7/12, 5/21, 4/17, 3/21, 2/18

We have Backup Exec 10d with SP3 installed.  When the problem occurs, Backup Exec's job log indicates about 16 GB of data saved.  On a good backup we have 33.5 GB of data saved.

The Resource Order is:
First Storage Group
Microsoft Exchange Public Folders
Microsoft Exchange Mailboxes
Microsoft SQL Server

I have no other crash dumps of any kind.  The error is pretty vague.

Here's some info from the job log on 8/29:
First Storage Group
Backed up 2 Exchange Server stores
Backed up 1 Exchange Server logs
Processed 16,430,310,190 bytes in  24 minutes and  5 seconds.
Throughput rate: 651 MB/min

Microsoft Exchange Public Folders
Backed up 1127 mail messages in 5 folders in 0 mailbox(es)
Processed 11,577,119 bytes in  24 seconds.
Throughput rate: 27.6 MB/min

Microsoft Exchange Mailboxes
V-79-57344-65072 - The Exchange Server is not responding. Backup set canceled.
Processed 0 bytes in  3 hours,  2 minutes, and  30 seconds.
Throughput rate: 0.000 MB/min

Job Operation - Verify
Verified 2 Exchange Server stores
Processed 16,430,310,190 bytes in  11 minutes and  5 seconds.
Throughput rate: 1414 MB/min

Microsoft Exchange Public Folders
Verified 1127 mail messages in 5 folders in 0 mailbox(es)
0 files were different.
Processed 11,577,119 bytes in  1 second.
Throughput rate: 662 MB/min

Job Completion Status
Job ended: Wednesday, August 29, 2007 at 7:13:08 AM
Completed status: Failed
Final error: 0xe000fe30 - A communications failure has occurred.
Final error category: Server Errors

For additional information regarding this error refer to link V-79-57344-65072

Errors:
Backup- Microsoft Exchange Mailboxes V-79-57344-65072 - The Exchange Server is not responding. Backup set canceled.
V-79-57344-65072 - The database server is not responding. Backup set canceled.

Hope this helps.



Message Edited by cypherx on 08-30-2007 12:46 PM

[Rainbow]

Hi,

We have the same problem with Backup Exec v9.1. I posted a query on the Backup Exec v9.1 forum on 30.08.07, but thus far I've not had a single reply.

Not sure if this helps in any way, but our problem started after installing MS Exchange SP2. Prior to that everything worked ok. With the help of our usual support, we have narrowed down the problem to the MS Mailboxes. By unchecking these our backups run fine. Not ideal, but I understand that mailboxes are backed up collectively from within Information Store, so we do still have some protection.

Tried to open a case with Symantec, but only partial support is available for v9.1 (no new patches, fixes, etc).

Will keep an eye on your thread. If you do find a solution please post either here or on mine if possible.

Good luck! 

[Ben_L_]

Cypherx,

In light of the information from Rainbow and it does appear that your job is failing at the same place, can you remove the Mailbox selection from your backups and run the job?  From what I can see with what you have posted so far it will more then likely be a good backup.


Regards,

[cypherx]

Ben,

Just a few questions on your response.  If I remove the Mailbox selection from this backup job, it would have to be a permanant change, as I never know which night the backup will fail with a C++ error.  So if I remove this selection, I will still have good full backups?  How do you restore say, one single user's inbox?  I thought I needed to back up the mailboxes so that I can do brick level restores incase an end user deletes all of their e-mail by accident.  We've actually had this happen before believe it or not.

Basically, my question is what is the difference between Symantec's Mailbox selections, and First Storage Group?

Thanks for your response.

[Ben_L_]

If you need to restore a single users mailbox you can still do so if you do not use Mailbox Level backups.  Mailbox level backups are just there for convenience to restore them easier.  You can restore a users mailbox by restoring the Information Store to a Recovery Storage Group, then use EXMerge or ESEUtil to either export the data to a PST or Merge the data back into the live Information Store.

Here are a couple technotes from Microsoft on the Recovery Storage Group.
http://support.microsoft.com/kb/824126
http://support.microsoft.com/kb/823176


Regards,

[cypherx]

Ok I guess there's that suggestion or another one from Brajendra who handled my support case.  His suggestion was:

1. Connect to the Exchange Server and Open regedt32 on the Server.

2. Browse to the registry key given below:

HKey_Local_Machine > Software > VERITAS > Backup Exec > Engine > Exchange > Bypass MBox properties

3. Change the above key's value to 1.

Note: Enabling this key will disable the auto-creation of Exchange mailboxes during a mailbox restore

This is one way to fix it.  He also said he could not find this issue for BackupExec 11.  So we'll keep monitoring the problem and see which route we want to go.

If I bypass the mbox properties he said I would have to create each individual mailbox before performing a restore. 

My questions about this solution:

1)  What if I'm restoring deleted mail to a mailbox that already exists?  Say Joe User deletes an important attachment and empties his deleted items folder.  He realizes this too late and gives me a call asking if I can restore the file.  Would the procedure be any different?

2)  Say its a full fledged disaster recovery situation.  If I restore the servers hard drive image from our weekly acronis backup, I should still be able to layer on BackupExec's exchange mailboxes as long as new users were not created since the last acronis hard drive image.  It's not like I'll reimage a new raid and have to create 50 mailboxes again, the image is updated every Saturday.

Thanks for your assistance.  After these two questions we should be about done with the issue, unless anything else comes to mind.

[Ben_L_]

1)  What if I'm restoring deleted mail to a mailbox that already exists?  Say Joe User deletes an important attachment and empties his deleted items folder.  He realizes this too late and gives me a call asking if I can restore the file.  Would the procedure be any different?

The registry change would not have any effect on this as the mailbox is already created.  But you may want to have him go to Tools -> Recover Deleted Items in outlook to see if the message he is looking for is in there. That only works if you have exchange configured to save deleted items for a certain amount of time though.  If the option is greyed out try clicking on the Deleted Items folder then go back to the tools menu.

2)  Say its a full fledged disaster recovery situation.  If I restore the servers hard drive image from our weekly acronis backup, I should still be able to layer on BackupExec's exchange mailboxes as long as new users were not created since the last acronis hard drive image.  It's not like I'll reimage a new raid and have to create 50 mailboxes again, the image is updated every Saturday.

If you are doing a full DR you are going to be restoring the entire information store and not mailbox level data.  When restoring the entire store it does not have to recreate the accounts as they already exist in the store that is being restored.

[cypherx]

Thanks Ben,

It sounds like the registry key change described in the workaround may fit our needs.

I want to appologize for my harsh posts earlier in this thread.  I was so frusterated when I'd come to work in the morning and the first words out of my coworkers mouth was "The exchange save failed with a C++ error again".

The support case process was easy to open.  I did type a novel the first time, took so long that my login to support center timed out and I lost everything.  Well next time I know to type up my case in word and cut and paste when ready to post.

Again thanks for your participation.  I'll take your advice in the future about these forums, and again I apologize for getting heated.

[cypherx]

The problem happend again today.  I WILL be turning on this registry key today.

I wonder why the error doesn't show us the C++ line that is causing the problem?  If it gave just a little more information it would be easy to solve. 

[cypherx]

Followed instructions from our original case # 320-065-707

 
We made the following changes on the registry of the Exchange 2003 server.

 1. Connect to the Exchange Server and Open regedt32 on the Server.

 2. Browse to the registry key given below:
 HKey_Local_Machine > Software > VERITAS > Backup Exec > Engine > Exchange > Bypass MBox properties

 3. Change the above key's value to 1.

 
Today we just received another one of Symantec's Brilliant C++ errors causing the backup to fail. 

THIS IS UNACCEPTABLE.  We rely on good nightly backups.  I urge you to update the beremote.exe program or send us updated C++ libraries that are up to Symantec's standards.

 
Here's what the System event logs are saying on the Exchange server:

 9/19/2007 5:15:51 AM EventID: 26
Application popup: Microsoft Visual C++ Runtime Library : Runtime Error!
Program: C:\Program Files\VERITAS\Backup Exec\RANT\beremote.exe
This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

At 9:06 AM we finally had time to go into the server room and notice this application popup.  Upon clicking OK on the error dialog the next items appear in the System event log:

 9/19/2007 9:06:07 AM
The Backup Exec Remote Agent for Windows Servers service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 60000 milliseconds: Restart the service.

9/19/2007 9:07:07 AM
The Network Load Balancing service failed to start due to the following error:
The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.

9/19/2007 9:07:10 AM
The Backup Exec Remote Agent for Windows Servers service entered the running state.

Now the Exchange backup as shown on our Windows 2003 primary domain controller is showing that the backup is verifying.  As it gets to 16 GB it fails.  It appears as if the C++ error occurs when backing up individual mailboxes.  Now we have to restart the backup and tie up our tape drive for the next 3 hours.  God hope no one want's us to restore anything in that time.

Here's what makes is so hard to troubleshoot...  It's completely RANDOM.  That's right.  We'll have this error then run it right afterwords and everything will be fine.  Explain that one?

I just opened another case.  I would look forward to hearing anything from the visitors of this forum.

Please advise.

[cypherx]

Ok, The symantec tech that got back with me and I both think we know why the registry change did not fix the problem.  The server has not been rebooted since the registry key was changed.

The system has an uptime of 33 days currently, and I first implemented the registry change on 9/10.  The fact that we had no problems between 9/10 and today is just that the C++ error is completely random.

Were scheduling a reboot tomorrow morning 6:30 am.  Once the system comes back up hopefully it will read the new registry keys and we won't have this issue anymore.

[korotkov]

At me the same problem. But it is shown only then when in options exchange there is a daw (Enable the restore of individual mail messages and folders from information Store backups)

[wforrester]

I know I'm late to the party but I was curious if anyone has figured out the source of the runtime error.  We've lived with this for a few years and simply restart bkexec services when it happens.  I'm still going to use mailbox-level backups as long as it's an option.  I find it laughable that the best answer is "Don't use the option we've provided in our software".  Lol...