07-13-2007 06:20 AM
Hello, I had this problem before, there might be an old thread about it but we never came up with a solution since the problem is so intermittent it's hard to find a reason for what causes it to happen.
Situation:
On our Windows 2003 server we have Backup Exec 10d. It is scheduled to do three backups.
1st, using the exchange add-on it does brick level backups of our MS Exchange 2003 server.
2nd, it does a full system backup of our Windows 2003 server.
Last, it does a full system backup of our Websense/Fax server.
Now each job has been monitored to see how long it takes, so we set time limits for each one. For example, if the Exchange server takes longer than 4 hours to complete, cancel the job. We don't want to run out of time because when you schedule jobs you can only provide a 'window of opportunity' as I like to call it. There's no option to 'start this job after previous job regardless of time'. Very limited for software that costs over $1000 don't you think?
Anyway here's the error that we see on our exchange server in the morning...
The title bar says: Microsoft Visual C++ Runtime Library
There's a red X, and it says Runtime Error!
Program: C:\Program Files\VERITAS\BackupExec\RANT\beremote.exe
This application has requested the Runtime to terminate it in an unusual way. Please contact the applications support team for more information.
[OK]
Now on the Backup Exec 10d server console we see:
Optiofax Full System Save Missed Elapsed Time 0:00:00 Byte Count: 0
Full System Save(Nightly) Missed Elapsed Time 0:00:00 Byte Count: 0
Exchange Save HP1 Backup Failed Elapsed Time: 13:38:24 Byte Count: 16,002,008,824
On a good day we should see:
Optiofax Full System Save Completed Elapsed Time 5:21:20 Byte Count: 16,520,617,250
Full System Save(Nightly) Completed Elapsed Time 5:31:15 Byte Count: 87,370,362,037
Exchange Save Completed Elapsed Time: 3:26:56 Byte Count: 40,696,217,511
So this Visual C++ error in beremote.exe is causing all of our backups to FAIL. This is NOT ACCEPTABLE. It doesn’t matter that I have Exchange set to cancel after 4 hours. Look at the log…. 13:38:24! That’s WAY more than 4 hours. I don’t know what kind of math Symantec is using to determine job run times.
So to recap…
We come into work in the morning, see that the Exchange save is still running, but no byte count is happening, and we go over to the Exchange server and find this visual c++ error in beremote.exe. We click OK on this error, then back on the BackupExec console we see the Exchange save fail “Communication Error”, and our next two backups go into a missed status, since were way beyond the time frame for them to run.
Can someone please help with this situation? I greatly appreciate your efforts. I hope someone out there is up to this challenge.
Details for our Exchange save:
Resource order:
First Storage Group
Microsoft Exchange Public Folders
Microsoft Exchange Mailboxes
Microsoft SQL Server
Resource Credentials:
All Successful
Priority and Availability:
Medium
Cancel backup job if not completed within 4 hours
Device and Media:
All Devices
Media Set 1
Overwrite Media
General Settings:
Exchange Save
Verify after backup completes
Hardware Compression (if available, otherwise none)
Advanced:
Back up files and directories by following junction points
Set Remote Agent priority: Normal Priority
Open file backup when Advanced Open File Option is not used: With a lock
Microsoft Exchange:
Exchange Server backup method: Full – Database & Logs (flush committed logs)
Mailbox backup method: Copy – Back up messages
Schedule: Run according to schedule.
S, M, T, W, Th, Fr
Start no earlier than: 4:45 PM
And no later than 5:00 PM
Do not delete the job
Error event logs on our Exchange Server:
Only error I see is once we hit the OK on the C++ runtime error.
Under System
Information 7/12/2007 5:12:22 PM Application Popup Event 26
Application popup: Microsoft Visual C++ Runtime Library: Runtime Error!
Program: C:\Program Files\VERITAS\Backup Exec\RANT\beremote.exe
This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
Error 7/13/2007 6:09:31 AM Service Control Manager Event 7031
The Backup Exec Remote Agent for Windows Servers service terminated unexpectedly. It has done this 1 time. The following corrective action will be taken in 60000 milliseconds. Restart the service.
07-26-2007 06:25 AM
08-07-2007 05:22 AM
08-29-2007 05:24 AM
08-30-2007 06:26 AM
08-30-2007 09:45 AM
09-05-2007 01:51 AM
09-05-2007 05:20 AM
09-05-2007 05:29 AM
09-05-2007 06:09 AM
09-06-2007 05:46 AM
09-06-2007 06:03 AM
09-06-2007 06:07 AM
09-10-2007 05:17 AM
09-19-2007 06:33 AM
Followed instructions from our original case # 320-065-707
We made the following changes on the registry of the Exchange 2003 server.
1. Connect to the Exchange Server and Open regedt32 on the Server.
2. Browse to the registry key given below:
HKey_Local_Machine > Software > VERITAS > Backup Exec > Engine > Exchange > Bypass MBox properties
3. Change the above key's value to 1.
Today we just received another one of Symantec's Brilliant C++ errors causing the backup to fail.
THIS IS UNACCEPTABLE. We rely on good nightly backups. I urge you to update the beremote.exe program or send us updated C++ libraries that are up to Symantec's standards.
Here's what the System event logs are saying on the Exchange server:
9/19/2007 5:15:51 AM EventID: 26
Application popup: Microsoft Visual C++ Runtime Library : Runtime Error!
Program: C:\Program Files\VERITAS\Backup Exec\RANT\beremote.exe
This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
At 9:06 AM we finally had time to go into the server room and notice this application popup. Upon clicking OK on the error dialog the next items appear in the System event log:
9/19/2007 9:06:07 AM
The Backup Exec Remote Agent for Windows Servers service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 60000 milliseconds: Restart the service.
9/19/2007 9:07:07 AM
The Network Load Balancing service failed to start due to the following error:
The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.
9/19/2007 9:07:10 AM
The Backup Exec Remote Agent for Windows Servers service entered the running state.
Now the Exchange backup as shown on our Windows 2003 primary domain controller is showing that the backup is verifying. As it gets to 16 GB it fails. It appears as if the C++ error occurs when backing up individual mailboxes. Now we have to restart the backup and tie up our tape drive for the next 3 hours. God hope no one want's us to restore anything in that time.
Here's what makes is so hard to troubleshoot... It's completely RANDOM. That's right. We'll have this error then run it right afterwords and everything will be fine. Explain that one?
Please advise.
09-19-2007 10:09 AM
05-29-2009 01:14 AM
03-04-2010 04:58 AM