BEREMOTE crashes on startup on WinXP SP2 clients

[Khee_Chan]

Greetings!

I have recently deployed a number of new WinXP SP2 client PC's. A week later I noticed that all the instances of the Backup Exec Remote Agent crashed upon startup after reboots of the PC's that they are installed on.

WinXP SP2 firewall exceptions were created for the BEREMOTE.EXE application and the NDMP port (TCP port 10000) when the remote agents were first installed. Had no problems with backing up to the BE server for the one week that the remote agents were running properly. Turning off the firewalsl made no difference - BEREMOTE still crashed on startup.

I had forcibly disabled DEP in the affected PC's by using the /NOEXECUTE=ALWAYSOFF in BOOT.INI but this had no effect to the BEREMOTE startups.

Inspection of the Dr. Watson logs in all the affected PC's show the following:
...
Application exception occurred:
App: C:\Program Files\VERITAS\Backup Exec\RANT\beremote.exe (pid=2452)
When: 21-Nov-2005 @ 19:19:40.162
Exception number: c0000005 (access violation)
...
and further down
...
function: ntdll!RtlSetUserValueHeap
7c91bf84 ff01 inc dword ptr
7c91bf86 0000 add ,al
7c91bf88 008bc30bc70f add ,cl
7c91bf8e 84cb test bl,cl
7c91bf90 e7ff out ff,eax
7c91bf92 ff8db5fbfdff dec dword ptr
7c91bf98 ffe9 jmp ecx
7c91bf9a 8de6 lea esp,esi
7c91bf9c ffff ???
7c91bf9e 4a dec edx
FAULT ->7c91bf9f 66833800 cmp word ptr ,0x0 ds:0023:00000008=????
7c91bfa3 7406 jz ntdll!RtlSetUserValueHeap+0x84a (7c91bfab)

I hope that someone will be able to give me some pointers to fix this problem.

Thanks.

[Deepali_Badave]

Hello,

Put the Backup Exec remote agent service in debug mode.

Please refer the following technote:

http://seer.support.veritas.com/docs/254212.htm

NOTE : If we do not receive your reply within two business days, this post would be marked assumed answered and would be moved to answered questions pool.

[Khee_Chan]

Here are the contents of the beremote00.log.

da8 22-Nov-2005 07:28:33: Opening debug log C:\Program Files\VERITAS\Backup Exec\RANT\logs\XXXXX-PC-beremote00.log
e14 22-Nov-2005 07:28:33: FS_InitFileSys
e14 22-Nov-2005 07:28:33: loaded bedsnt5.dll
e14 22-Nov-2005 07:28:33: loaded bedssql2.dll
e14 22-Nov-2005 07:28:33: loaded bedsxchg.dll
e14 22-Nov-2005 07:28:33: loaded bedsxese.dll

Then beremote crashes.The Dr. Watson log looks similar, the part around "FAULT" is as follows:

function: ntdll!RtlSetUserValueHeap
7c91bf84 ff01 inc dword ptr
7c91bf86 0000 add ,al
7c91bf88 008bc30bc70f add ,cl
7c91bf8e 84cb test bl,cl
7c91bf90 e7ff out ff,eax
7c91bf92 ff8db5fbfdff dec dword ptr
7c91bf98 ffe9 jmp ecx
7c91bf9a 8de6 lea esp,esi
7c91bf9c ffff ???
7c91bf9e 4a dec edx
FAULT ->7c91bf9f 66833800 cmp word ptr ,0x0 ds:0023:000000a8=????
7c91bfa3 7406 jz ntdll!RtlSetUserValueHeap+0x84a (7c91bfab)
7c91bfa5 03c6 add eax,esi
7c91bfa7 85d2 test edx,edx
7c91bfa9 75f3 jnz ntdll!RtlSetUserValueHeap+0x83d (7c91bf9e)
7c91bfab 2b85e8fbffff sub eax,
7c91bfb1 d1f8 sar eax,1
7c91bfb3 8985ecfbffff mov ,eax
7c91bfb9 83bdc0fbffff00 cmp dword ptr ,0x0
7c91bfc0 0f85ae010000 jne ntdll!RtlSetUserValueHeap+0xa13 (7c91c174)
7c91bfc6 8b8df8fbffff mov ecx,

Could it be related to the call to RtlSetUserHeap in NTDLL.DLL?

[Robert_M]

if your systems are logging onto a domain and you have DHCP and your Backup Exec services are using a superuser logon which exists on the domain you will need to set the XP machines to "wait for network" prior to stating to load services.

This way the XP machines wait for an IP address and Domain info prior to starting services that require network authentication

Enabling "Wait for network" can be done via Group policy

hope this helps

[Khee_Chan]

Robert,

Thank you for your suggestion. However all the affected systems have static IP addresses. they are members of the same domain as the BE server.

I should add that we also have a whole bunch of other Windows 2000 clients inthe same domain and backed up by the same BE server with the same version of BEREMOTE.EXE and all of these systems run OK.

[Deepali_Badave]

Hello,

Verify if you are getting any errors in the event logs.

NOTE : If we do not receive your reply within two business days, this post would be marked assumed answered and would be moved to answered questions pool.

[Khee_Chan]

On startup of BEREMOTE, the following are logged in the System Event Log with the same timestamps:

Event id: 7022
Source: Service Control Manager
Description: The Backup Exec Remote Agent for Windows Servers service hung on starting.

Event id: 7034
Source: Service Control Manager
Description: The Backup Exec Remote Agent for Windows Servers service terminated unexpectedly. It has done this 1 time(s).

No other event entries related to BEREMOTE were logged in the Security and Application event logs.

[Brett_James]

Has a solution been found for this?

I'm getting the exact same errors after a clean install of BE 9.1, then upgraded to sp4a on a Windows 2003 Sp1 machine.
The symptons are identical, except for the OS version


This machine has the latest Xeon processors with DEP, which is causing the failure. On each boot, a system error occurs stating the beremote.exe has caused a problem.
attemping to turn off DEP through the BOOT.INI (/NOEXECUTE=ALWAYSOFF) results in the same problem. If I put exceptions in DEP for all the BE executables, same problem.

The BE remote service is stopped at this point. If I restart it, start a backup, it all works (oracle db included) except for the Shadow Copies Componet. At this point the backup fails and I get the following error:

"Final error: 0xa00084f8 - The network connection to the Backup Exec Remote Agent has been lost. Please check for network errors. Final error category: Resource Errors"

The problem is, this is a backup the local machine/disks.

Event viewer has the following:
Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7034
Computer: Server
Description:
The Backup Exec Remote Agent for Windows Servers service terminated unexpectedly. It has done this 1 time(s).

I have performed a repair of of BE on the system with no change.

I know it is the DEP capability of the processor causing the problem. This machine is a DELL 2850. I also have a DELL 2650 sitting next to it, with 2003 SP1 & BE 9.1 running DEP in software mode with no issues at all.

[Brett_James]

Hello? Any word on the above? Any advice?

[Khee_Chan]

Brett,

I finally gave up waiting for a response from Symantec/Veritas on this forum and opened up an paid-for incident with Symantec/Veritas tech support.

After some fumbling around on the phone with the tech, I finally got him to understand that the issue is with the client and not the server and also getting through to him him that I DO have the Dr Watson log and dump files.

So the files were sent to him and after some 5 minutes of silence on the phone, he came back and told me to rename bedsxese.dll and BEDSMBox.dll to some other name. Essentially so that they will not be loaded by beremote.exe when it starts. After doing so, beremote starts properly.

I asked him the reason for renaming these files and he could not give me an answer at all! These DLL's support the backing up of Exchange mailboxes but since we don't use Exchange it's no big deal to me and I did not have the patience by then to push the tech further for a proper explanation. However it is NOT a very satisfactory solution.

Eventually I narrowed it down to just BEDSMBox.dll which was causing our installations of beremote to crash on start up. The version of this DLL is 9.1.4691.47 dated 1Mar-2005 and originally came from HotFix 52 for BE 9.1 and is now included in SP4a.

So our beremote clients now run but I am still leaving this question as UNANSWERED because nobody has provided an answer why BEDSMBox.dll crashes beremote.exe or acknowledge that there is a bug in BEDSMBox.dll.

On the other hand, Brett, I'm not sure if the above helps you because it would seem that something else is crashing your installation of beremote. All I can suggest is for you to make sense of the contents of the Dr Watson USER.DMP file. Be sure to select all the Dr Watson dump options.

[Brett_James]

Correction to my last post: Disabling DEP in the BOOT.INI file fixed the problem - a typo in my original edit of the .ini file wasn't disabling it

Great advice.. I took what you gave me one step further... Instead of renaming the files, I added c:\program files\veritas\nt\BEDSmbox.dll to the exceptions list for DEP.
It is found under Start -> Control Panel -> System -> Advanced -> Performance -> DEP.

I've added the above .dll and also beremote.exe to the exceptions

After a few reboots, and a succesful full backup (was failing when it would get to the shadow copy components) it all seems to be working fine.

Would be nice if a Veritas person would reply to this and maybe offer an explanation or a better fix for this?

[Deepali_Badave]

Hello,

Also refer the following technote:

http://support.veritas.com/docs/278033

NOTE : If we do not receive your reply within two business days, this post would be marked assumed answered and would be moved to answered questions pool.