Job Engine Exception

[Greg_Huntzinger]

We have BE V 10 installed on several servers. All of the servers are runni g Win2000 Server, current service packs, current updates. On one of those machines the backupExec Job service sometimes crashes during backups with a c0000005 exception at address 10432DA7 (copy). This has happened several times.

We have done a reinstall and installed all BE updates, so we already have SR1.

All of our agents are the current BE 10 versions.

I've seen several entries in this forum that seem similar to our problem, but none of them seem to have been resolved.

The only kbase article I've found was 276242, which says to contact Veritas technical services.

How can we get this problem fixed?

[Greg_Huntzinger]

Has anyone looked at this thread? Does anyone have a suggestion/answer?

This is a serious problem for us.

[Jim_Bollinger]

Check on this forum there are other threads on this topic

[Greg_Huntzinger]

Thanks for the reply, but the threads I've seen all seem to have some other stuff going on, like synthetic backup. My point with this thread is that we're just doing boring stuff here and the things not working.

I guess I'm having trouble understanding what the big deal with fixing this is. Services are not supposed to crash, period.

I've written services and currently write device drivers. Services are supposed to be able to get all kinds of wrong data and fail gracefully.

[Shyam_Sundar]

Hello,

We apologise for the inconvenience caused because of
the delayed response. Please let us know if your issue
still persists. If yes, please refer the steps mentioned
below. If not, we will mark this case as assumed
answered and move it to answered questions pool.

We would suggest you to perform a repair installation of Backup Exec referring the following document.



http://support.veritas.com/docs/253199

And the Service Pack 1 of Veritas Backup Exec should be installed again after the repair installation is done.

Do let us know if the issue persists.

NOTE : If we do not receive your reply within two
business days, this post would be marked ‘assumed
answered’ and would be moved to ‘answered questions’ pool.

[Greg_Huntzinger]

The repair installation was done weeks ago. The Service Pack install was done at initial installation time as were all of the hot fixes.

This problem continues to persist. The only way that we have been able to stop it from happening is to have the BE GUI runing all of the time.

It seems clear that there is an error in the job service code. When will this be corrected?

[Ajit_Kulkarni]

Hello,

Please elaborate on the following "This problem continues to persist. The only way that we have been able to stop it from happening is to have the BE GUI runing all of the time"

I also request you to tell that, if job engine service crashes during a specific backup job or any job ? Does it give any specific error ?

Waiting for your reply.

Regards

NOTE : If we do not receive your reply within two business days, this post would be marked "assumed answered" and would be moved to "answered questions" pool.

[Greg_Huntzinger]

"this problem continues to persist" - that means that the problem keeps happening, the problem happens over and over again, the job engine stops with Dr watson errors.

"The only way that we have been able to stop it from happening is to have the BE GUI running all of the time" - that means that if we keep the BE application running all of the time, the error does not happen. Having a service-side component dependent on the operation of a user-side component seems like an error.

The specific error from the system event log is: c0000005 exception at address 10432DA7 (copy).

Thanks.

[Jim_Bollinger]

I should have said I have seen this (though not repeatedly the way you are), and we're just doing boring stuff also.

[Greg_Huntzinger]

The whole business is getting a bit silly. This is a hard, system level failure that shouldn't be all that hard to get a handle on.

Instead of asking me to repeat the same information again and again, it seems to me that someone at Veritas should get me a version of the Job Engine sevice that has some instrumentation (assuming that they did not use WPP) so that I could get them some trace information.

Since that is not happening, my job is to sit here and keep saying, "See, it's still BROKEN!".

I should say that the workaround of keeping the GUI running all of the time seems to work. However, we don't like the idea of leaving a user logged in on the server console.

[Kelly_Harper]

We are running three Backup Exec 10 SP1 servers and all three started having Backup Exec Job Engine service failures with in two weeks of each other.

I'll start leaving the GUI running because I need to get data backed up but I would LOVE to hear from Veritas on this issue. These same servers ran flawlessly for months...

[Greg_Huntzinger]

Kelly,

I'll be interested to hear if this works on your site the way that it does here.

I found this workaround in another thread where it even described some problem where the Job Engine trys to post an alert and gets confused if the GUI is not running. Sounds like a bug to me.

[Kelly_Harper]

I'll let you know how it goes tonight for sure. The other frustrating side affect of this issue is that it marks my tapes with "end marker unreadable" effectively cutting the capicity way down. The only way to clear that status is to erase the tape...the joy of it all. Now where's my fresh cup of coffee.

[Kelly_Harper]

Leaving the GUI open all night seems to have worked. Thank you for posting that work around while we wait for Veritas to provide a solution.

[Sharvari_Deshmu]

Hello,

We regret the inconvenience caused.
Please do the following to resolve the issue:
I


1. Apply the latest service packs for OS as well as BE (in this case it is already done)

2.Update to the latest MDAC Version and if you have already installed latest MDAC version then reapply it.
Please see the link below:
http://www.microsoft.com/downloads/details.aspx?FamilyID=78cac895-efc2-4f8e-a9e0-3a1afbd5922e&DisplayLang=en

3.If that does not help then perform repair installation of BE. For performing Repair installation of Backup Exec please refer to the foll technote:
http://support.veritas.com/docs/253199

II
Also please try doing the following steps:


1) Split the Backup jobs into smaller ones and see if the error is occurring on any particular drive or file. If the error is occurring on any particular folder then exclude that folder from the backup or check its integrity.

2) Run a backup job on the Backup to Disk folder and see if you are getting the same error.

III

In Tools->Options->Preferences->Display progress Indicators for Backup jobs uncheck it if you have checked that option and then verify.

Please keep us updated.

Thanks,

NOTE : If we do not receive your reply within two business days, this post would be marked "assumed answered" and would be moved to "answered questions" pool.null

[Kelly_Harper]

What do we do when we've done all of that and this issue still occurs? Three servers with the same service crashing, all with in two weeks of each other after working fine for months.

Kelly

[Greg_Huntzinger]

All of these items have been covered in other threads and I have done all of them several times. I have not been able to narrow the problem down to a specific job or a specific client machine. I have this problem on one server frequently and two others less frequently.

Regardless, nothing that happens on a client machine should be able to make the Job Engine service crash. I should also be getting some sort of system error log entries that should help me narrow this down and I am not getting anything useful.

[Sharvari_Deshmu]

Hello,

Please check in drwatson for the exact error message.
the log file is drwtsn32.log


Thanks,

NOTE : If we do not receive your reply within two business days, this post would be marked "assumed answered" and would be moved to "answered questions" pool.

[Greg_Huntzinger]

Here's the Dr watson info from my latest crash:

Application exception occurred:
App: (pid=3232)
When: 6/2/2005 @ 04:04:21.468
Exception number: c0000005 (access violation)

Here's the state dump for the faulting thread:

State Dump for Thread Id 0x7ac

eax=012e7a30 ebx=00010000 ecx=0000068c edx=00010000 esi=028be5d0 edi=012f6000
eip=01432da7 esp=01c2bc9c ebp=01c2bd0c iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206


function: Copy
01432d87 e854efffff call MemoryMappedFile::CloseMap+0x140 (01431ce0)
01432d8c 8d4db8 lea ecx, ss:02445bf2=????????
01432d8f e87cfcffff call MemoryMappedFile::Allocate+0x460 (01432a10)
01432d94 8bf0 mov esi,eax
01432d96 8d4d9c lea ecx, ss:02445bf2=????????
01432d99 e872fcffff call MemoryMappedFile::Allocate+0x460 (01432a10)
01432d9e 8bf8 mov edi,eax
01432da0 8bcb mov ecx,ebx
01432da2 8bd1 mov edx,ecx
01432da4 c1e902 shr ecx,0x2
FAULT ->01432da7 f3a5 rep movsd ds:028be5d0=001c0008 es:012f6000=????????
01432da9 8bca mov ecx,edx
01432dab 83e103 and ecx,0x3
01432dae f3a4 rep movsb ds:028be5d0=08 es:012f6000=??
01432db0 8b75ec mov esi, ss:02445bf2=????????
01432db3 03f3 add esi,ebx
01432db5 8975ec mov ,esi ss:02445bf2=????????
01432db8 8b7d18 mov edi, ss:02445bf2=????????
01432dbb 8bc7 mov eax,edi
01432dbd 2bc6 sub eax,esi
01432dbf 8945dc mov ,eax ss:02445bf2=????????
01432dc2 8d4ddc lea ecx, ss:02445bf2=????????

All of the other threads are sitting on waits except for these:

State Dump for Thread Id 0x12d4

eax=0012fd18 ebx=00000000 ecx=01010101 edx=00000000 esi=00000000 edi=00000160
eip=77f82926 esp=0012fbd0 ebp=0012fc40 iopl=0 nv up ei pl zr na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000246


function: NtReadFile
77f8291b b8a1000000 mov eax,0xa1
77f82920 8d542404 lea edx, ss:00949ab7=????????
77f82924 cd2e int 2e
77f82926 c22400 ret 0x24


State Dump for Thread Id 0x1408

eax=00000000 ebx=000493e0 ecx=01cbe938 edx=00000000 esi=00140820 edi=000493e0
eip=77f8289c esp=0265febc ebp=0265fee4 iopl=0 nv up ei ng nz ac po cy
cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000297


function: ZwRemoveIoCompletion
77f82891 b8a8000000 mov eax,0xa8
77f82896 8d542404 lea edx, ss:02e79da3=????????
77f8289a cd2e int 2e
77f8289c c21400 ret 0x14

I can sent the whole file with several of these, but not on an open forum.

[Renuka_-]

Hello,


1. Please place the job engine in debug mode and see if you can place the faulting module and exact error message with which it is faulting.


2. Also check if you are receiving any alerts during the job engine crash.

3. Ensure that you have provided sufficient media for the job to complete. Does the backup span? does this occur only during backups or also other jobs?


4. Please try a repair install if you already haven't done so.


http://support.veritas.com/docs/253199



NOTE : If we do not receive your reply within two business days, this post would be marked assumed answeredand would be moved toanswered questions pool.