Agent crashing on Domino after upgrade to 12.

[curriertech]

We've been on 10d for quite a while, then a few weeks ago upgraded to 11d with no issues.  When the notification of 12 came out I jumped on it, and I haven't had a good Domino backup since because the agent keeps crashing.  The agents on our other non-Domino servers seem to be fine.  I recall this being a common issue with 11d when it was first released.  Here is the error that shows in the event log on the Domino server:

Faulting application beremote.exe, version 12.0.1364.0, faulting module bedsnote.dll, version 12.0.1364.0, fault address 0x00022040.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Any ideas?  I'll go back to 11d if I have to, but I'd much rather get this one issue worked out and stick with 12. 

[CWtechnician]

rlj - This may be the case but Cyberfed has not gotten a single full backup on BE v12.5; I wonder if he is getting the same issues as we all are or different one.

I have also found 0 documentation saying this has been fixed in v12.5 Do you know where I can find this, so that I can provide it to my CEO ETC.. in order to upgrade once more. I am also sure that Symantec will allow for a free upgrade seeing how we purchased the software very recently.

- D

[CWtechnician]

Can anyone; user, client or Symantec technician alike provide me with documentation of what was fixed in Symantec Backup Exec v12.5 that was broken and not addressed as a hot fix for v12.0?

Also can anyone else confirm after an upgrade from v12.0 to v12.5 you are not receiving successful backups?

Thanks to anyone and everyone reading this post (Distress signal)

- D

[Winzer]

We were having the same  issue with 12 after re-installing the remote agent and made the change to the registry from this thread and it solved our issue.

[Mikael_Regnstr_]

I have the same problem!

BE 12.0 SP2

Domino 8.0.1

[Russ_Perry]

Some additional info:

1) The 2 registry settings:

 - Enable Notes Database Filtering - During a browse of Domino databases beneath the Notes icon in BEWS, for every *.nsf and *.ntf found, an API call for database open and database close are sent to Domino to determine if we've found a 'real' database.  This is used in displaying what is available for backup.  Turning this off simply allows selection all *.nsf and *.ntf files, whether real databases or not.

 - Enable Change Journal - As stated earlier, this selects whether the agent depends on the Windows Change Journal to track database changes or not.  Along with this setting, the .UCJ file keeps track of databases.  Setting this setting to '0' causes the agent to use complete file scans to determine new and changed DBs.  If there is corruption in the Change Journal, it is usually best to delete the UCJ at the same time you turn off this setting.

Because the cause is usually some issue with Change Journal, and of the ease of workaround for the agent crash, the issue has not been addressed in any patch or update as far as I know.  The Technote for this is http://support.veritas.com/docs/302505.  Was this Technote not found or not helpful?  If not, are there additional keywords we can add to the Technote to make it easier to find?

2) About Belnapi.exe - The Remote Agent uses Belnapi.exe to communicate with Domino.  Because Belnapi remains running at all times with a connection to Domino, if for any reason the Remote Agent stops (crash, end-process, etc.) Domino detects this and starts with the console error messages about belnapi 'has terminated abnormally'.  These don't hurt anything but quickly fill up the console.  A restart of Domino is required to get rid of them.

Hope this helps!

Russ

[Mikael_Regnstr_]

2) About Belnapi.exe - The Remote Agent uses Belnapi.exe to communicate with Domino.  Because Belnapi remains running at all times with a connection to Domino, if for any reason the Remote Agent stops (crash, end-process, etc.) Domino detects this and starts with the console error messages about belnapi 'has terminated abnormally'.  These don't hurt anything but quickly fill up the console.  A restart of Domino is required to get rid of them.

This is my main problem right now! Its filling the console 24/7. Restart Domino!? How often? I have over 100+ servers all around the world with 24/7 wanted upptime...

2009-03-12 10:17:22   Process C:\Program
(7728/0x1E30) has terminated abnormally
2009-03-12 10:18:22   Process C:\Program
(7728/0x1E30) has terminated abnormally
2009-03-12 10:19:22   Process C:\Program
(7728/0x1E30) has terminated abnormally
2009-03-12 10:20:22   Process C:\Program
(7728/0x1E30) has terminated abnormally
2009-03-12 10:21:22   Process C:\Program
(7728/0x1E30) has terminated abnormally
2009-03-12 10:21:24   Admin Process: Sea
2009-03-12 10:22:22   Process C:\Program
(7728/0x1E30) has terminated abnormally
2009-03-12 10:23:22   Process C:\Program
(7728/0x1E30) has terminated abnormally
2009-03-12 10:24:22   Process C:\Program
(7728/0x1E30) has terminated abnormally

[Russ_Perry]

Mikael wrote:

"This is my main problem right now! Its filling the console 24/7. Restart Domino!? How often? I have over 100+ servers all around the world with 24/7 wanted upptime..."

Any time the Remote Agent crashes or is terminated by user while belnapi.exe is running with a connection to Domino, the messages will show up in the console...but they are not normal and you are not expected to 'live with it'.  I was simply explaining what causes them and how to get rid of them.  First, of course, it is not normal for the Remote Agent to crash.  If that happens, the reason must be found and corrected, possibly/probably with the help of tech support.  Second, if belnapi.exe is running, never terminate it via task manager or Drwatson32 -P.  If the Remote Agent needs to be stopped for some reason, use the Services applet.

Russ

[KEMIN_INDUSTRIE]

I tried to follow yoru steps 1- 4 but I am unable to find NOTESDB*.UCJ files in Symantec Backup Exec media server & Domino server ..

Therefore, I didn't managed to backup Lotus Domino database as backup drive of Domino server will automkatically exclude Lotus Domino databases.

Please help

Hoay Fern

[elwappo]

We are recieving the same error on domino 8.5.  I'm sure you guys have been asked this by now but I will ask it anyway.  Is it ok to try this fix on an 8.5 server recieving the same errors?

[elwappo]

This seems to be working for us now.  

From a customer point of view I am disappointed at the support for 8.5 not being ready on the release of the product.  Not trying to be negative but it really is something that I would expect to have been tested and ready before a production release of software from a big vendor.  Just something you may want to bring up at any "what are you hearing from customers" meetings.

[POMI76]

Hi

I appreciate this is an old (ish) thread but is there any update yet from Symantec as the impression I get from this thread that there is still no REAL fix..?  Mixed bag of results!  The problem still exists for me after trying EVERYTHING mentioned!

Help appreciated.

[POMI76]

implementing all the forementioned registry changes did not work for me until I changed the Resource Order.  By Default Symantec will back up any drives prior to the Lotus Domino Databases.  I pushed up the Lotus Domino Databases to first in the Resource order and have had successful backups since....

Apologies if this is already known.  Just thought i would mention it in case it helps anyone else...

Good luck

[John_Cox_3]

I have been getting this problem sporadically for a long time.  I applied the registry changes with no luck. Just now I did a local install of the agent (instead of a push install from the media server) as suggested in a previous post. As for the resource order, I used to have my Domino backup as first in sequence and it happened then as well. I changed it to last and the same thing happens.

BEX Server and Domino Server are W2K3 STD SP2
Bex is v12.5 Rev 2213
BEXRA is 12.5.2213
Domino is 6.5.3

[John_Cox_3]

So I re-read the posts and went to http://seer.entsupport.symantec.com/docs/302505.htm. This time I changed the registry entries on the media server. Up to now I had just changed the Domino server registry.

Fingers crossed!

[Russ_Perry]

Hi John

I haven't seen support cases come up on this issue for a long time so i don't know if it is still a problem. One item of note on your environment is that with Domino 6.5.3 and Backup Exec 12.5 you are in an untested/unsupported configuration (ftp://exftpp.symantec.com/pub/support/products/Backup_Exec_for_WindowsNT/307063.pdf).  There may not be much difference between 6.5.3 and 7.x with Domino agent backups but it's untested. 

In general, if anyone is still experiencing issues in a supported environment with the Domino agent as described in this thread that are not fixed with the registry settings, you must have a support case opened and escalated. 

[John_Cox_3]

Russ,

Thanks for the reply. The problem is still cropping up from time to time but hasn't happened now for a few weeks. We badly need to upgrade to Domino 8 but there's so much other stuff going on (what's new?) that I haven't had the time. I'm also looking at upgrading to BEX 2010 and getting the deduplication agent so lot's to do.

Thanks again,

John