backup_exit_notify version mismatches

[ajiwww]

Hello All,

My netbackup Enterprise 6.5.6 is running on AIX master 3.5.

I have issues in NBPEM getting terminated and core dumping. Symantec found that backup_exit_notify is the culprit. The version of this script is pretty old. I am not sure what happend to the script that was upgraded when 6.5.6 was applied on the master.

below are the present version showing and it is very old.

# grep Header backup_exit_notify
# $Header: backup_exit_notify.sh,v 1.3 2003/08/13 14:05:11 $

How to get this script upgraded to the latest version of 6.5.6 ? Any other files also need to upgraded along with this ?

If it cant be upgraded alone, is there any way to download the latest 6.5.6 version ?

--Thanks

Pradeep

[Anonymous]

NBU 6.5 supplied

# $Header: backup_exit_notify.sh,v 1.4 2005/10/18 14:50:27

You are right to have concerns.

Might be worth capturing your environment using the 'nbsu' tool, can get some useful info and version information from this about your whole NBU on the master. And some installation history. To possibly help you proceed forward.

See the text file(s)

NBU_versioninfo.txt

This essentially runs the binary 'versioninfo' tool found in the goodies support directory.

See my signature for nbsu technote.

[Marianne]

Do you have a media server in your environment that you can copy the file from?

Can't the Symantec engineer provide you with a 'good' copy?

If you still have the 6.5 install media, you should be able to extract the file from there.

Else, someone else with a 6.5 installation should be able to copy & paste for you?

I have copied and attached our own in-house file from a Linux master. Our NBU version is 7.1. There should be no problem using it as-is in your environment as there are no version-specific call made by the script.

I had to add '.txt' to the file to permit the upload.

Rename it once you have saved it.

Be careful when you copy the file to your master - use something like WinScp and copy in 'SCP' mode.

Double-check the contents of the file once transferred - there should be no ^M at the end of each line. This normally happens when text files are transferred via ftp in binary mode. (easily fixed with 'dos2unix filename  >new-file'

See if the 'ouffile' is produced at the end of a test-backup. this proofs that the script is good.

If nbpem still core dumps, ask the engineer to look for another reason...........

[Andy_Welburn]

there's no difference to the script Mariannne has provided & what we have on our 6.5.6 Solaris master (except the header & copyright lines that is!)

[ajiwww]

Thanks all and special kudos to marianne :)

i dont find any difference in the files rather than the header !!
i have the files available in the media server, but running on different OS

I have the installation package NB_6.5.6.rs6000_351559.tar with me. how to extract the script from it ?

[Will_Restore]

That would have been my guess.

Symantec found that backup_exit_notify is the culprit

How do they figure that?  Doesn't sound right to me.

[Marianne]

Chances are good that the 6.5.6 patch does not contain notify scripts. Patches normally contain updated binaries only. Best would be to extract original 6.5 install media to a temp folder.

BUT I REALLY don't think it is necessary to go to that length. Please send the support engineer a copy of your file and the file that I sent and ask him to point out where the problem is with the script.

Have you checked if the 'OUTF' (/usr/openv/netbackup/bin/BACKUP_EXIT_CALLED) is created?

If so, I cannot see how this notify script can cause nbpem to core dump.

Even if you read through the notes in the beginning of the file, it seems highly unlikely for this script to cause nbpem failure:

# This script is called by the NetBackup scheduler, after an individual
# client backup has completed (including media closure and image db validation.
#
# NOTE:  this script will always be run in "background" mode, meaning that
#        the NetBackup scheduler will NOT wait for it's completion.

PLEASE ask this engineer to look elsewhere or escalate....

[CRZ]

I could imagine a scenario where nbpem is waiting for a result from the script, but the script blows up, so it causes nbpem to blow up.

Have you considered renaming/relocating the script so that it doesn't get invoked and seeing if that affects how nbpem runs?  It would at least provide some ammo for you - or for your TSE :)

[WVT]

or look at with hex viewer (od) ...its the only way to prove they are the same.

what happens if you run it and pass it the variables manually

[ajiwww]

this is what symantec saying

The recent 3 core dumps generated  from "backup_notify_script" when making library system calls, which is reliant on AIX C++ library, which is most certainly outside of Netbackup caused from AIX

Library system calls use the AIX C++ library, which is most certainly outside of Netbackup and the root cause is triggered from AIX.

Please note that version of "backup_notify" is not the root cause, just wanted to make you aware that 6.5.6 version was not being used and does have changes in the parameter. So if it going to be used then I would test before applying to your production environment

If the problem is still happening then you will need to report this problem to IBM support.

IBM says this

It is interesting to observe that in cores 1 and 3 that nbpem is a single threaded process, and in core 2 it is a multi-threaded process with most of its threads in, apparently, a dead state.  I also see that cores 1 and 3 are caused when a spawn comand is doing something with putenv, while core 2 is caused when NetBackup is doing a printf() statement via internal code.
I see that nbpem is a 32 bit process.  
Although the core dumps are occurring during calls to libc routines, the routines are not checking the input they are receiving.  A core dump in printf() usually caused when a string pointer passed into it points to invalid memory.  I expect the same thing is occurring with putenv().  These string pointers are arising out of NetBackup libraries.   I would disagree with Symantec's assessment that this is a libc issue.

[Mark_Solutions]

Are you actually using the backup_exit_notify script for anything?

If not just move it, if it is not there NetBackup cannot call it and will just carry on without this core dump issue.

Obviously check all the pre-requisites / installation guide for AIX as i do remember some specific library files are required by NetBackup

Hope this helps

[ajiwww]

i need this for notification of any failure and get it monitored by a different software

[Mark_Solutions]

If this is causing you such issues could you not use NOM/OpsCenter for that?

[tom_sprouse]

Forget the Header, what is the script actually doing?

Have we confirmed that the script is causing PEM to CORE?

If the script is removed or renamed will the issue occur over time?

Would you be willing to post your script for the community to review?

Also, check to ensure there are no special characters in the file that could be causing this issue.  If the script was created on a different OS, or contents of the file were pasted from another host, there maybe something hidden we not seeing?

[Omar_Villa]

Why you dont move to OpsCenter? bp_exit_notify will get deprecated soon, so earliest you get rid of it will be better, maybe you can get a temporary fix but in the long term I think going to OpsCenter is the best option

[Marianne]

Have a look at this TN:

http://www.symantec.com/docs/TECH54293

[Marianne]

Pradeep, did you manage to solve this problem?