Is NetBackup 6.5.2 nbpemreq stubborrn?

[Stumpr2]

NetBackup 6.5.2

I removed a client from a policy yesterday afternoon. This morning the policy attempted to backup the client. It is as if the "Policy Update Interval" is not running. I manually ran nbpemreq -updatepolicies and will monitor to see if the stubborn NetBackup software will now acknowledge the removal of the client.

[Stumpr2]

additional info:

# vxlogview -p 51216 -o 116 --who 'doPolicyChanges' -t 24:00:00 -d all
V-1-1-12 There are no records to be displayed.

The above response to the command indicates that Netbackup is ignoring my changes that were made in the last 24 hours.

[dami]

Bob,

I've seen this sort of thing now and again but I ususally put it down to multiple GUI's being opened by various users and changes being lost when one saves but is not refreshed. Nbpemreq has normally done the trick. Mainly, I wanted to try out your

"vxlogview -p 51216 -o 116 --who doPolicyChanges -t 72:00:00 -d all"

command because I thought wow that would be cool. Nothing doing I've made changes over the last days and this is not reporting anything. I dont think it works in this form at least (yields the classic
V-1-1-12 There are no records to be displayed.) Shouldnt there be something () before the '--who' ?

[Andy_Welburn]

... and yet I've just run the same command as Bob & it picked up the one change I made today. (Running 6.5.1 tho').

[Stumpr2]

I cut/paste the command from a technote DOCUMENTATION: How to update the "nbpem" daemon after making changes to a policy without waiting for the Policy Update Interval to take effect http://support.veritas.com/docs/278492

maybe I need to set the logging level to 2 as described in the technote

Details:

Manual:  VERITAS NetBackup (tm) 6.0 System Administrators Guide, Volume I for UNIX and Linux, p. 414


Modification Type:  Addition

Modification:
The Veritas NetBackup (tm) 6.0 System Administrator's Guide describes a new global attribute called Policy Update Interval.  This controls how often the NetBackup Policy Execution Manager (nbpem) daemon will check for new policies or for updates to existing policies.  By default, the value is 10 minutes. This means that new policies or changes to existing policies will not go active immediately.  The nbpemreq command can be used in situations where a change to a policy needs to take effect immediately.

To submit changes to a policy immediately,  run the following command:
# cd /usr/openv/netbackup/bin/admincmd
# ./nbpemreq -updatepolicies

This will cause the nbpem daemon to update with any changes to policies.  A policy that has an open backup window and is scheduled to run will appear in the Activity Monitor.

Unified logging will show entries every ten minutes for JobScheduler::doPolicyChanges.  These messages will appear at a logging level of 5 for nbpem.  Anytime a policy is modified,  additional log messages will appear listing what policy has been updated.  These messages will be logged at a logging level of 2 for nbpem.

To list all policy changes that have occurred over the last twelve hours,  run the command:
# vxlogview -p 51216 -o 116 --who 'doPolicyChanges' -t 12:00:00 -d all

The following messages show a normal check for policy updates that detected no changes.

11/11/05 10:30:57.495 [Debug] NB 51216 nbpem 116 PID:5844 TID:1 [No context] 5 [JobScheduler::doPolicyChanges] +++ ENTERING +++ : obj = ffbff120 (JobScheduler.cpp:5013)

11/11/05 10:30:57.495 [Debug] NB 51216 nbpem 116 PID:5844 TID:1 [No context] 5 [JobScheduler::doPolicyChanges] --- EXITING --- : obj = ffbff120 (JobScheduler.cpp:5013)

The following messages show a normal check for policy updates that occurred ten minutes later.  This check detected one policy in need of updating.

11/11/05 10:40:57.458 [Debug] NB 51216 nbpem 116 PID:5844 TID:1 [No context] 5 [JobScheduler::doPolicyChanges] +++ ENTERING +++ : obj = ffbff120 (JobScheduler.cpp:5013)

11/11/05 10:40:57.459 [Debug] NB 51216 nbpem 116 PID:5844 TID:1 [No context] 2 [JobScheduler::doPolicyChanges] updating policy test-policy(JobScheduler.cpp:5024)

11/11/05 10:40:58.100 [Debug] NB 51216 nbpem 116 PID:5844 TID:1 [No context] 5 [JobScheduler::doPolicyChanges] --- EXITING --- : obj = ffbff120 (JobScheduler.cpp:5013)

The following messages were logged by running the nbpemreq command to update policies immediately.  This detected two policies in need of updating.

11/11/05 10:41:56.560 [Debug] NB 51216 nbpem 116 PID:5844 TID:9 [No context] 5 [JobScheduler::doPolicyChanges] +++ ENTERING +++ : obj = ffbff120 (JobScheduler.cpp:5013)

11/11/05 10:41:56.560 [Debug] NB 51216 nbpem 116 PID:5844 TID:9 [No context] 2 [JobScheduler::doPolicyChanges] updating policy __DSSU_POLICY_disk-stu1(JobScheduler.cpp:5024)

11/11/05 10:41:56.648 [Debug] NB 51216 nbpem 116 PID:5844 TID:9 [No context] 2 [JobScheduler::doPolicyChanges] updating policy __DSSU_POLICY_disk-stu0(JobScheduler.cpp:5024)

11/11/05 10:41:56.717 [Debug] NB 51216 nbpem 116 PID:5844 TID:9 [No context] 5 [JobScheduler::doPolicyChanges] --- EXITING --- : obj = ffbff120 (JobScheduler.cpp:5013)

The logs can be searched for these entries to verify that a change to a policy has been registered with NetBackup 6.0.

Message Edited by Stumpr on 01-08-2009 10:02 AM

[Andy_Welburn]

---

@Stumpr wrote:
 

maybe I need to set the logging level to 2 as described in the technote

---

Ours is set to 3 so ....

---

@Stumpr wrote:
Message Edited by Stumpr on 01-08-2009 10:02 AM

---

Thanks for that - eyes were starting to smart!

[dami]

I'm 6.5.3 but yeah that must be it - I just set all loging (legacy & unified) a few days ago to zero because I was getting fedup of the endless gigs and gigs of waffle the things were producing, clogging up my drives and cpu's.

Still doesnt solve the question of why the deleted client was still being backed up.

[marekkedzierski]

Go to master server properties and change number of retries per x hours.. I know it's stupid but I had very similar issue. I was trying to change backup window in schedule, noting happen so I was trying to set it using nbpemreq -updatepolicies. Then I've changed this option and nbpem was refreshed after few minutes.

In your case.. go to /usr/openv/netbackup/db/classes, find you policy and check if your client is not there.

You can also try to copy policy to new one and delete old policy.

marekk

[Stumpr2]

---

@Marek Kedzierski wrote:

In your case.. go to /usr/openv/netbackup/db/classes, find you policy and check if your client is not there.

You can also try to copy policy to new one and delete old policy.

marekk

---

The client does not show in the policy definition in the GUI.

The client does not show in the .../class/clients file.

Evidently it is squirreled away in some schedule table or memory register that is not getting updated.

This has happened before and a sure fix is to stop/start netback which could clear either a schedule table entry or memory register.

I followed your suggestion about changing the global attribute for "Schedule backup attempts:"

from 4 tries per 12 hours

to 3 tries per 12 hours

I then set it back to my preferred 4 tries per 12 hours.

I'll see if it runs tonight and post back the results.

[marekkedzierski]

Bob, maybe it's not related to your issue but.. if you have "4 tries per 12 hours" in master server properties and schedules that should to run every hour or every 30 minutes - it's working in your environment ?

[Stumpr2]

---

@Marek Kedzierski wrote:
Bob, maybe it's not related to your issue but.. if you have "4 tries per 12 hours" in master server properties and schedules that should to run every hour or every 30 minutes - it's working in your environment ?

---

yes, it works fine for hourly jobs. Each unique backup job (jobID) will attempt to run up to 4 times before ending with a final status code. Each hourly job has its own jobID so each one will do up to 4 attempts. If it fails then the next hourly job will also have up to 4 tries.

[marekkedzierski]

2 of my customers had issue on HP-UX with NB 6.5.3 and Solaris with NB 6.5.2A.. When there was a schedule that should to run every 20 minutes I had to change this option to 3 tries per hour.. Without this modification backup started once per 2 hours, once per 3 hours... Maybe it's not issue, maybe it's correct behaviour.

I had the same problem on disk staging schedules after upgrade from 5.1 to 6.5.. Relocation was starting or not, without any error message in activity monitor. When I changed schedule backup attempts to for example 1 for one hour - issue was resolved.

"Each hourly job has its own jobID so each one will do up to 4 attempts. If it fails then the next hourly job will also have up to 4 tries." hmm 4 tries during 12 hours window in your case ?

Message Edited by Marek Kedzierski on 01-08-2009 02:15 PM

[Stumpr2]

Perhaps the client has finally been removed from wherever it was stashed. At least it isn't predicted to run with the other clients in its former policy

nbpemreq -predict -date 01/09/2009 07:00:00 | grep -i $POLICY_NAME

[Stumpr2]

The results are inconclusive. I had to cycle Netbackup last night. Stopping and starting the NetBackup services has always worked to remove stubborn clients from the schedules.

[Har-D]

Hi Bob,

"Restart" is the master key...  for Everyone and Everywhere!!!.

The commands that are discussed , perhaps, will work for 6.5.2. But 6.5.2/2a has issues in nbpem, better look for an upgrade of NBU with 6.5.3, which has fixes for multiple issues observed in 6.5.2 related to pem and jm.

You can list out the known issues in 6.5.2/2a in Late Breaking News bulletin for 6.5.2/2a at the Symantec Knowledgebase. Also, once after having a look at the Known Issues in 6.5.2/2a, if planning for upgrade, there only you'll find a link to the lists of 6.5.3 Release Update downlaods.

Regards,