cancel
Showing results for 
Search instead for 
Did you mean: 

Netbackup continues running policy even de-activated or deleted

Amar-khodja
Level 3

Hi experts,

I have de-activated a policy in NetBackup, but it is still running. I have deleted it but no way.

What can I do to stop the policy from running while it is de-activated.

I have done the folloing but didn't solve the issue:

1. Stop NetBackup Services on the master server
2. Remove the following files and directories from the master server
 
    /usr/openv/netbackup/bin/bpsched.d/*
    /usr/openv/netbackup/bin/bprd.d/*
    /usr/openv/netbackup/db/jobs/pempersist2
    /usr/openv/netbackup/db/jobs/restart/*
 
3. Start NetBackup Services
4. Run the command below to force nbpem to re-read configurations and rebuild its cache instead of waiting the default time to do so:
 
    /usr/openv/netbackup/bin/admincmd/nbpemreq -updatepolicies
 

Thanks

 

22 REPLIES 22

quebek
Moderator
Moderator
   VIP    Certified

Hello, I need to admit you are brave ;)

I am not really following your issue! So you have a policy which is currently running in activity monitor? Did you try to cancel this policy? After restart of NBU it should be in state like cancelled or pending restart... If you deactivated this policy it should not be kicked off any longer...

Please provide you NBU version and describe the issue more...

I have missed one point:

When running, the job gives error code 247 and of course stops running in Activity Monitor. See error:

03/13/17 06:01:23.664 [Info] V-116-14 CLIENT bkpema15t1  POLICY bkpema15t_emadb_full  SCHED *NULL* EXIT STATUS 247 (the specified policy is not active) VBRF 1 247

NBU version is: 7.7.3

 

Deeps
Level 6

Hi AMar-khodja , check the output of bppllist <polictname> -U and verify the value for attribute "Active" . This wil verify if this attributed is updated for this policy in Netbackup database.If it stil shows "yes" that means it is not updated (issue with GUI -> not able to update policy attribute) .

For analysis purpose you may need to enable / review nbpem logs .

BTW What is the platform and version of Netbackup ?

 

DeepS

# /opt/openv/netbackup/bin/admincmd/bppllist bkpema15t_emadb_full -U |grep -i active
  Active:              no

Find below nbpem logs:

# /opt/openv/netbackup/bin/vxlogview -p 51216 -i 116 -d all -t 24:00:00 | grep bkpema15t_emadb_full
03/14/17 06:01:22.375 [Application] NB 51216 nbpem 116 PID:8022 TID:14 File ID:116 [jobid=-1 job_group_id=-1 client=bkpema15t1 type=2

server=*NULL* task=ID:1017c03b8 CTX:1017c03b8 policy=bkpema15t_emadb_full] [Info] V-116-82 received start user backup request for

client=bkpema15t1, policy=bkpema15t_emadb_full, schedule=?, stream #=0, client type=Oracle (4), parent jobid=-1
03/14/17 06:01:22.375 [Diagnostic] NB 51216 nbpem 116 PID:8022 TID:14 File ID:116 [jobid=-1 job_group_id=-1 client=bkpema15t1 type=2

server=*NULL* task=ID:1017c03b8 CTX:1017c03b8 policy=bkpema15t_emadb_full] 1 V-116-189 [ExtReqTask::run_specificPolicy] Failed to retrieve

policy bkpema15t_emadb_full, 247
03/14/17 06:01:22.375 [Debug] NB 51216 nbpem 116 PID:8022 TID:14 File ID:116 [jobid=-1 job_group_id=-1 client=bkpema15t1 type=2 server=*NULL*

task=ID:1017c03b8 CTX:1017c03b8 policy=bkpema15t_emadb_full] 1 [PemTask::taskComplete] (ID:1017c03b8 CTX:1017c03b8) Active subtask count=0

(PemTask.cpp:530)
03/14/17 06:01:22.384 [Diagnostic] NB 51216 nbpem 116 PID:8022 TID:14 File ID:116 [jobid=-1 job_group_id=-1 client=bkpema15t1 type=2

server=*NULL* task=ID:1017c03b8 CTX:1017c03b8 policy=bkpema15t_emadb_full] 1 V-116-215 [ExtReqTask::taskCompleted] jobid=-1 submitted to nbjm

for processing
03/14/17 06:01:22.386 [Application] NB 51216 nbpem 116 PID:8022 TID:13 File ID:116 [jobid=35043 job_group_id=-1 client=*NULL* type=2

server=*NULL* task=ID:0 policy=*NULL*] [Info] V-116-14 CLIENT bkpema15t1  POLICY bkpema15t_emadb_full  SCHED *NULL*  EXIT STATUS 247 (the

specified policy is not active) VBRF 1 247

The platform of the Netbackup is : SPARC M3000 Solaris 10

NBU version is: 7.7.3

quebek
Moderator
Moderator
   VIP    Certified

Is not this policy scheduled on client end?? You need to go to the client and have DBA or anyone else who is scheduling it to stop doing this.... I would have checked the client crontab (if unix based) or schedule tasks if windows based...

Nicolai
Moderator
Moderator
Partner    VIP   

Agree, this looks like it is a client initiated backup. All database backup like DB2, SAP and Oracle are in principle client initiated backup where netbackup doesn't control when to start the backup.

Below the script:

# /opt/openv/netbackup/bin/admincmd/bppllist bkpema15t_emadb_full -U |grep Include
  Include:  /opt/sogdb/maint/emadb/emadb_backup_level0.sh

/opt/sogdb/maint/emadb/emadb_backup_level0.sh

CUSER=`id |cut -d"(" -f2 | cut -d ")" -f1`

if [ "$CUSER" = "root" ]

then

     su - emadba -c "/opt/sogdb/maint/emadb/backup_db.sh backup_level_0"

else

    /usr/bin/sh -c "/opt/sogdb/maint/emadb/backup_db.sh backup_level_0"

Fi

I will share soon the root crontab from the client.

Tape_Archived
Moderator
Moderator
   VIP   

The backup selection inculdes a script and script suggests this a DB backup. Policy is triggered by the client server, please reivew your "cron" and once you find the scheudle for this script, hash it out or delete it from crontab. 

Here is the crontab in the client:

40 03 * * * root /opt/sogdb/maint/db_maint_tool.sh emadba rotate_dump_log

30 01 * * * root /opt/sogdb/maint/db_maint_tool.sh emadba remove_backup_file >/dev/null 2>&1

30 06,14,22 * * * root /opt/sogdb/maint/db_maint_tool.sh emadba standby_archive_log >/dev/null 2>&1

30 03 * * * root /opt/sogdb/maint/log_maint_tool.sh emadba rotate_dump_log

45 01 * * * root /opt/sogdb/maint/log_maint_tool.sh emadba remove_proclog_file >/dev/null 2>&1

00 00,04,06,08,10,12,14,16,18,20 * * * root /opt/sogdb/maint/log_maint_tool.sh emadba shrink_temp

30 00,04,06,08,10,12,14,16,18,20 * * * root /opt/sogdb/maint/log_maint_tool.sh emadba shrink_undo

38 18 * * * root /opt/sogdb/maint/backup_conf.sh all

30 00,04,06,08,10,12,14,16,18,20,22 * * * root /opt/sogdb/maint/emadb/rotate_audit_log.sh

00 00 * * * root /opt/sogdb/maint/emadb/rotate_data_guard_log.sh

*/5 * * * * root /opt/sogdb/bin/dataguardAlarm.sh

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
Check crontab for other users as well.
Such as oracle, emadba, etc.

sogadm@ema15t1:/etc/cron.d> more ema_maint_cron

 

45 04 * * * root /opt/sog/maint/sog_dump.sh

40 04 * * * root /opt/sog/maint/rotate_all_logs daily

42 04 * * * root /opt/sog/maint/remove_old_logs.sh -f 7

 

 

Regarding emadba,  there is no job :

 

emadba@ema15t1:~> crontab -l

no crontab for emadba

 

I will check for oracle crontab.

I confirm there is no cron for oracle user.

Hello

 

Interesting....

Have you tried to run nbpemreq command and check any future schdule in queue for this policy?

#nbpemreq -predict

Please check, is there any folder exist in class folder with policy name.

Thanks & Regards

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
From the evidence that you posted earlier, it is clear that the master has received a user-directed request.

You can check bprd log on the master to see which IP address the request is coming from. After that, there is nothing more that can be done from NBU side.
You need to check that particular server/client to find out what/how the script is called.
Like a thirdparty scheduler. Or a dba manually running scripts, etc...

quebek
Moderator
Moderator
   VIP    Certified

hey 

other option you may face (I faced here in my env): UNIX team cloned some prod machine to a sandbox, forgetting to disable crontabs - then I took me hours to figure out from which IP/Client were coming these 'fake' backup request as my DBA said as they are unware of a this DB job failing...

So check with your unix team if they did not clone a prod machine to something new....

The policy doesn't appear in queue:

 # /opt/openv/netbackup/bin/admincmd/nbpemreq -due -date 03/29/2017 23:59:59
Predicted work assuming no job run between now and Wed Mar 29 23:59:59 2017
      rl client                    policy                    schedule                  type  Due time
----- -- ------------------------- ------------------------- ------------------------- ----- -------------------------
# /opt/openv/netbackup/bin/admincmd/nbpemreq -predict -date 03/29/2017 23:59:59
Predicted work assuming no job run between now and Wed Mar 29 23:59:59 2017
rl client               policy               schedule             type
-- -------------------- -------------------- -------------------- ----------
#

 

And, yes, there is a directory with the policy name under class:

# cd /opt/openv/netbackup/db/class
# ls -l

...

drwxr-xr-x   3 root     root         512 Mar 22 14:50 bkpema15t_emadb_full

....

# ls -l bkpema15t_emadb_full
total 10
-rw-------   1 root     root         147 Mar 16 15:31 clients
-rw-------   1 root     root          46 Mar 16 15:31 includes
-rw-------   1 root     root        1447 Mar 22 14:50 info
drwxr-xr-x   4 root     root         512 Mar 16 15:31 schedule

root@Backup # ls -l bkpema15t_emadb_full
total 10
-rw-------   1 root     root         147 Mar 16 15:31 clients
-rw-------   1 root     root          46 Mar 16 15:31 includes
-rw-------   1 root     root        1447 Mar 22 14:50 info
drwxr-xr-x   4 root     root         512 Mar 16 15:31 schedule
root@Backup #
root@Backup # ls -l bkpema15t_emadb_full/schedule/
total 4
drwxr-xr-x   2 root     root         512 Mar 16 15:31 Default-Application-Backup
drwxr-xr-x   2 root     root         512 Mar 16 15:31 Full
root@Backup #
root@Backup # ls -l bkpema15t_emadb_full/schedule/Full/
total 6
-rw-------   1 root     root         401 Mar 16 15:31 Calendar
-rw-------   1 root     root          42 Mar 16 15:31 days
-rw-------   1 root     root         473 Mar 16 15:31 info
root@Backup # cat bkpema15t_emadb_full/schedule/Full/days
0 0 0
1 0 0
2 0 0
3 0 0
4 0 0
5 0 0
6 0 0
root@Backup # cat bkpema15t_emadb_full/schedule/Full/info
SCHED_TYPE 0
RETENTION_LEVEL 9 1 1 1 1 1 1 1 1 1
INCREMENTAL_TYPE 0
FREQUENCY 2419200
CALENDAR 0
MAX_FRAG_SIZE 0
MAX_MPX 1
NUMBER_OF_COPIES 0
FAIL_ON_ERROR 0
SYNTHETIC 0
CHECKSUM_VALIDATION 0
PFI_RECOVERY 0
RESIDENCE *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL*
POOL *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL*
SHARE_GROUP *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL*
RES_IS_SS 0
SCHED_INDEXING 0
root@Backup # cat bkpema15t_emadb_full/schedule/Full/Calendar
[General]
Object Type={C2EDA523-69C6-11D2-8A7C-0000C0B839E5}
Object Version=0x20032
m_ScheduleID={0F385820-1DD2-11B2-98D4-0003BA12F730}
m_StartTime=0x58C9C770
m_EndTime=0x58CB18EF
m_TimeDuration=0x1517F
m_bUseEffectiveDate=FALSE
m_dtEffectiveDate=0x4ABAFCD0
m_bUseRestartInterval=FALSE
m_RestartInterval=0x0
m_bUseRepeatInterval=FALSE
m_iRepeatInterval=0x2
m_AnchorDate=0x4ABAFCD0
m_CustomBitSet=0x0

Hello,

 

Kindly rename the folder and script files.

bkpema15t_emadb_full

/opt/sogdb/maint/emadb/emadb_backup_level0.sh

/opt/sogdb/maint/emadb/backup_db.sh

/opt/sogdb/maint/emadb/backup_db.sh 

Restart the NBU services on Master server.

If still facing issue then enable the bprd logs on master server and share bprd log, failed job details status and original client server IP address.

 Note: We need IP address to confirm the request coming from original client server.

 

Regards,

Hello,

To add my previous post. Delete the policy folder from class folder. In some case I noticed if you rename or any corrupt policy cause to nbpem failure.

 

Thanks & Regards,

 

Hi Tousif,

Can I move the folder to another place to not loose it and can use it later.