03-23-2017 02:35 AM
Hi experts,
I have de-activated a policy in NetBackup, but it is still running. I have deleted it but no way.
What can I do to stop the policy from running while it is de-activated.
I have done the folloing but didn't solve the issue:
1. Stop NetBackup Services on the master server
2. Remove the following files and directories from the master server
/usr/openv/netbackup/bin/bpsched.d/*
/usr/openv/netbackup/bin/bprd.d/*
/usr/openv/netbackup/db/jobs/pempersist2
/usr/openv/netbackup/db/jobs/restart/*
3. Start NetBackup Services
4. Run the command below to force nbpem to re-read configurations and rebuild its cache instead of waiting the default time to do so:
/usr/openv/netbackup/bin/admincmd/nbpemreq -updatepolicies
Thanks
03-23-2017 03:23 AM
Hello, I need to admit you are brave ;)
I am not really following your issue! So you have a policy which is currently running in activity monitor? Did you try to cancel this policy? After restart of NBU it should be in state like cancelled or pending restart... If you deactivated this policy it should not be kicked off any longer...
Please provide you NBU version and describe the issue more...
03-23-2017 03:28 AM
I have missed one point:
When running, the job gives error code 247 and of course stops running in Activity Monitor. See error:
03/13/17 06:01:23.664 [Info] V-116-14 CLIENT bkpema15t1 POLICY bkpema15t_emadb_full SCHED *NULL* EXIT STATUS 247 (the specified policy is not active) VBRF 1 247
NBU version is: 7.7.3
03-23-2017 03:41 AM
Hi AMar-khodja , check the output of bppllist <polictname> -U and verify the value for attribute "Active" . This wil verify if this attributed is updated for this policy in Netbackup database.If it stil shows "yes" that means it is not updated (issue with GUI -> not able to update policy attribute) .
For analysis purpose you may need to enable / review nbpem logs .
BTW What is the platform and version of Netbackup ?
DeepS
03-23-2017 04:15 AM
# /opt/openv/netbackup/bin/admincmd/bppllist bkpema15t_emadb_full -U |grep -i active
Active: no
Find below nbpem logs:
# /opt/openv/netbackup/bin/vxlogview -p 51216 -i 116 -d all -t 24:00:00 | grep bkpema15t_emadb_full
03/14/17 06:01:22.375 [Application] NB 51216 nbpem 116 PID:8022 TID:14 File ID:116 [jobid=-1 job_group_id=-1 client=bkpema15t1 type=2
server=*NULL* task=ID:1017c03b8 CTX:1017c03b8 policy=bkpema15t_emadb_full] [Info] V-116-82 received start user backup request for
client=bkpema15t1, policy=bkpema15t_emadb_full, schedule=?, stream #=0, client type=Oracle (4), parent jobid=-1
03/14/17 06:01:22.375 [Diagnostic] NB 51216 nbpem 116 PID:8022 TID:14 File ID:116 [jobid=-1 job_group_id=-1 client=bkpema15t1 type=2
server=*NULL* task=ID:1017c03b8 CTX:1017c03b8 policy=bkpema15t_emadb_full] 1 V-116-189 [ExtReqTask::run_specificPolicy] Failed to retrieve
policy bkpema15t_emadb_full, 247
03/14/17 06:01:22.375 [Debug] NB 51216 nbpem 116 PID:8022 TID:14 File ID:116 [jobid=-1 job_group_id=-1 client=bkpema15t1 type=2 server=*NULL*
task=ID:1017c03b8 CTX:1017c03b8 policy=bkpema15t_emadb_full] 1 [PemTask::taskComplete] (ID:1017c03b8 CTX:1017c03b8) Active subtask count=0
(PemTask.cpp:530)
03/14/17 06:01:22.384 [Diagnostic] NB 51216 nbpem 116 PID:8022 TID:14 File ID:116 [jobid=-1 job_group_id=-1 client=bkpema15t1 type=2
server=*NULL* task=ID:1017c03b8 CTX:1017c03b8 policy=bkpema15t_emadb_full] 1 V-116-215 [ExtReqTask::taskCompleted] jobid=-1 submitted to nbjm
for processing
03/14/17 06:01:22.386 [Application] NB 51216 nbpem 116 PID:8022 TID:13 File ID:116 [jobid=35043 job_group_id=-1 client=*NULL* type=2
server=*NULL* task=ID:0 policy=*NULL*] [Info] V-116-14 CLIENT bkpema15t1 POLICY bkpema15t_emadb_full SCHED *NULL* EXIT STATUS 247 (the
specified policy is not active) VBRF 1 247
The platform of the Netbackup is : SPARC M3000 Solaris 10
NBU version is: 7.7.3
03-23-2017 04:20 AM
Is not this policy scheduled on client end?? You need to go to the client and have DBA or anyone else who is scheduling it to stop doing this.... I would have checked the client crontab (if unix based) or schedule tasks if windows based...
03-23-2017 04:28 AM - edited 03-23-2017 04:30 AM
Agree, this looks like it is a client initiated backup. All database backup like DB2, SAP and Oracle are in principle client initiated backup where netbackup doesn't control when to start the backup.
03-23-2017 05:14 AM
Below the script:
# /opt/openv/netbackup/bin/admincmd/bppllist bkpema15t_emadb_full -U |grep Include
Include: /opt/sogdb/maint/emadb/emadb_backup_level0.sh
/opt/sogdb/maint/emadb/emadb_backup_level0.sh
CUSER=`id |cut -d"(" -f2 | cut -d ")" -f1`
if [ "$CUSER" = "root" ]
then
su - emadba -c "/opt/sogdb/maint/emadb/backup_db.sh backup_level_0"
else
/usr/bin/sh -c "/opt/sogdb/maint/emadb/backup_db.sh backup_level_0"
Fi
I will share soon the root crontab from the client.
03-23-2017 05:49 AM
The backup selection inculdes a script and script suggests this a DB backup. Policy is triggered by the client server, please reivew your "cron" and once you find the scheudle for this script, hash it out or delete it from crontab.
03-23-2017 01:32 PM
Here is the crontab in the client:
40 03 * * * root /opt/sogdb/maint/db_maint_tool.sh emadba rotate_dump_log
30 01 * * * root /opt/sogdb/maint/db_maint_tool.sh emadba remove_backup_file >/dev/null 2>&1
30 06,14,22 * * * root /opt/sogdb/maint/db_maint_tool.sh emadba standby_archive_log >/dev/null 2>&1
30 03 * * * root /opt/sogdb/maint/log_maint_tool.sh emadba rotate_dump_log
45 01 * * * root /opt/sogdb/maint/log_maint_tool.sh emadba remove_proclog_file >/dev/null 2>&1
00 00,04,06,08,10,12,14,16,18,20 * * * root /opt/sogdb/maint/log_maint_tool.sh emadba shrink_temp
30 00,04,06,08,10,12,14,16,18,20 * * * root /opt/sogdb/maint/log_maint_tool.sh emadba shrink_undo
38 18 * * * root /opt/sogdb/maint/backup_conf.sh all
30 00,04,06,08,10,12,14,16,18,20,22 * * * root /opt/sogdb/maint/emadb/rotate_audit_log.sh
00 00 * * * root /opt/sogdb/maint/emadb/rotate_data_guard_log.sh
*/5 * * * * root /opt/sogdb/bin/dataguardAlarm.sh
03-23-2017 03:45 PM
03-24-2017 03:51 AM
sogadm@ema15t1:/etc/cron.d> more ema_maint_cron
45 04 * * * root /opt/sog/maint/sog_dump.sh
40 04 * * * root /opt/sog/maint/rotate_all_logs daily
42 04 * * * root /opt/sog/maint/remove_old_logs.sh -f 7
Regarding emadba, there is no job :
emadba@ema15t1:~> crontab -l
no crontab for emadba
I will check for oracle crontab.
03-27-2017 10:58 AM
I confirm there is no cron for oracle user.
03-27-2017 09:30 PM
Hello
Interesting....
Have you tried to run nbpemreq command and check any future schdule in queue for this policy?
#nbpemreq -predict
Please check, is there any folder exist in class folder with policy name.
Thanks & Regards
03-27-2017 09:53 PM
03-28-2017 01:49 AM
hey
other option you may face (I faced here in my env): UNIX team cloned some prod machine to a sandbox, forgetting to disable crontabs - then I took me hours to figure out from which IP/Client were coming these 'fake' backup request as my DBA said as they are unware of a this DB job failing...
So check with your unix team if they did not clone a prod machine to something new....
03-29-2017 01:54 AM
The policy doesn't appear in queue:
# /opt/openv/netbackup/bin/admincmd/nbpemreq -due -date 03/29/2017 23:59:59
Predicted work assuming no job run between now and Wed Mar 29 23:59:59 2017
rl client policy schedule type Due time
----- -- ------------------------- ------------------------- ------------------------- ----- -------------------------
# /opt/openv/netbackup/bin/admincmd/nbpemreq -predict -date 03/29/2017 23:59:59
Predicted work assuming no job run between now and Wed Mar 29 23:59:59 2017
rl client policy schedule type
-- -------------------- -------------------- -------------------- ----------
#
And, yes, there is a directory with the policy name under class:
# cd /opt/openv/netbackup/db/class
# ls -l
...
drwxr-xr-x 3 root root 512 Mar 22 14:50 bkpema15t_emadb_full
....
# ls -l bkpema15t_emadb_full
total 10
-rw------- 1 root root 147 Mar 16 15:31 clients
-rw------- 1 root root 46 Mar 16 15:31 includes
-rw------- 1 root root 1447 Mar 22 14:50 info
drwxr-xr-x 4 root root 512 Mar 16 15:31 schedule
root@Backup # ls -l bkpema15t_emadb_full
total 10
-rw------- 1 root root 147 Mar 16 15:31 clients
-rw------- 1 root root 46 Mar 16 15:31 includes
-rw------- 1 root root 1447 Mar 22 14:50 info
drwxr-xr-x 4 root root 512 Mar 16 15:31 schedule
root@Backup #
root@Backup # ls -l bkpema15t_emadb_full/schedule/
total 4
drwxr-xr-x 2 root root 512 Mar 16 15:31 Default-Application-Backup
drwxr-xr-x 2 root root 512 Mar 16 15:31 Full
root@Backup #
root@Backup # ls -l bkpema15t_emadb_full/schedule/Full/
total 6
-rw------- 1 root root 401 Mar 16 15:31 Calendar
-rw------- 1 root root 42 Mar 16 15:31 days
-rw------- 1 root root 473 Mar 16 15:31 info
root@Backup # cat bkpema15t_emadb_full/schedule/Full/days
0 0 0
1 0 0
2 0 0
3 0 0
4 0 0
5 0 0
6 0 0
root@Backup # cat bkpema15t_emadb_full/schedule/Full/info
SCHED_TYPE 0
RETENTION_LEVEL 9 1 1 1 1 1 1 1 1 1
INCREMENTAL_TYPE 0
FREQUENCY 2419200
CALENDAR 0
MAX_FRAG_SIZE 0
MAX_MPX 1
NUMBER_OF_COPIES 0
FAIL_ON_ERROR 0
SYNTHETIC 0
CHECKSUM_VALIDATION 0
PFI_RECOVERY 0
RESIDENCE *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL*
POOL *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL*
SHARE_GROUP *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL* *NULL*
RES_IS_SS 0
SCHED_INDEXING 0
root@Backup # cat bkpema15t_emadb_full/schedule/Full/Calendar
[General]
Object Type={C2EDA523-69C6-11D2-8A7C-0000C0B839E5}
Object Version=0x20032
m_ScheduleID={0F385820-1DD2-11B2-98D4-0003BA12F730}
m_StartTime=0x58C9C770
m_EndTime=0x58CB18EF
m_TimeDuration=0x1517F
m_bUseEffectiveDate=FALSE
m_dtEffectiveDate=0x4ABAFCD0
m_bUseRestartInterval=FALSE
m_RestartInterval=0x0
m_bUseRepeatInterval=FALSE
m_iRepeatInterval=0x2
m_AnchorDate=0x4ABAFCD0
m_CustomBitSet=0x0
03-29-2017 02:25 AM
Hello,
Kindly rename the folder and script files.
bkpema15t_emadb_full
/opt/sogdb/maint/emadb/emadb_backup_level0.sh
/opt/sogdb/maint/emadb/backup_db.sh
/opt/sogdb/maint/emadb/backup_db.sh
Restart the NBU services on Master server.
If still facing issue then enable the bprd logs on master server and share bprd log, failed job details status and original client server IP address.
Note: We need IP address to confirm the request coming from original client server.
Regards,
03-29-2017 07:12 PM
Hello,
To add my previous post. Delete the policy folder from class folder. In some case I noticed if you rename or any corrupt policy cause to nbpem failure.
Thanks & Regards,
03-30-2017 02:39 AM
Hi Tousif,
Can I move the folder to another place to not loose it and can use it later.