09-25-2014 07:50 AM
Hi Team,
We are facing one strange issue. Some one create a policy with xxxxxx/ ( with slash) and this backup is failing with EC-84, but Original backup is running fine without any issue. We have deleted the policy from GUI, but still it is running. We have checked in db/cass/ folder there we find the / policy name. We have deleted the policy there also. But still it is running. No idea, where it is running.
We have updated/run the below command, but no use. Stil the job is running.
nbpemreq -updatepolicies
is any help how to delete the policy from the NBU without recycle the NBU services ?
Thanks for your help.
Solved! Go to Solution.
10-01-2014 08:58 AM
Since the policy has been deleted and continues to run for whatever the reason, first thing you should do is rename the script the job runs on the Oracle client. This will prevent the job from running albeit with a failed status code attached to it but at least you stop it from running. At that point I would certainly look to restart the services for NetBackup WITHOUT rebooting your production servers. Make sure you suspend or cancel jobs currently running. Make note of the cancelled jobs and restart them once the services come back up. Resume any suspended jobs.
Questions...
Snipped from one of your posts to Marianne
"I can see the original policy only, but not the "\" policy, which we already removed from the NBU policy GUI." ( I think you meant "/" in this sentence..yes?)
What is this original policy? Does it have the same name sans the /??????????
Also, out of curiosity have you attempted to run the find command on the policy name? I ask this because I don't know how a policy was able to be created with a "/" in the string to begin with and I'm curious to see if there may now be a directory out there with the name DB-ORA-DB_SID-ARCH-NET. Just for kicks and giggles run the following
# find / -name DB-ORA-*
or
# find / -name DB-ORA-* -print (If Solaris)
09-25-2014 08:09 AM
You could try a bppldelete policyname but i suspect that it is just embeded in the scheduler.
Last time i saw one like this i had to recreate the policy with the same name .. add a schedule with the same name but with no window - run the nbpemreq command again to clear it out of the scheduled timers list and then delete the policy.
See if that works for you
09-25-2014 08:39 AM
Do you if there's a script launching the backup from its client?
We faced that a time ago and we need to check the client's crontab.
09-25-2014 11:33 AM
By still running do you mean that the policy is still listed or are you refering to an active job from that policy?
If an active job, Get the job numbers by running /usr/openv/netbackup/bin/admincmd/bpdbjobs -report
Then kill those jobs with by running <install path>/bin/admincmd/bpdbjobs -delete <job number [job number,job number,...]>
09-25-2014 07:39 PM
Hi ,
Try to find the lock file under /usr/openv/netbackup/bin which should be cleanup.lock then Check which process is using it using fuser command in UNIX/LINUX.
Check if you can kill the process that hold that file then move the file to somewhere else like cleanup.lock.orig then try to delete the policy again.
09-25-2014 09:18 PM
Hi All,
Thank you for comment here,
There are no jobs in crontab.
It is very difficult to cancel the jobs. Since it is archive backup ( it will run every hour)
Unable to delete the policy, below is the error.
sudo ./bppldelete DB-ORA-DB_SID-ARCH-NET/
failed accessing daemon lock file
Please help here
09-26-2014 01:22 AM
Hi Leed,
Thanks for your information, I tried below, but no luck.
ls -ltr cleanup.loc*
-rw------- 1 root root 17 Apr 23 2013 cleanup.lock
-rw------- 1 root root 17 Sep 26 12:44 cleanup.lock_1
-rw------- 1 root root 17 Sep 26 12:44 cleanup.lock_2
-rw------- 1 root root 17 Sep 26 12:45 cleanup.lock_3
-rw------- 1 root root 17 Sep 26 13:39 cleanup.lock_4
$ fuser cleanup.lock_1
cleanup.lock_1:
$ fuser cleanup.lock_2
cleanup.lock_2:
$ sudo fuser cleanup.lock_2
cleanup.lock_2:
$ sudo fuser cleanup.lock_3
cleanup.lock_3:
$ sudo fuser cleanup.lock_4
cleanup.lock_4:
$ sudo fuser cleanup.lock
cleanup.lock:
sudo ./bppldelete DB-ORA-DB_SID-ARCH-NET/
failed accessing daemon lock file
But I can found below lock files .
ls -ltr *.lock*
-rw------- 1 root root 17 Oct 30 2010 sync.lock
-rw------- 1 root root 17 Apr 23 2013 cleanup.lock
-rw------- 1 root root 16 Jul 10 12:34 bpcompatd.lock
-rw------- 1 root root 16 Jul 10 12:34 bpdbm.lock
-rw------- 1 root root 16 Jul 10 12:34 bpjobd.lock
-rw------- 1 root root 16 Jul 10 12:34 bprd.lock
-rw------- 1 root root 17 Sep 26 12:44 cleanup.lock_1
-rw------- 1 root root 17 Sep 26 12:44 cleanup.lock_2
-rw------- 1 root root 17 Sep 26 12:45 cleanup.lock_3
-rw------- 1 root root 17 Sep 26 13:39 cleanup.lock_4
-rw------- 1 root root 17 Sep 26 15:54 catbackup.lock
Please let me know, which file I can check.
09-26-2014 04:33 AM
Try to delete the policy by adding a \ before the / :
sudo ./bppldelete DB-ORA-DB_SID-ARCH-NET\/
(there is a \ at the end followed by / - not a V ! )
09-29-2014 06:05 PM
HI Marianne,
Thank you for comment here. I tried wheat you said as above, but still no luck. Same error again.
/usr/openv/netbackup/bin/admincmd/bppldelete DB-ORA-DB_SID-ARCH-NET\/ ( This is not v \ and /)
failed accessing daemon lock file
I think this is Ghost policy in our environmet.
09-29-2014 10:10 PM
Can you see this name in class folder?
cd /usr/openv/netbackup/db/class
ls -l DB-ORA-DB_SID-ARCH*
If so, we will be able to get rid of it at OS level.
09-30-2014 01:27 AM
Hi Marianne,
I can see the original policy only, but not the "\" policy, which we already removed from the NBU policy GUI. But no idea, where this orphend policy starting and backups are keep on failing. I spoke to some one from my company ( senior ), he said" reboot(restart the services) to solve the issue.
This is production server, we need take lot of approvals before reboot/restart services. If there is no chance, then I will raise change to reboot the master server.
Thanks
09-30-2014 01:44 AM
Can you please output of bppllist DB-ORA-DB_SID-ARCH-NET/ -L
How did you create this policy at the first place, / is not allowed in policy and I am wondering how you were able to create this policy.
It looks like a oracle policy and seems to be getting trigged via some kind of script
10-01-2014 07:56 AM
Hi Sazz,
I have told in my previous updates. some one created it and we have deleted the policy from NBU GUI. But no idea where now the job is running now. there is no class DB also.
Thanks,
10-01-2014 08:58 AM
Since the policy has been deleted and continues to run for whatever the reason, first thing you should do is rename the script the job runs on the Oracle client. This will prevent the job from running albeit with a failed status code attached to it but at least you stop it from running. At that point I would certainly look to restart the services for NetBackup WITHOUT rebooting your production servers. Make sure you suspend or cancel jobs currently running. Make note of the cancelled jobs and restart them once the services come back up. Resume any suspended jobs.
Questions...
Snipped from one of your posts to Marianne
"I can see the original policy only, but not the "\" policy, which we already removed from the NBU policy GUI." ( I think you meant "/" in this sentence..yes?)
What is this original policy? Does it have the same name sans the /??????????
Also, out of curiosity have you attempted to run the find command on the policy name? I ask this because I don't know how a policy was able to be created with a "/" in the string to begin with and I'm curious to see if there may now be a directory out there with the name DB-ORA-DB_SID-ARCH-NET. Just for kicks and giggles run the following
# find / -name DB-ORA-*
or
# find / -name DB-ORA-* -print (If Solaris)
10-01-2014 10:56 AM
10-01-2014 11:55 AM
http://www.symantec.com/business/support/index?page=content&id=TECH43177
http://www.symantec.com/connect/forums/unknown-job-stuck-activity-monitor
I know these are for jobs in the activity monitor, but if you also delete the temporary policy execution manager file in Install Location/netbackup/db/jobs (exact name escapes me at the moment) that will force it to be rebuilt upon the restarting of all the services.
If you truely were able to delete the policy, it should not reappear in the new work list created by nbpemreq.
Note: If the policy was an Oracle Policy which called RMAN scripts in order to execute, once launched the script will continue to run; whether or not there is a policy physically present in the Policy List on the Master.
Hope this helps.....kind of round about way of getting to where you want to be; but, only solves the problem if the problem was your policy was still being executed by PEM.
I have been no help at all, I realize, if you see the Policy in your Policy List and just want to delete it.
To delete the policy, drop all NBU services, go to Install Location/netbackup/db/class and simply delete the directory named: DB-ORA-DB_SID-ARCH-NET\/
#rm -fR "Install Location"/netbackup/db/class/policynametodelete (as Marianne suggested above)
Let us know if you solved your issue and what the final fix was....Thanks!