11-22-2018 11:14 AM
Hey fellow guru's,
I'm a long time fellow NetBackup geek, under a new profile due to a change in employment. I've stumbled onto something I've not seen before, and so far, my google fu coupled with my years of working with NB is more foo than anything on this.
I'm managing a NB 7.7.2 env. that isn't particularily complicated in setup - Master, couple of media servers, some libraries and a bit of MSDP - all build your own.
I've just recently finished creating a whole raft of new policies and making adjustments to a few existing ones and run into the weirdest thing. I've got a couple of policies that, for whatever reason, I cannot modify via the GUI. I've been trying to adjust the schedules to put a bit more time buffer between the start of each policy (they all impact the same client) but the changes will not take via the GUI no matter what account I use to run the Java GUI with on Windows 2012.
I discovered I can manipulate the policies via bpplschedrep successfully, but when I tried to use the policy as named (All caps) it barked. I re-ran the command using lower case, and the change took. The GUI even correctly reflects the new time values.
When you look at the output of bppplist, there's only the policies in question in all caps. When the jobs run, I get two jobs running - one for all upper case, one for all lower case. I've now got double the data being sent to tape needlessly, and this phatom policy that I can't list.
My current plan is to wait for the next minimal activity in the overall job window, and stop/start NB to see if this'll flush the duplicate job. That'll take a day or so to identify with the way the schedules break out since it's the daily incrementals that are impacted, and today's are done already.
After that, I'm going to have to do a dance around manipulating the incremental schedule to see if I can eliminate the lower case policy. I can't simply create a new policy and delete the old one, as encryption is in use, and I can't have the next batch of incrementals all start as fulls - load balancing on this environment has a number of challenges already, all outside of my control.
From what I can tell, other than the policy name being lower case, there is no difference in the two policies - there's no lower case version of this policy in the ../db/class folder, so this is currently some odd memory state with bpjobd and nbpem (and possibly the nbdb and emm) from what I can tell.
Thoughts on the plan of action? Anyone else end up here and find a clean way out (not deleting and recreating policies)?
Thanks in advance for reading, and replying if you feel so inclined. :)
11-22-2018 11:35 PM
Firstly, there's a bug in NBU 7.7.2 and 7.7.3 related to editing of schedules in the GUI.
There is a hotfix for this: https://www.veritas.com/support/en_US/article.000109333
About the problem with duplicate jobs - all I have to suggest is to try
but this process is supposed to automatically run every 10 minutes.
nbpemreq -predict_all may help to identify future duplicate jobs until you can restart NBU/nbpem.
Then kill them as soon as they start.
11-23-2018 11:28 AM
Thanks for pointing out the bug and hotfix I wasn't finding it in my searches as I was dealing with a Frequency based schedule, not a calendar based one, but it fits. I may or may not be upgrading this environment in the new year - holiday change freeze will be coming soon and I won't have enough time to put an upgrade plan in front of my customer's management to review and bless. If we do, it'll go to NB 8.x.
I'd hoped the auto-run of the policy updates would have resolved it as well, but with a couple of days of ghost policies creating duplicate work, that was clearly not going to work.
The stop/start of the master seems to have cleared it (I got down to fewer than 10 active jobs so I made the call to bring things down now, before the 5 pm kick off of the weekend fulls) - one of the incrementals ran immediately after I restarted things, and only one job ran. A quick review of the nbpemreq -predict_all output for the next three days out shows only the desired policies are running. I should have a solid yea/nay on this by this time tomorrow.
11-26-2018 07:39 AM
So, things look a lot better after this weekend's runs. No more ghost policies! whew. :)