09-28-2020 12:15 PM
Hello All,
One of our customers wants to know why there are some 'ghost' restore jobs on the activity monitor every Sunday night.
These jobs start at midnight (00:00) every Sunday and some errors are reported in Problem reports at the same time.
error messages :
Error 0 General Could not connect to for virtual browse operation, errno=0, bpcd_status=48
11 messages for 11 restore jobs.
The restore jobs don't show anything only requesting resources and status code is 0.
we looked up into bprd, nbpem, nbjm, bpdbm logs, and all what I can find is some information regarding the execution time and something like :
9/20/2020 00:00:57.898 [ExportRequest::getName] (00000000038B4E50) request has no name(ExportedResource.cpp:1965)
(000000000505BD40) NO matching exported resource for request of type=13, reqNum=0, consume=false(ExportedResourceMgr.cpp:626)
in bprd logs, I can see that there is an nbdeployutil started at the same time but I don't know from which server has initiated it.
We checked the task scheduler on master, opscenter we found nothing and on the opscenter, in reports, we didn't find any report that is executed at that time.
But I suspect this comes from the Opscenter, because of this technote
https://www.veritas.com/support/en_US/article.100028008.
On the scl.conf file, this line was already commented :
#nbu.scl.collector.enableBreakupJobDataCollection=false.
Any ideas please?
Thanks,
Master server: Windows Server 2016
NBU version: 8.2
Solved! Go to Solution.
09-28-2020 04:18 PM
Not sure I can fully answer your question, but I can give you some help.
As for the ghost restore jobs - no idea, I've never seen that before. Can you share some more details on the jobs themselves (such as job details, assuming there are any)?
Refer to his URL for details on how to control the nbdeployutil scheduling to see if this is causing the ghost jobs: https://www.veritas.com/content/support/en_US/doc/24437881-136359134-0/v137470892-136359134
10-05-2020 04:09 AM
Hello @davidmoline ,
I just got an update regarding this case, by disabling nbedployutil (by setting the value FREQUENDY_BY_DAYS=0 in nbdeployutilconfig.txt) the client didn't get any error or restore jobs and he's happy with it..
I don't know if disabling nbdeployutil is a good thing or not? could you please advise regarding this?
Thank you;
09-28-2020 04:18 PM
Not sure I can fully answer your question, but I can give you some help.
As for the ghost restore jobs - no idea, I've never seen that before. Can you share some more details on the jobs themselves (such as job details, assuming there are any)?
Refer to his URL for details on how to control the nbdeployutil scheduling to see if this is causing the ghost jobs: https://www.veritas.com/content/support/en_US/doc/24437881-136359134-0/v137470892-136359134
09-29-2020 05:11 AM - edited 09-29-2020 05:12 AM
Hello @davidmoline,
Thank you for your reply, I appreciate it.
you actually kind of answered one of my main questions which is about nbdeployutil getting started every Sunday night at midnight so we might know what is causing those restore jobs
regarding restore jobs in AM, as described in the technote, It has been found that data collection inside of OpsCenter can trigger these restore jobs, so this can happen.
the restore jobs appear with only a resource request and resource granted message inside of the job's detailed status section.no other information:
Sep 09, 2020 12:01:02 AM - requesting resource @aaaad
Sep 09, 2020 12:01:03 AM - granted resource MediaID=@aaaad;DiskVolume=PureDiskVolume;DiskPool=DISKPOOL01;Path=PureDiskVolume;StorageServer=Mediaserver.com;MediaServer=Mediaserver.com
the requested operation was successfully completed (0)
Thanks again for your help.
I have asked the client to send me the file nbdeployutilconfig.txt to take a look at it.
09-29-2020 03:09 PM
Hi @Hamza_H two things - first is that the nbdeployutilconfig.txt file only exists if you want to change something (i.e. it is not there by default - and then the default values are used as shown in the link I provided. Second is that you should have the customer try uncommenting that parameter from the old technote to see if it makes a difference.
09-29-2020 03:43 PM
09-30-2020 07:05 PM
Hi @Hamza_H
Out of curiousity I set the scl.conf parameter to true for my 8.3 lab. When I did this, I was seeing about 60 "ghost" restore jobs in my activity monitor every hour (on the hour), starting shortly after restarting OpsCenter.
Depending on how many times OpsCenter has been upgraded, it may be that this value needs to be set explicitly to false. Although you are only seeing them once per week, not hourly.
Hmm - mroe interesting.
David
10-01-2020 02:46 AM
Hello @davidmoline ,
Thank you for your reply,
was the line commented in your scl.conf?
#nbu.scl.collector.enableBreakupJobDataCollection=false
then you changed it to:
nbu.scl.collector.enableBreakupJobDataCollection=true ?
could you please do this test (do not comment it) and tell me if you still have ghost restore jobs?
nbu.scl.collector.enableBreakupJobDataCollection=false
10-01-2020 04:29 AM
Hi @Hamza_H
Indeed, it was originally commented out as you show.
I then uncommented the line changed the value to true - this is when the ghost restores started.
I then changed the value back to false (rather than commenting out the line again) - so the line is there setting the value explicitly to false. The restores stopped appearing.
Cheers
David
10-01-2020 05:30 AM
Hello @davidmoline ,
Good, so I think we should comment out the line and set it to false and then wait.
I already asked the client to disable nbdeployutil for this weekend (FREQUENCY_BY_DAYS=0) to confirm - or not, that this is what is causing the restore jobs every Sunday.
I will let you know as soon as I get an answer.
Again, thank you so much for your tests and especially the time you took to help us :)
10-02-2020 12:55 AM
Hi @Hamza_H
I've done some more investigation. It doesn't appear to make a difference whether the parameter “nbu.scl.collector.enableBreakupJobDataCollection” is explicitly set to false or just commented out after being set to true. This is in my 8.3 lab with both scenarios tested after having the value set to true, both stopped the ghost restores.
What appears to be happening on my 8.3 lab is that OpsCenter (?) is attempting to perform a list/restore of the active directory backups from the last 30 days every hour (which in my lab is 60 restore jobs). Why this is happening is another question. Why it is choosing the AD backups is also a mystery (although the backup policy is the first by alphbetical order).
The relevant difference in the logs that triggers the ghost restores is something that is shown as "image_db Q_IMAGE_LIST_FILES" which in turn run nbfsd to look at a granular recovery of the AD backups.
I’ve attached a combined extract of most of the relevant logs from a single restore event. The vxlogs are combined into a single extract.
Gets more interesting
10-02-2020 04:14 AM
Hello @davidmoline,
Do you have errors associated with these restore jobs in the Problems report?
The errors we have in the Problems report are saying 'Could not connect to for virtual browse operation'.. I don't know if it has a relation with AD backups or another type of backup/operation.
Do you have any report in your ops center that is running every 1hour? any nbdeployutil command in the logs?
I am still waiting for this weekend to see what would happen..
and yes as you said, this gets more interesting and surly weird because we don't really know what is causing these errors..
Unless mistaken, I don't see the attached extract.
10-02-2020 04:05 PM
Hi @Hamza_H - ah yes - I had a malfunction when I first tried to reply (with the file attached) and forgot to add the second time. Attached now.
No errors shown with the restores - all are successful.
I only have one regular scheduled report which runs daily at 6am. That's it. But don't forget OpsCenter is continually collecting data from the master server at various time intervals (from 15 mins to 4 hours).
The only mention is see of nbdeployutil in the logs is the bprd log file which shortly after midnight each day shows this 4 times:
00:00:01.849 [2574828.2574828] <2> run_nbdeployutil: Successfully set environment variable.
Cheers
David
10-02-2020 04:15 PM
10-05-2020 04:09 AM
Hello @davidmoline ,
I just got an update regarding this case, by disabling nbedployutil (by setting the value FREQUENDY_BY_DAYS=0 in nbdeployutilconfig.txt) the client didn't get any error or restore jobs and he's happy with it..
I don't know if disabling nbdeployutil is a good thing or not? could you please advise regarding this?
Thank you;
10-05-2020 07:09 AM
You can manually run nbdeployutil when you'd like to receive that information. If you want it weekly, you could set up a cron job or script that initiates at the specific time, or just make it a part of your regular work routine.
Only caveat I can think of is that disabling may make Smart Meter have issues. I can not say whether this is the case, as I have not seen that happen, but I could see it being a "gotcha" down the road if there are inconsistencies observed.
10-05-2020 02:59 PM
Hi @Hamza_H
There is no problem having the scheduled incremental nbdeployutil disabled. What your customer will "miss" out on is the ability to track license usage via Smart Meter (and also as a consequence, your Veritas account manager is unable to track this as well - not that, in my experince, many do).
Nbdeployutil can always be run manually when desired/required/requested.
Cheers
David
10-06-2020 05:03 AM
Hi @EthanH ,
Thanks for your reply,
I am actually aware of that, I am just wondering if disabling it could be a 'bad' thing to do or may have some impact somewhere..
You can schedule it as you want by creating the file nbdeployutilfconfig.txt and setting the frequency that suits you..
10-06-2020 05:05 AM - edited 10-06-2020 05:05 AM
Thanks a lot @davidmoline for your assistance :)
I will mark one of your replies as a solution.