cancel
Showing results for 
Search instead for 
Did you mean: 

Ghost restore jobs on AM

Hamza_H
Moderator
Moderator
   VIP   

Hello All,

One of our customers wants to know why there are some 'ghost' restore jobs on the activity monitor every Sunday night.

These jobs start at midnight (00:00) every Sunday and some errors are reported in Problem reports at the same time.

error messages :

Error 0 General Could not connect to for virtual browse operation, errno=0, bpcd_status=48

11 messages for 11 restore jobs.

The restore jobs don't show anything only requesting resources and status code is 0.

we looked up into bprd, nbpem, nbjm, bpdbm logs, and all what I can find is some information regarding the execution time and something like :

9/20/2020 00:00:57.898 [ExportRequest::getName] (00000000038B4E50) request has no name(ExportedResource.cpp:1965)

(000000000505BD40) NO matching exported resource for request of type=13, reqNum=0, consume=false(ExportedResourceMgr.cpp:626)

in bprd logs, I can see that there is an nbdeployutil started at the same time but I don't know from which server has initiated it.

We checked the task scheduler on master, opscenter we found nothing and on the opscenter, in reports, we didn't find any report that is executed at that time.

But I suspect this comes from the Opscenter, because of this technote

https://www.veritas.com/support/en_US/article.100028008.

On the scl.conf file, this line was already commented :

#nbu.scl.collector.enableBreakupJobDataCollection=false.

Any ideas please?

Thanks,

Master server: Windows Server 2016

NBU version: 8.2

2 ACCEPTED SOLUTIONS

Accepted Solutions

davidmoline
Level 6
Employee

Not sure I can fully answer your question, but I can give you some help. 

  1. The NetBackup master server will run an incremental nbdeployutil every Sunday night at midnight. This is new and used by Smart Meter to provide for continual reporting back to Veritas on license usage. The files are available for you to examine - on Windows they can be found under <INSTALL>\Veritas\NetBackup\var\global\incremental (log files and previous nbdeployutil reports).
  2. The OpsCenter item would need to be uncommented to make a difference. Although that tech note relates to NetBackup 7.5.0.3 so not sure if it is relvant still (no harm in trying though). 

As for the ghost restore jobs - no idea, I've never seen that before. Can you share some more details on the jobs themselves (such as job details, assuming there are any)?

Refer to his URL for details on how to control the nbdeployutil scheduling to see if this is causing the ghost jobs: https://www.veritas.com/content/support/en_US/doc/24437881-136359134-0/v137470892-136359134

 

 

View solution in original post

Hamza_H
Moderator
Moderator
   VIP   

Hello @davidmoline ,

I just got an update regarding this case, by disabling nbedployutil (by setting the value FREQUENDY_BY_DAYS=0 in nbdeployutilconfig.txt) the client didn't get any error or restore jobs and he's happy with it.. 

I don't know if disabling nbdeployutil is a good thing or not? could you please advise regarding this?

 

Thank you;

View solution in original post

17 REPLIES 17

davidmoline
Level 6
Employee

Not sure I can fully answer your question, but I can give you some help. 

  1. The NetBackup master server will run an incremental nbdeployutil every Sunday night at midnight. This is new and used by Smart Meter to provide for continual reporting back to Veritas on license usage. The files are available for you to examine - on Windows they can be found under <INSTALL>\Veritas\NetBackup\var\global\incremental (log files and previous nbdeployutil reports).
  2. The OpsCenter item would need to be uncommented to make a difference. Although that tech note relates to NetBackup 7.5.0.3 so not sure if it is relvant still (no harm in trying though). 

As for the ghost restore jobs - no idea, I've never seen that before. Can you share some more details on the jobs themselves (such as job details, assuming there are any)?

Refer to his URL for details on how to control the nbdeployutil scheduling to see if this is causing the ghost jobs: https://www.veritas.com/content/support/en_US/doc/24437881-136359134-0/v137470892-136359134

 

 

Hamza_H
Moderator
Moderator
   VIP   

Hello @davidmoline,

Thank you for your reply, I appreciate it.

you actually kind of answered one of my main questions which is about nbdeployutil getting started every Sunday night at midnight so we might know what is causing those restore jobs

regarding restore jobs in AM, as described in the technote, It has been found that data collection inside of OpsCenter can trigger these restore jobs, so this can happen.

the restore jobs appear with only a resource request and resource granted message inside of the job's detailed status section.no other information:

Sep 09, 2020 12:01:02 AM - requesting resource @aaaad
Sep 09, 2020 12:01:03 AM - granted resource MediaID=@aaaad;DiskVolume=PureDiskVolume;DiskPool=DISKPOOL01;Path=PureDiskVolume;StorageServer=Mediaserver.com;MediaServer=Mediaserver.com
the requested operation was successfully completed (0)

Thanks again for your help.

I have asked the client to send me the file nbdeployutilconfig.txt to take a look at it.

Hi @Hamza_H two things - first is that the nbdeployutilconfig.txt file only exists if you want to change something (i.e. it is not there by default - and then the default values are used as shown in the link I provided. Second is that you should have the customer try uncommenting that parameter from the old technote to see if it makes a difference.

Hamza_H
Moderator
Moderator
   VIP   
Hello @David,
Thank you for your reply,
Yes I noticed that the file doesn’t exist by default so I asked the EC to create and to put below option in it and test for the next weekend (to confirm if this is what is really causing the restore jobs)
FREQUENCY_IN_DAYS = 0
Then if no nbdeployutil occurs this weekend, we will then uncomment that line in scl.conf and reset back the option in nbdeployutilconf file..
Thanks again for your help, I will keep you in touch.

Hi @Hamza_H 

Out of curiousity I set the scl.conf parameter to true for my 8.3 lab. When I did this, I was seeing about 60 "ghost" restore jobs in my activity monitor every hour (on the hour), starting shortly after restarting OpsCenter. 

Depending on how many times OpsCenter has been upgraded, it may be that this value needs to be set explicitly to false. Although you are only seeing them once per week, not hourly.

Hmm - mroe interesting.

David

Hamza_H
Moderator
Moderator
   VIP   

Hello @davidmoline ,

Thank you for your reply,

was the line commented in your scl.conf?

#nbu.scl.collector.enableBreakupJobDataCollection=false 

then you changed it to:

nbu.scl.collector.enableBreakupJobDataCollection=true ?

could you please do this test (do not comment it) and tell me if you still have ghost restore jobs?

nbu.scl.collector.enableBreakupJobDataCollection=false

 

Hi @Hamza_H 

Indeed, it was originally commented out as you show.

I then uncommented the line changed the value to true - this is when the ghost restores started.

I then changed the value back to false (rather than commenting out the line again) - so the line is there setting the value explicitly to false. The restores stopped appearing. 

Cheers
David

Hamza_H
Moderator
Moderator
   VIP   

Hello @davidmoline ,

Good, so I think we should comment out the line and set it to false and then wait.

I already asked the client to disable nbdeployutil for this weekend (FREQUENCY_BY_DAYS=0) to confirm - or not, that this is what is causing the restore jobs every Sunday.

I will let you know as soon as I get an answer.

Again, thank you so much for your tests and especially the time you took to help us :)

 

Hi @Hamza_H 

I've done some more investigation. It doesn't appear to make a difference whether the parameter “nbu.scl.collector.enableBreakupJobDataCollection” is explicitly set to false or just commented out after being set to true. This is in my 8.3 lab with both scenarios tested after having the value set to true, both stopped the ghost restores.

What appears to be happening on my 8.3 lab is that OpsCenter (?) is attempting to perform a list/restore of the active directory backups from the last 30 days every hour (which in my lab is 60 restore jobs). Why this is happening is another question. Why it is choosing the AD backups is also a mystery (although the backup policy is the first by alphbetical order).

The relevant difference in the logs that triggers the ghost restores is something that is shown as "image_db Q_IMAGE_LIST_FILES" which in turn run nbfsd to look at a granular recovery of the AD backups.

I’ve attached a combined extract of most of the relevant logs from a single restore event. The vxlogs are combined into a single extract.

Gets more interesting

Hamza_H
Moderator
Moderator
   VIP   

Hello @davidmoline,

Do you have errors associated with these restore jobs in the Problems report?

The errors we have in the Problems report are saying 'Could not connect to for virtual browse operation'.. I don't know if it has a relation with AD backups or another type of backup/operation.

Do you have any report in your ops center that is running every 1hour? any nbdeployutil command in the logs?

I am still waiting for this weekend to see what would happen..

and yes as you said, this gets more interesting and surly weird because we don't really know what is causing these errors..

Unless mistaken, I don't see the attached extract.

 

Hi @Hamza_H  - ah yes - I had a malfunction when I first tried to reply (with the file attached) and forgot to add the second time. Attached now.

No errors shown with the restores - all are successful.

I only have one regular scheduled report which runs daily at 6am. That's it. But don't forget OpsCenter is continually collecting data from the master server at various time intervals (from 15 mins to 4 hours).

The only mention is see of nbdeployutil in the logs is the bprd log file which shortly after midnight each day shows this 4 times:
00:00:01.849 [2574828.2574828] <2> run_nbdeployutil: Successfully set environment variable.

Cheers
David

Hamza_H
Moderator
Moderator
   VIP   
Hello @David,
Thank you for your reply,
We have exactly same errors

Bpcd :
Could not connect to for virtual browse operation, errno=2, bpcd_status=48
And it is the same error we have in problems report (we don’t have failing restore jobs, only successfull jobs the errors are seen in problems report)
Also nbpem logs you shared We have the same message as you :
Starting foreign jobs..

Hamza_H
Moderator
Moderator
   VIP   

Hello @davidmoline ,

I just got an update regarding this case, by disabling nbedployutil (by setting the value FREQUENDY_BY_DAYS=0 in nbdeployutilconfig.txt) the client didn't get any error or restore jobs and he's happy with it.. 

I don't know if disabling nbdeployutil is a good thing or not? could you please advise regarding this?

 

Thank you;

You can manually run nbdeployutil when you'd like to receive that information. If you want it weekly, you could set up a cron job or script that initiates at the specific time, or just make it a part of your regular work routine.

Only caveat I can think of is that disabling may make Smart Meter have issues. I can not say whether this is the case, as I have not seen that happen, but I could see it being a "gotcha" down the road if there are inconsistencies observed. 

Hi @Hamza_H 

There is no problem having the scheduled incremental nbdeployutil disabled. What your customer will "miss" out on is the ability to track license usage via Smart Meter (and also as a consequence, your Veritas account manager is unable to track this as well - not that, in my experince, many do). 
Nbdeployutil can always be run manually when desired/required/requested.

Cheers
David

Hamza_H
Moderator
Moderator
   VIP   

Hi @EthanH ,

Thanks for your reply,

I am actually aware of that, I am just wondering if disabling it could be a 'bad' thing to do or may have some impact somewhere..

You can schedule it as you want by creating the file nbdeployutilfconfig.txt and setting the frequency that suits you..

 

Hamza_H
Moderator
Moderator
   VIP   

Thanks a lot @davidmoline  for your assistance :)

I will mark one of your replies as a solution.