None of the jobs appear in Activity monitor

[Siddesh]

Hi All,

There was a weired issue yesterday, During 18:30hrs to 20:00hrs we have nearly 30 jobs running (both scheduled and user based backups); but none of the jobs were reflected in the activity monitor.

There was only a single restoration job which was running and had been triggered @ 17:35.

When I noticed this I tried to trigger 3 to 4 a manual backup but that too dint appear in the activity monitor.

Checked the following:

/var/core but no core dump was generated.

All Netbackup master services were in running.

Used nbrbutil –dump

It showed nearly 18 Allocations (AllocationSeq from index=0 … index=13)

And there was 1 entry in MDS Allocation that was of the restoration job.

After a while, our team decided to re-cycle the Netbackup services on Master server.

But to our surprise at 20:04 all, the jobs that were triggered during the past few hours showed up in the activity monitor and were in writing state without any services being restarted.

All the jobs showed 20:04 in start time.

Later we noticed no such resource allocation issue and all the policies were running as per schedule.

I was unable to isolate the issue; could anyone suggest as to what logs can be checked for the issue.

Master Server: SUN Solaris 10

Tape Libraries: L700 &  L180 having 10 drives each

SAN Media Servers: Nearly 80(all with different OS)

Netbackup version : 6.5.4 

Regards,

Siddesh

[Andy_Welburn]

e.g. Problems, All Log Entries for that period?

Anything at all of any relevance in any of the jobs "Job Details"? Altho' if they all appeared to start at 20:04 there probably won't be.

[MOHAMED_PATEL]

Had an issue like this 2 weeks ago - But this was for NB 7.0

Had to log a call with Symantec and was supplied an EEB -

http://www.symantec.com/docs/TECH129641
 

If you go through the logs, have a look at the job schedule and check if they had begun but were than rescheduled to the next run time/date.

Sent a whole lot of other logs to Symantec as well.......

Now fixed.

[Marianne · Accepted solution]

I found this fix in the NBU 656 Release Update on p.74:

■ Description:
Communication delays with bpjobd caused a long delay before nbjm would send a resource allocation request for a job to nbrb. During this period, nbjm did not send any jobs to nbrb for evaluation. A change was made to address this issue.

p.76:
■ Description:
The NetBackup Resource Broker (NBRB) appeared to go into an infinite loop and hang if the RB_RESPECT_REQUEST_PRIORITY parameter is set to one in rb.conf file.

NetBackup 6.5.6 can be obtained at the following location:
  http://www.symantec.com/docs/TECH129076

NetBackup 6.5.6 Documentation Updates can be found here:
  http://www.symantec.com/docs/TECH128244

[Zahid_Haseeb]

As marianne said there is a fix. What i think that this may be the issue with GUI. did you try to see the Activity Monitor from command prompt ?

Run the command ""bpdbjobs.exe -report"" under admincmd. 

[Siddesh]

Apologies, for the delay.

No at that particular moment I did not excute " bpdbjobs.exe -report".

But I dont suspect that GUI had an issue; coz as mentioned earlier the nbrbutil -dump would provide me with the MDS allocation ID's which it dint provide.

I agree with Marriane, there may be communication delays.
Symante TSE's also have suggested to upgrade the Master server to 6.5.6 as alot of resource allocation issue have been faced in 6.5.4. 

Regards,

Siddesh

[Zahid_Haseeb]

SO why not you are just trying to run the command " bpdbjobs.exe -report".