getting Info nbjm (pid=xxxx) Waiting in NetBackup scheduler work queue on server has been updated.

Question

Hi,

There is serious issue i am facing in my environment with respect to RMAN oracle backup
	newly generated streems remain in queue for long time though netbackup resources are available

01/07/2015 23:19:45 - requesting resource nbu_master.NBU_POLICY.MAXJOBS.RMAN_FCRM10DB_ARCHIVE
	&lt;= skipped =&gt;
	01/07/2015 23:45:45 - Info nbjm (pid=13731) Waiting in NetBackup scheduler work queue on server xxxxxxxx

it waited approximatly 25 min if a backup with 50 streems and each streem wait for such long time resulted into large backup window please suggest&nbsp;

i found below tech note but can not co relate to this
	http://www.symantec.com/business/support/index?pag...
	http://www.symantec.com/business/support/index?pag...
	&nbsp;

Master server - Solaris 10 / netbackup 7.5.0.6

Total RMAN policy - 800

Thanks&nbsp;

marianne · Answer

There are many reasons why jobs are queued, such as:

MPX settings for STU or policy was reached
	All drives are busy
	Available media in use
	Max jobs per client was reached
	Max jobs per policy was reached
	etc...

Above log extract seems as if Max Jobs per Policy was reached.

Best to have a look at queued jobs in Activity Monitor to see reason for jobs being queued.

Another possibility may be the Resource Broker on the master server that is battling to keep up with amount of jobs.

Have a look at these performance tuning docs:

NetBackup 7.6 Best Practices: Optimizing Performance

Updated NetBackup Backup Planning and Performance Tuning Guide for Release 7.5 and Release 7.6

mph999 · Answer

Persoanlly I would get the following:

Persoanlly I would ask for mds, nbpem, jm, rb and emm at Debug/ Diag 6 and 6 in raw format. &nbsp;These should cover a restart and left until the system slows and the problem is seen.

In fact, nbrb should only be at debug 4 (hope I've remembered that right) as any higher could cause issues itself.

So, 111, 116, 117, 118 + 143

143 logs into 111, so only 4 to collect.

vxlogcfg-a -p 51216 -o 143&nbsp;-s DebugLevel=6 -s DiagnosticLevel=6

vxlogcfg-a -p 51216 -o 111 -s DebugLevel=6 -s DiagnosticLevel=6

vxlogcfg&nbsp;-a -p 51216 -o 116 -s DebugLevel=6 -s DiagnosticLevel=6&nbsp;

vxlogcfg&nbsp;-a -p 51216 -o 117 -s DebugLevel=6 -s DiagnosticLevel=6

vxlogcfg&nbsp;-a -p 51216 -o 118 -s DebugLevel=4 -s DiagnosticLevel=6

Leave the system running long enough that a jobs starts and the delay is seen after the logs have been enabled.

Collect logs nbjm, nbpem, nbrb, nbemm

nbrb log would be the place to start, need to run the raw log through vxlogview with -d all option,.

If you look for lines containing

[ResBroker_i::doEvaluation]

This should show how long the evaluation cycle is taking.

Use same commands but DebugLevel=1 to set back to defaults.

Numerous possible causes, but above may give more clues and if nothing else will give an insight as to other potential performance issues.

Is there any nbrb/ nbemm tuning applied ?

I'd also stop services and see if there are any 'stale' files in /usr/openv/netbackup/db/media/drives and /usr/openv/netbackup/db/media/tpreq &nbsp;(these files would be in the media server)

nicolai · Answer

MPH999 is on a clue

If EMM has to little memory &nbsp;to play with, it can cause&nbsp;ridiculous long resource handling in general. You might consider to set the EMM memory configuration to large

Run /usr/openv/db/bin/dbadm

Select

2)&nbsp;Database Space and Memory Management

4) Adjust Memory Settings,

3) Large.

Re-start of Netbackup required.

That said - Marianne also has a very valid point also - but we can't with the given information available right now tell wrong. Consider also to increase the policy priority 99999, this will let the Oracle job come first in queue.

raaavan · Answer

Dear All
	
	Apologies for the delay in reply&nbsp;

Tech support suggested me to remove&nbsp; offline server list from EMM &amp; also unwanted entry from bp.conf&nbsp; [ a proper decommission ( 18 was the count at that time)&nbsp; ] since NBRB while scanning for the resource request it tries to contact each entry in bp.conf and it takes significant amount of time there.
	
	I did that though issue is not totally resolved but there is improvement 
	
	Still I have 10 offline servers ,On other part some team doing POC&nbsp; &amp; servers are not gracefully added &amp; removed in EMM
	I trying to get count below 5-4 but there are some application dependacy like switch back switch overs , LDOM server etc.&nbsp; so it become task to achive&nbsp; 
	
	In parallel i am looking in to Oracle backup tuning since this environment having 723 Oracle Type &amp; 584 MS-SQL-server Type polices which are quit a large in number
	There are some flaws from DBA end since RMAN &amp; netbackup repository maintenance is not done ( CROSSCHECK &amp; Obsolete backup pieces build up over the period ) also "%t" factor in rman FORMAT templet ( NBU 7.0 )
	
	also observed that minimum patch level to run netbackup 7.5.0.6 on solaris box was missing very soon we will patch master with latest patch level&nbsp;
	
	I can't see any thumb rule to get rid out of this
	
	@Nicola - Checked enough memory is there
	
	@MPH999 - i have not collected this logs i'll check &amp; update on this
	
	@Marianne - I have crosschecked all setting suggested by you &amp; they are appropriate

raaavan · Answer

Issue started reoccurring at this moment i don't have offline media server in EMM but still allocation is getting delay.
	
	I have 300 + RMAN backup runs in production hours. i have observed that though DBA are using RMAN template comes with NBU
	
	they having additional statement like
	
	DELETE NOPROMPT ARCHIVELOG ALL BACKED UP 2 TIMES TO 'SBT_TAPE' COMPLETED BEFORE 'SYSDATE -2';
	
	and
	
	and other maintenance task command like crosscheck , etc ..
	
	I suspect this request must be hitting EMM &amp; making it busy which might be causing allocation delay but not sure so need help like what comes in bprd or bpdbm logs when any maintains statement is&nbsp; run by RMAN

Forum Discussion

getting Info nbjm (pid=xxxx) Waiting in NetBackup scheduler work queue on server <master server> has been updated.

Related Content

Re: getting Info nbjm (pid=xxxx) Waiting in NetBackup scheduler work queue on server <master serv

Re: Bare Metal Restore - boot server configuration

Re: Cannot Start NetBackUp Administration Console

Resilient network - client -> server is not working

Multiple [tldd] <defunct> process on Media server

Recent Discussions

Instant Access for vmware problem

Netbackup Catalog backup growing

Need an advise on Backup strategy for long term retention

Did a catalogue restore, but seem to have missing information.

ost datadomain download file