cancel
Showing results for 
Search instead for 
Did you mean: 

NetBackup media server has ERRORs in NBSL log

X2
Moderator
Moderator
   VIP   

I was checking the logs on the media server (Win 2008R2 SP2 x64, NBU 7.6.1.1) and the NBSL logs have errors in 50% of the lines. Has anyone see this?

The log repeats the following lines about 41,000 times:

0,51216,132,132,1252824,1429106095953,6216,4652,0:,88:ERROR: Callback is down. Will not deliver events, Caller ID:0(EventSupplierTask.cpp:188),31:EventSupplierTask::shallEnqueue,1
0,51216,132,132,1252825,1429106095953,6216,4652,0:,63:Could not able to enqueue NBSL event(EventSupplierTask.cpp:268),26:EventSupplierTask::enqueue,1

Backups are running fine. No failed backups except the ones which are due to non-NBU issues.

Any information about this will be helpful.

Thanks in advan.ce

13 REPLIES 13

sdo
Moderator
Moderator
Partner    VIP    Certified

I can't tell you what they're about... but your count of 41000 messages does seem to be indicative of a problem somewhere at somepoint.

It's not obvious from the raw logs what date they're from.  They could be old, and no longer relevant.

What are the message counts for the previous hour?

> vxlogview -p 51216 -o 132 -t 01:00:00 | find /c "Callback is down"

.

FYI - I've just checked two new v2.6.1.1 master/media appliances which are not doing very much at all at the moment, except backing up a few clients and replicating, and I found:

27 messages re "Callback is down"

27 messages re "Could not able"

...on one appliance.  And 22 occurences of each message on the other appliance...

...all of which are from 10 days ago, which I assume is probably from when I rebooted the appliances, or bounced the application.

X2
Moderator
Moderator
   VIP   

The snippet I provided earlier was from yesterday's log file.

Here is the output of vxlogview as requested:

C:\Program Files\Veritas\NetBackup\bin>vxlogview.cmd -p 51216 -o 132 -t 01:00:00
 | find /c "Callback is down"
3574

I will check on my second media server where the replication happens for the same issue.

X2
Moderator
Moderator
   VIP   

There is only one file on the second media server under nbsl log directory


C:\Program Files\Veritas\NetBackup\bin>vxlogview.cmd -p 51216 -o 132 -t 01:00:00 | find /c "Callback is down"
V-1-45 No log files found.
0

 

sdo
Moderator
Moderator
Partner    VIP    Certified

Ok - so you know you're getting c. 3574 alerts per hour (which looks like one alert per second) on one media server.

I can't offer any deep diagnosis.  All I can think of is to perhaps check each master and media server... to see whether "nbsl" is running?

Windows:    > bpps | find "nbsl"

Unix:           # bpps | grep "nbsl"

Anyone else know what this alert means?

mph999
Level 6
Employee Accredited

Can't find much on this that gives a clear answer.

Looks like a previous eTrack , but this was back in 2012 ...

"looks like its rare scenario when due to the network issue the catalog update has failed due
to a TRANSIANT exception on OpsC callback object"

However, it seems the OsCenter Code was then changed to be more accomdating of this - clearly we can't fix  a network issue in the code, but we can try and work around it a bit.

That said, there was a memory leak reported on nbsl / 7.1.0.4 but this was on Aix - that is not to say it couldn't happen on Windows - the symptoms were the same messages in the log as you report.

So in summary, not enough is known without further investigation - are you on a very old version, are you seeing high memory usage etc ...  If it's a similar issue to the etrack, perhaps you have no obvious 'physical' network issues, but how is the TCP tuning at the OS level.  If you reboot the machine, do the errors then disappear before returning after an amount of time.

X2
Moderator
Moderator
   VIP   

My environment has NetBackup 7.6.1.1 (with the media server in question being Win2k8 SP1 x64). Memory usage seems to be normal, hanging around 34 GB out of 48 GB with spoold taking around 28 GB.

I'm going to reboot the machine on Monday, will keep an eye out for the errors after reboot.

X2
Moderator
Moderator
   VIP   

My apologies for not updating earlier. The server was rebooted a week ago after applying Windows updates. NBSL logs have not shown any difference though.
 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Probably time to log a Support call with Symantec if this is still bugging you....

X2
Moderator
Moderator
   VIP   

That is what I was planning on but there are other issues which came up and I have 3 cases already open. Will wait a couple of weeks to get the 3 sorted out and then concentrate on this one.

I wonder if this thread can remain open for a few more weeks so that I can update it after opening the case.

mph999
Level 6
Employee Accredited

Yes, np leaving the thread, just come back to it when you are ready.

watsons
Level 6

Some other points to think about:

1) Is the current 132 (nbsl) log level at default value?  If not, are these messages still appearing even if you set the "DebugLevel=1" & "DiagnosticLevel=6"?

2) What is your master server's NBU version? Same as this media server at 7.6.1.1? If it's not, try to make it the same. (note that I believe 7.6.1 master server can still work with 7.6.1.1 media server, is that correct?)

3) If your environment has a OpsCenter server monitoring the master server, try to disable its data collection maybe for a short period, and check if you still get these NBSL messages?  OpsCenter data collection is known to be gathering info via nbsl processes.

Ed3
Level 3

I am having a similar issue. Seeing this across all my 7.6.1.1 and 7.5.0.6 media servers (Windows and AIX). New installs and upgrades.Logging levels all set to minumum. Master Server is 7.6.1.1 recently upgraded from 7.5.0.6, upgraded before any other client/media server. No OpsCenter in this environment.

 

I have opened a case with Symantec. Will try to report back what they find.

Ed3
Level 3

Mine is specific to the fact I do not have OpsCenter in my environemnt. It is a known issue. Currently there is no fix. Workaround is to set DebugLevel on NBSL to 0 (default = 1):

vxlogcfg -a -p 51216 -o 132 -s DebugLevel=0

According to my SE NBSL's sole purpose is to report to OpsCenter. If you are not using it in your environment, no need to pay attention to nbsl logs.

If there is an OpsCenter in the environment, then further investigation may be needed.