OpsCenter 7.5.0.3 stopped collecting data from NBU 7.1.0.4

[effiko]

We have a customer that uses the OpsCenter to monitor the backups and send a daily report.

For no known reason the OpsCenter stopped collecting data from NBU and the daily report came out empty.

I tried restarting the  OpsCenter services by the command :

INSTALL_PATH\OpsCenter\server\bin\opsadmin.bat   stop

and then

INSTALL_PATH\OpsCenter\server\bin\opsadmin.bat   start

It didn't help and I had to restart the services from the system panel.

Questions:

1. Is this a known issue? (we had it in another site monitoring the 5220)

2. is there a fix for that?

3. Can I generate an alert going out from OpsCenter when data colection stops?

[tom_sprouse · Accepted solution]

effiko,

You shouldn't have to cycle the services to get data collection running again.

You can disable / enable data collection for one or more masters from the Settings / Configuration / NetBackup tabs within OpsCenter.

You can also click on the Data Collection Left Navigation Item on the Home Page after login for OpsCenter.

If you see a master reporting offline, partially connected, etc... click on the name of the master.

Then at the bottom of the screen, locate the Data Collection Tab, and click on it.

You will see all Data Types listed for collection for the master you have clicked on previously.

Scroll though the items, and see if there are any failures reported.

If so, please reply with the failure message, and the Data Type the failure is occurring on.

-----

Also, you may want to check the host properties of the master, under the server section.  You may find, an admin may have removed the OpsCenter hostname from the list of additional servers, or if the master is 7.5.x.x there is now an OpsCenter Tab.

----

Is your 5220 setup as a master or media server?

----

And you can setup alerting on your OpsCenter host for when it looses connectivity to the master, essentially interupting data collection.

Please see page, 487 - http://www.symantec.com/docs/DOC5042 for additional information on Managing - Alert Policies.

If you need step by step instructions, please post a new thread for this.

----

If you have any quesitons, please let me know.

--Tom

[Rick_Brown]

Been using OpsCenter for a long time now to monitor 7 NBU masters and it does - out of the blue, stop collecting data from the NBU master servers for no apparent reason. I have 2 choices when this happens: 1. Stop & restart OpsCenter services. 2. Wait for the slow login process to OpsCenter and cycle the data collectores on ALL NBU masters and wait for it to catch back up. Without exception, at some point over a weekend, at least 1 of the 7 NBU masters will stop having its data collected by OpsCenter and cascade downhill from there. This product requires manual maintence everyday. VBR 6.6 on the otherhand worked perfectly day after day without any manual intervention. OpsCenter has never lived up to its marketing hype.

[RonCaplinger]

I've been noticing the same thing with 7.5.0.4.  Data collection seems to fail every other week or so.  I have only been on this version for a few months and haven't been able to track the cause yet, since it is sporadic.

[DR_W]

Hello friends,

I wrestled data collection issues with NetBackup 7.1.0.4 (AIX) and OpsCenter Analytics 7.5 - 7.5.0.4 for months.  I was able to trace mine to "[OpsCenter database exception]: OpsCenterDb Column value issue around:domain_Job".  Here are the steps I took (using Windows):

1.  Check to see if you are experiencing the issue.  If you aren't, you can obviously ignore the rest of the post - From a command prompt in the server logs folder - C:\Program Files\Symantec\OpsCenter\server\logs>findstr /s /i /m /c:"Database Exception" *.log (you can also output this to a file by tacking on > filename.txt).

2.  If you are experiencing the issue, you should see the names of the log files for the dates it occurred.  This will confirm you are experiencing the issue.

3.  Follow the steps in this tech note: http://www.symantec.com/business/support/index?page=content&id=TECH187938 (you can ignore that it is listed as being for Exchange backups).

4.  You will need to establish the ODBC connection to vxpmdb on your OpsCenter server to use the dbisqlc tool.  Please note that even if you already setup the ODBC connection in Windows, you MUST still set it up from within the dbisqlc tool, and it is slightly different.  It isn't listed in the tech note from step 3, but you can use this link to get you through it: http://www.symantec.com/business/support/index?page=content&id=TECH141827

Hopefully this helps some of you.

[sgt_why]

We ran into this issue recently ... OpsCenter and Master servers running 7.5.0.1 versions on a mixture of Linux and Solaris hosts.

Just a few weeks after I setup daily SQL reports, to be used by tier1 - as list of what to troubleshoot for the day.

It sent out a couple empty reports ... all weekend and when I came in Monday, I noticed the (2) master servers were marked as partially connected.

Then, (after some digging) I realized that with 7.5 the opscenter clients are no longer installed onto the master servers -- instead, its all "built in".

So, no agents to bounce ... instead, I had to bounce the ops center server daemons (/etc/init.d/OpsCenterServer stop / start)

That got it working again and 5m later ... I was able to generate valid reports.

The next time around, I opened a support case ... and rather than bouncing the daemons, I logged into OpsCenter GUI, went to manage and clicked the tab on bottom left for "netbackup data collection".

Then selected the master server (by clicking directly on its underlined name) ... which opens a new page, then clicked on the "Data Collection Status". From there, it shows you all of the collectors ... and guess what?

It had failed (and hung) on the "cloud" collector. But as I watched, it re-polled all of the collectors and this time everything came back as "completed" with exception of cloud which was "running".

So I opened a case with symantec support ... they said there is a known bug, where collections often hang up if no cloud collectors are configured. And they suggested a EEB fix.

Sounded good ... except that they circled back. The "fix" is actually for 7.5.0.3 and as included in 7.5.0.5

So their suggestion was to upgrade my entire NB infrastructure to 7.5.0.5 and the upgrade OpsCenter to 7.5.0.5 as well.

But we are currently in a "production freeze" which means no changes to production.

I asked if I could just upgrade OpsCenter -- but was told NO, the changes must happen on the MASTER servers to fix this issue.

So --- for the time being, I just cron'd up a daily bounce of the OpsCenter server daemons ... 30m prior to when the reports are generated. And it seems to be doing the trick.

[effiko]

I got the system stable now for a long time. So I'll close this loop.