Forum Discussion

mrmadej's avatar
mrmadej
Level 4
11 years ago
Solved

NetBackup unexpectedly going offline - VCS

Hello,

I have looked on many forums but can not find nothing helpfull.

I have NetBackup 7.6.0.2 on VCS with GCO on RedHat 6.

Since last upgrade I have issue with restarting the NBU by VCS. It looks like one of monitored NBU processes is not responding.

I can not find what kind of process exactly is not responding.

AGENT_DEBUG.log shows nothing and also VCS engine_A.log and NetBackup_A.log are without detailed information.

I have enabled debug logging on VCS but without expected result.

 

Can someone help me how to find that particular process?

Many thanks in advance
Madej

Backup and Recovery
Linux
NetBackup
Troubleshooting
  • mrmadej's avatar
    mrmadej
    11 years ago

    Hi,

     

    The problem is still not solved.

    But we have found the workaround which gives us time to wait for official fix in the 7.6.1 release.

    The workaround is to set the ToleranceLimit to '1' on the whole resource type NetBackup on VCS.

    And it looks well. In the engine_A.log we can find the entries like this:

    2014/09/26 04:20:33 VCS INFO V-16-2-13075 (*****) Resource(nbu_server) has reported unexpected OFFLINE 1 times, which is still within the ToleranceLimit(1).

     

    Thanks All

    Regards

    Madej

  • trv's avatar
    trv
    Level 4
    11 years ago
    Check logs in /usr/openv/netbackup/logs/cluster/ on active node. There should be something like 02:22:31.972 [27044] <16> monitor:processStatus: Some Processes are DOWN while others are UP 02:22:31.972 [27044] <16> monitor:processStatus: Following Process are found DOWN: bprd 02:22:31.972 [27044] <16> monitor:processStatus: Following Process are found UP: nbevtmgr nbstserv vmd bpdbm nbpem nbjm nbemm nbrb NB_dbsrv nbaudit nbsl nbrmms nbatd nb disco 02:22:32.854 [27060] <4> Offline::main: Offline called with 2 Parameters Yes we are seeing some issue on our NBU clusters too. It's always bprd in our case.
  • mrmadej's avatar
    mrmadej
    Level 4
    11 years ago

    Hi,

     

    Thanks !!

    I see the issue with bprd also.

    03:53:01.308 [12386] <16> monitor:processStatus: Some Processes are DOWN while others are UP
    03:53:01.308 [12386] <16> monitor:processStatus: Following Process are found DOWN: bprd
    03:53:01.308 [12386] <16> monitor:processStatus: Following Process are found UP: nbevtmgr nbstserv vmd bpdbm nbpem nbjm nbaudit nbsl nbrmms nbemm nbrb NB_dbsrv nbatd nbdisco
    03:53:02.122 [12434] <4> Offline::main: Offline called with 2 Parameters
    03:53:02.122 [12434] <4> Offline::main: Initializing NBCluster using /usr/openv/netbackup/bin/cluster/NBU_RSP file

    Have you found any solution?

  • trv's avatar
    trv
    Level 4
    11 years ago
    Nope, no solution I have found so far. We didnt log the case either as there is no core dump and nothing in standard logs. It is happening roughly once a month in our case and we can't afford to turn on verbose bpcd logging - tens of thousands of jobs daily on every master server affected. I am waiting for some bigger customer to push this issue through 1st level of support by force tbh :)
  • Nicolai's avatar
    Nicolai
    Moderator
    11 years ago

    You may have to enable core dumping. 

    http://www.unixmen.com/how-to-enable-core-dumps-in-rhel6/

    But you should still see a message in /var/log/messages if it does a segmentation fault. However if bprd quit internally you will see no cores.

     

  • Marianne's avatar
    Marianne
    Level 6
    11 years ago

    I would like to see command-line output of a manual start-up of NBU.

    So, use hares to offline the NBU resource, leaving all other resources online:

    hares -offline nbu_server -sys <live-nbu-node>

    Use bpps -x to ensure all server processes terminate.
    bpcd, vnetd and pbx must be left running.

    Create bprd log folder under /usr/openv/netbackup/logs.

    Start NetBackup: 
    /usr/openv/netbackup/bin/goodies/netbackup start 

    Check output and run 'bpps -x' in another window.
    Repeat bpps +- every 30 seconds and repeat 3-4 times.

    If bprd starts up initially and then 'dies', check bprd log for errors.
    Please copy the log file to bprd.txt and upload here as File attachment.

    Please also paste output of 'netbackup start' and bpps output into a text file (e.g. startup.txt) and upload as File attachment.
    This will tell us why bprd is terminating and if any other processes are affected.
    I have previously seen that expired eval key caused bprd to be stopped.
     

  • mrmadej's avatar
    mrmadej
    Level 4
    11 years ago

    Hi,

    The problem occurs from time to time, average once per two or three weeks. 
    I found that the cluster debug log file contains entries that the bprd is DOWN, but on the same time in bprd debug log file are still logged entries. It looks like the process is still alive.
    As trv wrote we also have thousands of jobs per day, so, we can't turn on the verbose logging. And in addition we can't stop NBU just like that, we have to schedule this activity.

    In the messages is nothing about segmentation.

    I have to look at the new 7.6.0.3 MR. Maybe the problem is solved in this release.

    Regards

  • trv's avatar
    trv
    Level 4
    11 years ago
    We have core dumping enabled of course :) But there is no segfault message in /var/log/messages or dmesg so I assume bprd exits cleanly or there is something wrong with cluster process monitoring. dang so many typos ... edit this silently
  • Marianne's avatar
    Marianne
    Level 6
    11 years ago

    I agree - no need to turn on verbose logging. 
    Level 0 logging is perfectly fine.
    bprd is one of the logs that I view as a 'must' on a master server.

    We have seen at busy environments that the monitor cycle for NBU is simply taking too long, resulting in false timeout/failure reports.

    So, check the current MonitorInterval and MonitorTimeout attributes for nbu_server resource.
    If you change MonitorTimeout to more than 60 sec (e.g. 80 or 90 sec) ensure that MonitorInterval is more or equal value.

  • mrmadej's avatar
    mrmadej
    Level 4
    11 years ago

    I think we have default settings:

    NetBackup    MonitorInterval        60
    NetBackup    MonitorTimeout         60

    Do you think that I should change this values for example:

    NetBackup    MonitorInterval        90
    NetBackup    MonitorTimeout         80

     

    Regards