06-02-2013 01:04 PM
Hi Guys , need your help on this.
I have my master server configured on RedHat 6.2 installed with Netbackup 7.5.01.
strangely , for last two days , all master services go down just after midnight.
It a clustered server , with a single node cluster , as the DR node is still not active.
I saw below messages in log files on both the occasions of master server services going down.
Jun 2 00:02:53 main-nb-mast abrt[32250]: saved core dump of pid 1416 (/usr/openv/netbackup/bin/nbpem) to /var/spool/abrt/ccpp-2013-06-02-00:02:53-1416.new/coredump (54398976 bytes) Jun 2 00:02:53 main-nb-mast abrtd: Directory 'ccpp-2013-06-02-00:02:53-1416' creation detected Jun 2 00:02:53 main-nb-mast abrtd: Package 'SYMCnetbp' isn't signed with proper key Jun 2 00:02:53 main-nb-mast abrtd: Corrupted or bad dump /var/spool/abrt/ccpp-2013-06-02-00:02:53-1416 (res:2), deleting and under engine_A.log 2013/06/02 00:03:00 VCS ERROR V-16-2-13067 (main-nb-mast) Agent is calling clean for resource(nbu_server) because the resource became OFFLINE unexpectedly, on its own. 2013/06/02 00:05:29 VCS INFO V-16-2-13716 (main-nb-mast) Resource(nbu_server): Output of the completed operation (clean) I am really puzzled as I have never seen anything like this before , need your expertise on this06-02-2013 01:42 PM
nbpem is crashing
Jun 2 00:02:53 main-nb-mast abrt[32250]: saved core dump of pid 1416 (/usr/openv/netbackup/bin/nbpem) to /var/spool/abrt/ccpp-2013-06-02-00:02:53-1416.new/coredump (54398976 bytes)
Log a call, send in :
nbpem binary itself
Copy of core file
nbpem log that covers the time it crashes (raw format)
Increase log level like this
vxlogcfg -a -p 51216 -o 116 -s DebugLevel=6 -s DiagnosticLevel=6
(nbpem log will be in /usr/openv/logs/nbpem)
Decrease logging after - vxlogcfg -a -p 51216 -o 116 -s DebugLevel=1 -s DiagnosticLevel=6
os messages file that covers the time it crashes
nbsu -c -t output
Find the core file, run these commands in the TN to get the debug output, and send in output
http://clientui-kb.symantec.com/kb/index?page=content&id=TECH52289
Martin
06-02-2013 01:52 PM
http://www.symantec.com/docs/TECH164346 (did say Solaris in the "enviroment" however)
http://www.symantec.com/docs/TECH189326
Consider to patch to 7.5.0.5 since the patch NB_7.5.0.1_ET2785535_3.zip has been pulled.
06-02-2013 07:00 PM
agree with nicolai you try to install
http://www.symantec.com/business/support/index?page=content&id=TECH189326
or upgrade to netbackup 7.5.5 + latest EEB
06-02-2013 10:24 PM
Thank you so much guys for your posts.
Strangely , the error didn't occur last night.
When the services , went down on previous occasion,I rebooted the server,but when the server came up, the filesystem /opt/VRTSnbu wasn't getting mounted,I had to import the diskgroup nbudg , then bring up resouce nbudg and nbu_mount , after which all the services came up.
However , when I rebooted the server yesterday , the filesystem /opt/VRTSnbu was automatically mounted (as it should have been ideally)
Don't know where exactly to look into , but I think upgrading to NBU 7.5.0.5 will be sensible thing to do here.
@Martin - is the logging level you suggested for nbpem different from maximum logging level option available for nbpem under "Host Properties" -> "Master Server" -> Logging , please suggest.
06-02-2013 11:06 PM
Default loglevels are Debug 1 / Diag 6 - so yes, different.
Martin