cancel
Showing results for 
Search instead for 
Did you mean: 

Master Server services go down after 12:00 AM (midnight)

shashi_pratap
Level 3
Accredited

Hi Guys , need your help on this.

I have my master server configured on RedHat 6.2 installed with Netbackup 7.5.01.

strangely , for last two days , all master services go down just after midnight.

It a clustered server , with a single node cluster , as the DR node is still not active.

I saw below messages in log files on both the occasions of master server services going down.

Jun 2 00:02:53 main-nb-mast abrt[32250]: saved core dump of pid 1416 (/usr/openv/netbackup/bin/nbpem) to /var/spool/abrt/ccpp-2013-06-02-00:02:53-1416.new/coredump (54398976 bytes) Jun 2 00:02:53 main-nb-mast abrtd: Directory 'ccpp-2013-06-02-00:02:53-1416' creation detected Jun 2 00:02:53 main-nb-mast abrtd: Package 'SYMCnetbp' isn't signed with proper key Jun 2 00:02:53 main-nb-mast abrtd: Corrupted or bad dump /var/spool/abrt/ccpp-2013-06-02-00:02:53-1416 (res:2), deleting and under engine_A.log 2013/06/02 00:03:00 VCS ERROR V-16-2-13067 (main-nb-mast) Agent is calling clean for resource(nbu_server) because the resource became OFFLINE unexpectedly, on its own. 2013/06/02 00:05:29 VCS INFO V-16-2-13716 (main-nb-mast) Resource(nbu_server): Output of the completed operation (clean) I am really puzzled as I have never seen anything like this before , need your expertise on this
5 REPLIES 5

mph999
Level 6
Employee Accredited

nbpem is crashing

Jun 2 00:02:53 main-nb-mast abrt[32250]: saved core dump of pid 1416 (/usr/openv/netbackup/bin/nbpem) to /var/spool/abrt/ccpp-2013-06-02-00:02:53-1416.new/coredump (54398976 bytes)

Log a call, send in :

nbpem binary itself

Copy of core file

nbpem log that covers the time it crashes (raw format)

Increase log level like this

vxlogcfg -a -p 51216 -o 116 -s DebugLevel=6 -s DiagnosticLevel=6

(nbpem log will be in /usr/openv/logs/nbpem)

Decrease logging after - vxlogcfg -a -p 51216 -o 116 -s DebugLevel=1 -s DiagnosticLevel=6

os messages file that covers the time it crashes

nbsu -c -t output

Find the core file, run these commands in the TN to get the debug output,  and send in output

http://clientui-kb.symantec.com/kb/index?page=content&id=TECH52289

 

Martin

Nicolai
Moderator
Moderator
Partner    VIP   

http://www.symantec.com/docs/TECH164346 (did say Solaris in the "enviroment" however)

http://www.symantec.com/docs/TECH189326

Consider to patch to 7.5.0.5 since the patch NB_7.5.0.1_ET2785535_3.zip has been pulled.

huanglao2002
Level 6

agree with nicolai you try to install 

http://www.symantec.com/business/support/index?page=content&id=TECH189326

 

or upgrade to netbackup 7.5.5 + latest EEB

shashi_pratap
Level 3
Accredited

Thank you so much guys for your posts.

Strangely , the error didn't occur last night.

When the services , went down on previous occasion,I rebooted the server,but when the server came up, the filesystem /opt/VRTSnbu wasn't getting mounted,I had to import the diskgroup nbudg , then bring up resouce nbudg and nbu_mount , after which all the services came up.

However , when I rebooted the server yesterday , the filesystem /opt/VRTSnbu was automatically mounted (as it should have been ideally)

Don't know where exactly to look into , but I think upgrading to NBU 7.5.0.5 will be sensible thing to do here.

@Martin - is the logging level you suggested for nbpem different from maximum logging level  option available for nbpem under "Host Properties" -> "Master Server" -> Logging , please suggest.

mph999
Level 6
Employee Accredited

Default loglevels are Debug 1 / Diag 6 - so yes, different.

Martin