Netbackup services stopping randomly - HELP!

[jagged]

we have had a lot of issues with our backups failing due to various Netbackup services randomly stopping over the last few months. We're currently running Netbackup 7.0.1 with one master/media server, Sepaton VTL and Adic Scalar 24 robot.

The windows event log doesnt show any 'errors' but show various services stopping during the backups. The fatal issue seems to be when Enterprise Media Manager service stops - at which point all Netbackup services are forcefully stopped until a user logs back in to the Admin console (at which point a box pops up with an option to continue, cancel or repair NB services).

I've taken a look at the EMM logs but cannot fathom what is actually taking the services down (ie. the root cause)

any help/suggestions appreciated

here's the part of the EMM log when the services stopped:

3/9/2011 20:58:48.971 V-111-1061 [DeviceAllocatorImpl::updateMachineState] Client signature HOST=<tpr-bak01> VER=<700000> APP=<ltid> PID=<2724>
3/9/2011 20:58:48.987 [DeviceAllocatorImpl::updateMachineState] retval - <0>
3/9/2011 20:59:10.472 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
3/9/2011 20:59:10.472 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
3/9/2011 20:59:10.472 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
3/9/2011 20:59:10.472 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
3/9/2011 20:59:10.472 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
3/9/2011 20:59:10.472 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
3/9/2011 20:59:10.488 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
3/9/2011 20:59:10.488 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
3/9/2011 20:59:10.488 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
3/9/2011 20:59:10.488 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
3/9/2011 20:59:46.910 [FATClientORBConfig::shutServant] Calling fini() on FAT Client
3/9/2011 20:59:46.910 [FATClientORBConfig::shutServant] Done fini() on FAT Client
3/9/2011 20:59:46.910 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBFSMCLIENT>
3/9/2011 20:59:46.910 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBFSMCLIENT>
3/9/2011 20:59:46.910 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBFSMCLIENT>
3/9/2011 20:59:46.910 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBFSMCLIENT>
3/9/2011 20:59:46.910 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBFSMCLIENT>
3/9/2011 20:59:46.910 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBFSMCLIENT>
3/9/2011 20:59:46.910 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBFSMCLIENT>
3/9/2011 20:59:46.910 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBFSMCLIENT>
3/9/2011 20:59:46.910 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBFSMCLIENT>
3/9/2011 20:59:46.910 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBFSMCLIENT>
3/9/2011 20:59:46.910 [REMORBConfig::shutServant] Calling fini() on REM
3/9/2011 20:59:46.988 [REMORBConfig::shutServant] Done fini() on REM
3/9/2011 20:59:46.988 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBREM>
3/9/2011 20:59:46.988 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBREM>
3/9/2011 20:59:46.988 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBREM>
3/9/2011 20:59:46.988 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBREM>
3/9/2011 20:59:46.988 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBREM>
3/9/2011 20:59:46.988 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBREM>
3/9/2011 20:59:46.988 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBREM>
3/9/2011 20:59:46.988 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBREM>
3/9/2011 20:59:46.988 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBREM>
3/9/2011 20:59:46.988 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBREM>
3/9/2011 20:59:46.988 [DSMFSMORBConfig::shutServants] Calling fini() on FSM
3/9/2011 20:59:47.067 [DSMFSMORBConfig::shutServants] Done fini() on FSM
3/9/2011 20:59:47.067 [DSMFSMORBConfig::shutServants] Calling fini() on DSM
3/9/2011 20:59:48.067 [DSMFSMORBConfig::shutServants] Done fini() on DSM
3/9/2011 20:59:48.067 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBDSMFSM>
3/9/2011 20:59:48.067 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBDSMFSM>
3/9/2011 20:59:48.067 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBDSMFSM>
3/9/2011 20:59:48.067 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBDSMFSM>
3/9/2011 20:59:48.067 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBDSMFSM>
3/9/2011 20:59:48.067 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBDSMFSM>
3/9/2011 20:59:48.067 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBDSMFSM>
3/9/2011 20:59:48.067 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBDSMFSM>
3/9/2011 20:59:48.067 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBDSMFSM>
3/9/2011 20:59:48.067 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBDSMFSM>
3/9/2011 20:59:48.067 [EMMServer::fini] EMM Server shutting down
3/9/2011 20:59:48.067 [Info] V-111-1003 [EMMServer::fini] EMMServer shutting down
3/9/2011 20:59:48.067 [EMMServer::fini] No more CORBA calls in progress
3/9/2011 20:59:48.067 [ServerImpl::DeactivateMap] 
3/9/2011 20:59:48.067 [ServerImpl::DeactivateMap] Deactivating factory for Media
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] 
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] deactivating servant  version 2
3/9/2011 20:59:48.067 [MediaImpl_2::~MediaImpl_2()]  Exiting
3/9/2011 20:59:48.067 [MediaImpl::~MediaImpl()]  Exiting
3/9/2011 20:59:48.067 [ServerImpl::DeactivateMap] Deactivating factory for MediaPool
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] 
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] deactivating servant  version 2
3/9/2011 20:59:48.067 [MediaPoolImpl_2::~MediaPoolImpl_2()]  Exiting
3/9/2011 20:59:48.067 [MediaPoolImpl::~MediaPoolImpl()]  Exiting
3/9/2011 20:59:48.067 [ServerImpl::DeactivateMap] Deactivating factory for MdsServer
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] 
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] deactivating servant  version 1
3/9/2011 20:59:48.067 [MdsServerImpl::~MdsServerImpl()]  Exiting
3/9/2011 20:59:48.067 [ServerImpl::DeactivateMap] Deactivating factory for ExtMappings
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] 
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] deactivating servant  version 1
3/9/2011 20:59:48.067 [ExtMappingsImpl::~ExtMappingsImpl()]  Exiting
3/9/2011 20:59:48.067 [ServerImpl::DeactivateMap] Deactivating factory for Machine
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] 
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] deactivating servant  version 1
3/9/2011 20:59:48.067 [MachineImpl::~MachineImpl()]  Exiting
3/9/2011 20:59:48.067 [ServerImpl::DeactivateMap] Deactivating factory for DeviceAllocator
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] 
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] deactivating servant  version 1
3/9/2011 20:59:48.067 [DeviceAllocatorImpl::~DeviceAllocatorImpl]  Exiting
3/9/2011 20:59:48.067 [ServerImpl::DeactivateMap] Deactivating factory for BarcodeRule
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] 
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] deactivating servant  version 1
3/9/2011 20:59:48.067 [BarcodeRuleImpl::~BarcodeRuleImpl()]  Exiting
3/9/2011 20:59:48.067 [ServerImpl::DeactivateMap] Deactivating factory for Ndmp
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] 
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] deactivating servant  version 1
3/9/2011 20:59:48.067 [NdmpImpl::~NdmpImpl()]  Exiting
3/9/2011 20:59:48.067 [ServerImpl::DeactivateMap] Deactivating factory for DeviceConfig
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] 
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] deactivating servant  version 1
3/9/2011 20:59:48.067 [DeviceConfigImpl::~DeviceConfigImpl() ]  Exiting
3/9/2011 20:59:48.067 [ServerImpl::DeactivateMap] Deactivating factory for Drive
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] 
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] deactivating servant  version 1
3/9/2011 20:59:48.067 [DriveImpl::~DriveImpl()]  Exiting
3/9/2011 20:59:48.067 [ServerImpl::DeactivateMap] Deactivating factory for Library
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] 
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] deactivating servant  version 1
3/9/2011 20:59:48.067 [LibraryImpl::~LibraryImpl()]  Exiting
3/9/2011 20:59:48.067 [ServerImpl::DeactivateMap] Deactivating factory for DriveSeed
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] 
3/9/2011 20:59:48.067 [ServerImpl::DeactivateMap] Deactivating factory for StorageUnit
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] 
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] deactivating servant  version 2
3/9/2011 20:59:48.067 [StorageUnitImpl_2::~StorageUnitImpl_2()]  Exiting
3/9/2011 20:59:48.067 [StorageUnitImpl::~StorageUnitImpl()]  Exiting
3/9/2011 20:59:48.067 [ServerImpl::DeactivateMap] Deactivating factory for MachineConfig
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] 
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] deactivating servant  version 1
3/9/2011 20:59:48.067 [MachineConfigImpl::~MachineConfigImpl()]  Exiting
3/9/2011 20:59:48.067 [ServerImpl::DeactivateMap] Deactivating factory for AllocationStatus
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] 
3/9/2011 20:59:48.067 [ServerImpl::DeactivateMap] Deactivating factory for ServerCtrl
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] 
3/9/2011 20:59:48.067 [ServerImpl::DeactivateMap] Deactivating factory for ServerGroup
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] 
3/9/2011 20:59:48.067 [ServerImpl::DeactivateMap] Deactivating factory for ImageCatalog
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] 
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] deactivating servant  version 1
3/9/2011 20:59:48.067 [ImageCatalogImpl::~ImageCatalogImpl()] Exiting
3/9/2011 20:59:48.067 [ServerImpl::DeactivateMap] Deactivating factory for Version
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] 
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] deactivating servant  version 1
3/9/2011 20:59:48.067 [VersionImpl::~VersionImpl()]  Exiting
3/9/2011 20:59:48.067 [ServerImpl::DeactivateMap] Deactivating factory for AppCredentials
3/9/2011 20:59:48.067 [VersionFactory::CleanAllForShutdown] 
3/9/2011 20:59:48.067 [EMMServer::fini] Destroyed servant m_pServerImpl
3/9/2011 20:59:48.067 [Info] V-111-1004 [EMMServer::fini] terminating DA thread pool
3/9/2011 20:59:48.067 [EMMServer::fini] Terminating DA thread pool
3/9/2011 20:59:48.317 [EMMServer::fini] All DA svc threads terminated <0>
3/9/2011 20:59:48.317 [EMMServer::fini] DA thread pool terminated
3/9/2011 20:59:48.317 [Info] V-111-1059 [EMMServer::fini] terminating notify thread pool
3/9/2011 20:59:48.317 [EMMServer::fini] Terminating Notify thread pool
3/9/2011 20:59:48.317 [Notify_Thread_Pool::exit] Signalling threads to exit
3/9/2011 20:59:48.567 [Notify_Thread_Pool::exit] Deleting handle to Event Manager on host tpr-bak01
3/9/2011 20:59:48.567 [EMMServer::fini] All Notify svc threads terminated <0>
3/9/2011 20:59:48.567 [EMMServer::fini] Notify thread pool terminated
3/9/2011 20:59:48.567 [EMMServer::fini] Destroyed m_FetchPOA
3/9/2011 20:59:48.567 [ServerImpl::~ServerImpl()]  Exiting
3/9/2011 20:59:48.567 [EMMServer::fini] Destroyed m_POA
3/9/2011 20:59:48.832 [EMMServer::fini] Deactivated DB connections
3/9/2011 20:59:48.832 [EMMServer::fini] EMM Server shut down complete
3/9/2011 20:59:48.832 [EMMMain] <EXIT>
3/9/2011 20:59:48.832 [Log::deinit] 
3/9/2011 20:59:48.832 [EMMServer::~EMMServer] 

[j083418 · Accepted solution]

Are you running any Anti-Virus software? If so what is it?

I've seen Mcafee randomly stop services before.

[dharmz]

Please check your license details.

Check whether the license has expired.

Warm Regards.

[jagged]

j083418 : we are running McAfee VirusScan Enterprise 8.7.0i - but with 'on access scanning' disabled. However, I've uninstalled this (+agent) now and will see if this makes any difference.

dharmz: I've checked and all licences are ok.

...i'll keep you posted....

[Panz05]

Have a look at http://www.symantec.com/business/support/index?page=content&id=TECH56658

[Marianne]

Agree with TN posted above - I've experienced this problem at a customer a while back.

Also check for disk full condition. EMM will shut down automatically to prevent database corruption.

[jagged]

Hi All,

just to let you know that the issue seem to have completely disappeared since I uninstalled McAfee.

What a surprise eh?

Thanks for your suggestions and help!