nbpem down issue on NBU 7.1.0.4 HP-UX 11.31

[FlyMountain]

Hi, does anyboy has suffered same nbpem down issue and knowing any solution or patches from Symatec?  It happened three time this week already. A ticket opened but i trusted this board better.

[revarooo]

Nope not come across it. What does your nbpem log show when it happens?

[Nicolai]

Do you have enough space in /usr/openv. If the free space drop below 10% EMM shutdown.

[FlyMountain]

We do have about 15% free on /usr/openv.

[mph999]

What does the nbpem log show, and if it is disk space, the emm log would show something also.

martin

[Nicolai]

vxlogview  examples:

vxlogview -p 51216 -o 111 -b "08/17/12 12:00:00 PM" -e "8/11/12 00:00:00 PM"

vxlogview -p 51216 -o 111 

Other vxlogview examples:

http://www.symantec.com/docs/TECH42794

[FlyMountain]

Couldn't find any erros from nbemm, here are captured messages during nbpem down and restart time period. 

thanks. 

08/16/12 05:13:38.568 [PemService::getOverduePriority] (ID:60000000003d3610) priority limited to max(../PemService.cpp:626)

08/16/12 05:13:38.569 V-116-215 [BaseJob::run] jobid=2054021 submitted to nbjm for processing

08/16/12 05:13:39.578 V-116-256 [CompoundJob::submitOneChildJob] policy MM_VM2, client mmdcnt0bvms003.chsli.org, schedule Full-Month

ly, stream -1 submitted for execution

08/16/12 05:13:39.579 [RecoverableJob::writeBooleans] (ID:6000000001e86330 CTX:60000000007310e0) flags=35(../RecoverableJob.cpp:102)

08/16/12 05:13:39.579 [RecoverableJob::writeBooleans] (ID:6000000093fd41d0 CTX:60000000007310e0) flags=293(../RecoverableJob.cpp:102

)

08/16/12 05:13:39.580 [PemTask::unlinkFromHandler] (ID:6000000094282ec0 CTX:60000000007310e0) Policy Cache Handler attempted to unli

nk from task but was never linked(../PemTask.cpp:728)

08/16/12 05:13:43.467 [RecoverableJob::run] (ID:6000000094282ec0 CTX:60000000007310e0) RUNID:9fffffff5ec7b6a4 Ignored action GET_CLI

ENT_POLICIES (1)(../RecoverableJob.cpp:322)

08/16/12 05:13:43.468 [RecoverableJob::run] (ID:6000000094282ec0 CTX:60000000007310e0) RUNID:9fffffff5ec7b6a4 Ignored action GET_CLI

ENT_POLICY (2)(../RecoverableJob.cpp:322)

08/16/12 05:13:43.468 [RecoverableJob::run] (ID:6000000094282ec0 CTX:60000000007310e0) RUNID:9fffffff5eb5b6a4 Ignored action GET_CLI

ENT_POLICY (2)(../RecoverableJob.cpp:322)

08/16/12 05:13:43.468 [RecoverableJob::run] (ID:6000000094282ec0 CTX:60000000007310e0) RUNID:9fffffff5eb5b6a4 Ignored action GET_CLI

ENT_POLICY (2)(../RecoverableJob.cpp:322)

08/16/12 05:13:43.468 [RecoverableJob::run] (ID:6000000094282ec0 CTX:60000000007310e0) RUNID:9fffffff5ea3b6a4 Ignored action GET_CLI

ENT_POLICY (2)(../RecoverableJob.cpp:322)

08/16/12 05:13:44.134 [RecoverableJob::registerForEmail] (ID:6000000094282ec0 CTX:60000000007310e0) Registered for email(../Recovera

bleJob.cpp:2728)

08/16/12 05:13:44.957 [RecoverableJob::writeBooleans] (ID:6000000094282ec0 CTX:60000000007310e0) flags=293(../RecoverableJob.cpp:102

)

08/16/12 05:13:44.958 [PemService::getOverduePriority] (ID:60000000003d3610) priority limited to max(../PemService.cpp:626)

08/16/12 05:13:44.958 V-116-215 [BaseJob::run] jobid=2054022 submitted to nbjm for processing

08/16/12 05:13:46.093 V-116-249 [ProcessDefinition::notify] command /usr/openv/netbackup/bin/backup_exit_notify, pid=1551, completed

 with status 0

08/16/12 06:30:37.936 [Info] V-116-272 nbpem startup, release 7.1.0.4 , host mmdcux17

08/16/12 06:30:38.255 V-116-246 [PemCORBAClient::doConnect] connection to nbevtmgr::VRTS_NBU_PEM_Channel on host mmdcux17 succeeded

after 1 attempts

08/16/12 06:30:38.792 V-116-267 [PemChangeNotifier::callReturned] connection to event services established

08/16/12 06:30:39.913 V-116-286 [NBProxy::doConnect] started nbproxy /usr/openv/var/nbproxy_pem.ior, pid=15,5

[FlyMountain]

Hello Nicolai,

Just posted the logs, can you please help take a look?

thanks,

[mph999]

Looks like it just stopped dead ...

08/16/12 05:13:44.958 V-116-215 [BaseJob::run] jobid=2054022 submitted to nbjm for processing

08/16/12 05:13:46.093 V-116-249 [ProcessDefinition::notify] command /usr/openv/netbackup/bin/backup_exit_notify, pid=1551, completed

 with status 0

08/16/12 06:30:37.936 [Info] V-116-272 nbpem startup, release 7.1.0.4 , host mmdcux17

In which case there shoud be a core file.

Make sure the system is set to save core files, then follow this TN and send the details in to support.

Output from commands in TN

Core file

nbpem binary itseld

system messages file 

http://www.symantec.com/docs/TECH52286

Martin

[watsons]

Does not look like the process shutdown by itself.. check for core dump.

Cluster environment? If so, it might be shutdown by cluster due to some fault on the nbu_group.

Check if only nbpem is the only one shutting down, what happen to other crucial processes such as nbemm, nbrb, nbjm, nbsl or nbproxy.