nbpem at 100% - 6.5.3.1

I would start by pulling out the nbpem data from the unified logs:

vxlogview -o 116 -b "MM/DD/YY hh:mm:ss AM/PM -e "MM/DD?YY hh:mm:ss AM/PM" > outfile.txt

Run the command above and modify the time stamps to capture a relevant period of time.  If I were you I would grab a time period that includes the application start up so you can see whats going on with nbpem as the application starts, as it jumps to 100%.  You may not see the cause if you look at a time period after the daemon has already gotten out of control. 

There were a number of known issues with nbpem throughout 6.5 versions but 6.5.3.1 should be stable.

Here is a extract from the list of fixes for 6.5.4

Etrack Incident = ET1480041

Description:
   A fix has been made to address an issue that caused NBPEM to core dump if it
   was unable to obtain the configuration information during startup.


NBPEM was prone to core'ing through 6.5.2 and 6.5.3 for a whole number of reasons.

Does your system have core dumps sitting around. Clean them up if so.

Also, have a search through the INDEX of technotes for NetBackup.
http://seer.support.veritas.com/docs/NBUESVR_index.htm
Just hit in a keyword nbpem to search for technotes with that in the title page.
Or use Google for that
http://www.google.com/search?hl=en&safe=active&q=intitle%3Anbpem+site%3Aseer.entsupport.symantec.com&btnG=Search&meta=

Or use the actual KB search on the NetBackup Support page.

Did you extract the nbpem portion of the unified logs?  If so were you able to find anything during the time period that the application starts when the daemon throttles up to 100%.  If you are not seeing anything in the logs, you may want to bump your logging level up to 5 so that you capture more diagnostic info. 

thanks for your help guys.  I was on to Symantec for a  long time about this one!  and eventually, it was just corrupt policies.
so basically, we removed the /usr/openv/netbackup/db/class directory and restored from tape.

A quick restart later and we were there!  As a well known meercat said, "Simples"