Problem with the EMM database. Cannot see tape drives.

[manmountain]

Hi, I have had some issues with root file system filling up on master server. I thien had some problems with the performance of the tape drives so I rebuilt the SG driver and attempted to add back in the tape drives. However this failed due to error on the EMM database. This is the output I am getting from robtest. Solaris OS can see the tape drives fine.

$ robtest
EMM interface initialization failed, status = 77

$ ./sgscan
/dev/sg/c0t0l0: Disk (/dev/rdsk/c0t0d0): "HP      LOGICAL VOLUME"
/dev/sg/c0tw201a3cd92b3060fel0: Changer: "HP      MSL G3 Series"
/dev/sg/c0tw500143801600db10l0: Tape (/dev/rmt/0): "HP      Ultrium 5-SCSI"
/dev/sg/c0tw500143801600db16l0: Tape (/dev/rmt/1): "HP      Ultrium 5-SCSI"

When I attempt to add drives using the wizard I get error:

EMM database error (196)

Any help would be much appreciated as I am under a small bit of pressure to get this fixed!! :)

Thanks

[manmountain]

Seems to be a problem when I am nbemmcmd -getemmserver I am not getting any response other than:

Could not connect to vnetd over PBX. Trying connection directly to vnetd service on port = 13724

$ nbemmcmd -listhosts -verbose -display_server -machinename attembak5 -machinetype media
NBEMMCMD, Version:7.0.1
Failed to initialize EMM connection.  Verify that network access to the EMM server is available and that the services nbemm and pbx_exchange are running on the EMM server. (195)
Command did not complete successfully.

EMM server is sitting on the master server. I have had a look at the services in maintenance and I noticed all these were in maintenance. I attempted to clear and re-enable services. However I am still having the same problem.

$ svcs -xv
svc:/milestone/multi-user:default (multi-user milestone)
 State: maintenance since Fri May 17 17:48:28 2013
Reason: Start method died on Killed (9).
   See: http://sun.com/msg/SMF-8000-KS
   See: man -M /usr/share/man -s 1M init
   See: /var/svc/log/milestone-multi-user:default.log
Impact: 10 dependent services are not running:
        svc:/milestone/multi-user-server:default
        svc:/application/autoreg:default
        svc:/system/basicreg:default
        svc:/eric/install/eric_bootstrap_wrapper:default
        svc:/system/console-login:default
        svc:/system/zones:default
        svc:/network/dhcp-server:default
        svc:/application/stosreg:default
        svc:/system/emcpower/emcp_mond:default
        svc:/application/cde-printinfo:default

I have rebooted the server and the same services have transtioned into maintenance mode. The logs do not give much detail on what is going on.

Legacy init script "/etc/rc2.d/S99powermig" exited with return code 0.
Executing legacy init script "/etc/rc2.d/S99sneep".
Legacy init script "/etc/rc2.d/S99sneep" exited with return code 0.
[ May 19 01:17:04 Method "start" exited with status 0 ]
[ May 19 01:30:26 Executing start method ("/sbin/rc2 start") ]
Executing legacy init script "/etc/rc2.d/S00mpfs".
Legacy init script "/etc/rc2.d/S00mpfs" exited with return code 0.
Executing legacy init script "/etc/rc2.d/S10lu".
Legacy init script "/etc/rc2.d/S10lu" exited with return code 0.
Executing legacy init script "/etc/rc2.d/S20sysetup".
Legacy init script "/etc/rc2.d/S20sysetup" exited with return code 0.
Executing legacy init script "/etc/rc2.d/S40llc2".
Legacy init script "/etc/rc2.d/S40llc2" exited with return code 0.
Executing legacy init script "/etc/rc2.d/S42ncakmod".
Legacy init script "/etc/rc2.d/S42ncakmod" exited with return code 0.
Executing legacy init script "/etc/rc2.d/S47pppd".
Legacy init script "/etc/rc2.d/S47pppd" exited with return code 0.
Executing legacy init script "/etc/rc2.d/S70uucp".
Legacy init script "/etc/rc2.d/S70uucp" exited with return code 0.
Executing legacy init script "/etc/rc2.d/S71vxazd".
Legacy init script "/etc/rc2.d/S71vxazd" exited with return code 0.
Executing legacy init script "/etc/rc2.d/S72autoinstall".
Legacy init script "/etc/rc2.d/S72autoinstall" exited with return code 0.
Executing legacy init script "/etc/rc2.d/S73cachefs.daemon".
Legacy init script "/etc/rc2.d/S73cachefs.daemon" exited with return code 0.
Executing legacy init script "/etc/rc2.d/S76ACT_dumpscript".
Legacy init script "/etc/rc2.d/S76ACT_dumpscript" exited with return code 0.
Executing legacy init script "/etc/rc2.d/S77mpfs".
Legacy init script "/etc/rc2.d/S77mpfs" exited with return code 0.
Executing legacy init script "/etc/rc2.d/S77netbackup".
NetBackup network daemon started.
NetBackup client daemon started.
NetBackup SAN Client Fibre Transport daemon started.

[huanglao2002]

It's most like nbemm process not startup well.

Can you provide the "bpps -x" output and vxlogview -p nb -o nbemm -t 00:30:00 output.

PS:

-t : Displays the messages for the last hh:mm:ss time period.

[Marianne]

I agree - we need to see bpps output and emm log.

If master server is running low on disk space, EMM will be shutdown to prevent corruption of database.

[manmountain]

Hi there,

$ bpps -x
NB Processes
------------
    root  5577     1   0 11:53:05 ?           0:00 /usr/openv/netbackup/bin/admincmd/bpstsinfo -UPDATE
    root  1114     1   0 01:30:30 ?           0:00 /usr/openv/netbackup/bin/bpcd -standalone
    root  1111     1   0 01:30:30 ?           0:00 /usr/openv/netbackup/bin/vnetd -standalone
    root  3899  1413   0 11:25:45 ?           0:02 /usr/openv/netbackup/bin/bpdbm
    root  1170     1   0 01:30:34 ?           0:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/
    root  1378     1   0 02:52:53 ?           0:02 /usr/openv/netbackup/bin/nbevtmgr
    root  2759  1415   0 02:59:37 ?           0:17 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  3901  1415   0 11:25:46 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem_cleanup
    root  1444     1   0 02:53:02 ?           0:01 /usr/openv/netbackup/bin/nbsl
    root  1415     1   0 02:52:59 ?           0:06 /usr/openv/netbackup/bin/nbpem
    root  1417  1413   0 02:52:59 ?           0:00 /usr/openv/netbackup/bin/bpjobd
    root  1401     1   0 02:52:57 ?           0:02 /usr/openv/netbackup/bin/bprd
    root  1420     1   0 02:52:59 ?           0:36 /usr/openv/netbackup/bin/nbstserv
    root  1425     1   0 02:53:00 ?           0:05 /usr/openv/netbackup/bin/nbrmms
    root  1407     1   0 02:52:57 ?           0:00 /usr/openv/netbackup/bin/bpcompatd
    root  1410     1   0 02:52:58 ?           0:41 /usr/openv/netbackup/bin/nbjm
    root  1413     1   0 02:52:58 ?           0:00 /usr/openv/netbackup/bin/bpdbm
    root  1479     1   0 02:53:10 ?           0:00 /usr/openv/netbackup/bin/nbsvcmon
    root  1472     1   0 02:53:09 ?           0:00 /usr/openv/netbackup/bin/nbvault

MM Processes
------------
    root  1399     1   0 02:52:57 ?           0:01 vmd
    root  1393     1   0 02:52:56 ?           0:22 /usr/openv/volmgr/bin/ltid

Shared Symantec Processes
-------------------------
    root   839     1   0 01:30:25 ?           1:00 /opt/VRTSpbx/bin/pbx_exchange

• Java Web Console is also not starting

svc:/system/webconsole:console (java web console)
 State: maintenance since Sun May 19 12:11:43 2013
Reason: Start method exited with $SMF_EXIT_ERR_FATAL.
   See: http://sun.com/msg/SMF-8000-KS
   See: man -M /usr/share/man -s 1M smcwebserver
   See: /var/svc/log/system-webconsole:console.log
Impact: This service is not running.

• EMM log shows that some files maybe missing :o See attached

Thanks for the help!!

[manmountain]

Hi,

I have sent a previous message with logs and bpps output. Doesn't seem to be appearing as of yet.

The prob seems to be due to this script failing to start and timeouts waiting for some reason.

Executing legacy init script "/etc/rc2.d/S77netbackup".
NetBackup network daemon started.
NetBackup client daemon started.
NetBackup SAN Client Fibre Transport daemon started.
[ May 19 02:00:27 Method or service exit timed out.  Killing contract 82 ]

Would increasing the timeout on the multi-user svc help in this instance. Seems like there is probably a bigger issue that this.

[Marianne]

NBDB transaction log may have become corrupted as a result of disk filling up.

Have a look at server.log to see if there is additional info.
(Location: <program Files\Veritas\NetBackupDB\log>

To recreate NBDB log, follow steps in this TN:  http://www.symantec.com/docs/TECH162344

Let us know if this helps.

[manmountain]

Hi Marianne okay i'll take a look... would you be aware of a technote similar to above for Unix system to recreate NBDB transaction log?

I don't have the 'NetbackupDB' directory. I presume in Unix it should be here /usr/openv/db/data?

[manmountain]

Seems that you are correct marianne.... Output from Server.log

I. 05/19 19:58:27.     Transaction log: /usr/openv/db/data/BMRDB.log...
E. 05/19 19:58:27. Error: Cannot open transaction log file -- No such file or directory
I. 05/19 19:58:27.     Transaction log: /usr/openv/db/data/NBDB.log...
E. 05/19 19:58:27. Error: Cannot open transaction log file -- No such file or directory
E. 05/19 19:58:27. *** ERROR *** Assertion failed: 201117 (11.0.1.2279)
Attempt to close a file marked as dirty
I. 05/19 19:58:28. Recovery complete

[Marianne · Accepted solution]

Apologies!!! I don't know what made me post TN for Windows....

Here is the Unix TN: http://www.symantec.com/docs/TECH55797

[manmountain]

Ya beauty ya :)

Followed this technote and it's back!!

http://www.symantec.com/business/support/index?page=content&id=TECH55797

Thanks a million Marianne!!

[Marianne]

Great stuff!