NetBackup 6.5.1 cannot delete drive and media server

[asg2ki]

Hi All,

I need some help with removing a drive and host record from the EMM database of NetBackup 6.5.1. Hopefully someone will be able to help me out.

I can't remove a drive and media server neither by CLI nor by GUI.

My configuration is as follow:

NetBackup Master instance in a 2-Node Windows 2003 Cluster environment.

NetBackup Media instances - Solaris 9, 10 and Windows 2000.

The problem came in first when I installed a new Win2K media server with attached to it DLT 8000 drive via Compaq Modular Data Router. The device was detected just fine and I was able to configure it through NetBakcup GUI propperly by all "port", "bus", "target" and "lun" parameters, however I wasn't able to make any backups due to error messages regarding system ioctl problems. Most probably the configuration wouldn't work at all due to incompatibility reasons, but anyway I decided to decommision both the box and attached tape drive. When I tried removing the media server from NetBackup GUI, I received the following error message:

Unable to remove device host media_server from the Enterprise Media Manager Server master_server: global device database record not found(152).

So I tried removing the media server from CLI with the help of "nbemmcmd.exe -deletehost -machinetype media -machinename media_server" but I received another error message:

requested drive is already reserved (130)

Command did not complete successfully.

After looking more carefully at the situation I decided to try first removing the drive and its corresponding volume from EMM. The volume was deleted successfully, however now I'm stuck with the drive. Whenever I try to remove the drive I receive this message from GUI:

Error connecting to oprd on media_server: oprd returned abnormal status(96).

However if I try to remove the drive manually through CLI by initiating "tpconfig -delete -drive -asciiname drive_name", I'm getting this message:

Drive name drive_name does not exist

...which is strange because I can see the drive by its name within the NetBackup interface even after fully restarting the master server. I noticed another thing within the media server's application logs. Whenever the "NetBackup Device manager" service is started, I'm getting an alert with event ID 2753 which says:

ltid can not be started while resources are assigned to the host.

So now I'm really stuck. It's also strange that if the same service is started, I cannot rescan the host for new media devices, however if it's stopped then the rescan operation goes propperly and it redetects the attached hardware, respectively it updates the EMM database.

I also tried removing the server and the drive while the media server was turned off (thought this might be a sync related issue), but NetBackup just keeps rejecting the removal with:

Error connecting to oprd on media_server: cannot connect to vmd(70).

...for the drive and:

Unable to remove device host media_server from the Enterprise Media Manager Server master_server: global device database record not found(152).

...for the media server messages.

Any help will be appreciated very much.

Kind regards

[ahlip]

Hi,

Is your NetBackup Master server also clustered by MSCS?

Have you tried running "nbemmcmd -listhosts" and see if the residual entry for the decom Media server is still there?

[asg2ki]

Hi ahlip,

First of all nice to see you on this thread too

Only the master server is clustered by MSCS. All the media servers are standalone boxes. And yes I can see the faulty server in the list after initiating the "nbemmcmd -listhosts".

[Omar_Villa]

have u deleted the STU's regarding this box? maybe u need to get rid of everything who points to the box, then you will be able to delete it, so ensure tpconfig, tpautoconf, vmglob and emm DB are cleaned up and ensure that the drives is not being use by an other shared box.

hope this helps.

regards

[asg2ki]

Hi Omar,

I deleted everything related to this media server and its drive as per my humble knowledge but I'll appreciate very much if you could give me some examples on exact usage of the mentioned commands.

I've already tried some of them but without much of positive results.

Kind regards

[Rakesh_Khandelw]

You can try this -

1. run "vmglob -listall -b" and check what all is associated with the media server you are trying to delete.
2. Try to delete remaining drives and robot through vmglob -delete

For example -

/usr/openv/volmgr/bin/vmglob -listall -b

VMGLOB4.5 drive STK.T10000A.002 531001003335 mediasrv1 t0q258 0 7 TLD hcart3 0x2000 - 0 STK~~~~~T10000A~~~~~~~~~1.32 - - - -1 -1 -1

   /usr/openv/volmgr/bin/vmglob -delete -drive -drvtype hcart3 -serial 531001003335 -name STK.T10000A.002 -devhost mediasrv1

3. Now try to delete the media server
   /usr/openv/volmgr/bin/vmglob -delete -drive -drvtype hcart3 -serial 531001003335 -name STK.T10000A.002 -devhost mediasrv1

To verify the media server has been removed run:

/usr/openv/netbackup/bin/admincmd/nbemmcd -listhosts

The command now should be successful.

Restart NetBackup to ensure the changes have taken affect.

[asg2ki]

Well it's still the same...

After I initiated the "vmglob -listall -b" command the output was very simple:

C:\Program Files\VERITAS\Volmgr\bin>vmglob.exe -listall -b
device
type    device name           serial number             host name
------------------------------------------------------------------------
drive   Drive006              PMB48P2453                alpha
drive   Drive001              CX849S1115                nbmedia02
drive   Drive000              CX906S7243                nbmedia02
drive   Drive003              CX901S4072                nbmedia01
drive   Drive002              CX711S0110                nbmedia01

So I used the "vmglob -install" instead to find out all parameters related to Drive006:

C:\Program Files\VERITAS\Volmgr\bin>vmglob.exe -listall
=====================================================================
device type:               drive
device name:               Drive006
serial number:             PMB48P2453
hostname:                  alpha
volume database hostname:  zeus
robot number:              -1
robot drive number:        -1
robot type:                NONE
drive type:                dlt
flags:                     0x0
worldwide name:            -
worldwide name type:       0
inquiry:                   COMPAQ  DLT8000         0259
library name:              -
vendor drive name:         -
robot control host:        -
ACS:                       -1
LSM:                       -1
PANEL:                     -1
DRIVE:                     -1
=====================================================================

Based on the information above I engaged the following command:

C:\Program Files\VERITAS\Volmgr\bin>vmglob.exe -delete -drive -drvtype dlt -serial PMB48P2453 -name Drive006 -devhost alpha

Now this resulted with the following error message:

global device database record not found (152)

I also tried to engage the same command without specifying the "-devhost" parameter but then I get this error:

invalid host name (136)

Any other suggestions ?

[AKopel]

Did you guys ever figure this out?
I'm having the same problem...


Thanks!
AK

[AKopel]

Nevermind... I fixed this by bpdowning/bpuping my master...

Aaron

[Manoj_Siricilla]

Hi,

You could try this...

start vmd on the media server on which the drive needs to be removed.

vmd can be started in two ways

1. /usr/openv/volmgr/bin/vmd (or)

2. /usr/openv/volmgr/bin/vmctrldbm -i

and remove the drive by the index no. using the command

tpconfig -delete -drive <drive-index>

This should sync with the EMM database.

Let me know the outcome.

Thanks,

Manoj

[MWoody]

We had a similar problem with oprd and the 96 error. It turned out that we had a problem where resources were orphaned. This caused some of our media servers to not be active for tape. We could still use disk storage units but we could not use and or remove any of the tape resources. It seems that this is a NetBackup 6.5.x problem. Below is what we did to solve the problem:

Stop all NetBackup jobs
Shutdown NetBackup
Restart NetBackup on the master server

Run:
nbrbutil -resetAll

Clear out all of the files (except the README file) in the following 
directory on the effected media servers:
/usr/openv/volmgr/misc

Make sure there are not any jobs are running before executing the nbrbutil -resetAll command. This command removes all entries from the database that have to do with allocation of resources. If a job is running with a reserved resource and you run this command, who knows what strange things will happen.