Unable to decommission Medias server using nbdecommission command

[Anth105]

I am trying to decommission an old media server using nbdecommission without any luck

nbdecommission -oldserver ouragan -newserver "newserver_hostname"

You want to decommission ouragan. Is that correct? (y/n) [n]: y

Administrative Pause set for machine ouragan

Before proceeding further, please make sure that no jobs are running on

media server ouragan. This command may not be able to decommission the media

server with active jobs on it.

Continue? (y/n) [n]: y

Server ouragan is a member of the following media sharing groups:

UNRESTRICTED_SHARING_GROUP

----------------------------

------------------------

Processing config file on masterserer

nothing to update

Stopping Volume Manager daemon/service on ouragan

Could not connect to vmd on host ouragan (70)

Deleting host ouragan

unable to delete host ouragan

Here is the background of the issue:

Env:

OS :Linux 6.2

Netbackup 7. 5 running both master and media servers

Upgrade from 6.5.6 to 7.5 ( no patch installed )

Move all volumes from the old server to new media server

# bpmedia -movedb -allvolumes -newserver newserver_hostname  -oldserver ouragan -v

<2>bpmedia: INITIATING: version NetBackup 7.5 created: 2012020801

<8>bpmedia: BPMEDIA: moving all media from server ouragan to server to newserver_hostname

<8>change_server_all_volumes: Change media server succeeded for all media

Next step is to delete the old hostname.

nbemmcmd -deletehost -machinename ouragan -machinetype media

NBEMMCMD, Version: 7.5

The function returned the following failure status:

the media is allocated for use (199)

Command did not complete successfully.

Here is an excerpt of the admin logs:

15:34:48.098 [6207] <2> : NBEMMCMD -deletehost -machinename ouragan -machinetype media

15:34:48.120 [6207] <2> Orb::init: Enabling ORBNativeCharCodeSet UTF-8(Orb.cpp:594)

15:34:48.121 [6207] <2> Orb::init: initializing ORB EMMlib_Orb with: nbemmcmd -ORBSvcConfDirective "-ORBDottedDecimalAddresses 0" -)

15:34:48.122 [6207] <2> Orb::init: caching EndpointSelectorFactory(Orb.cpp:840)

15:34:48.123 [6207] <2> Orb: etOrbConnectTimeout: timeout seconds: 60(Orb.cpp:1481)

15:34:48.123 [6207] <2> Orb: etOrbRequestTimeout: timeout seconds: 1800(Orb.cpp:1490)

15:34:48.270 [6207] <16> emmlib_DeleteHost: DeleteMachine failed, emmError = 2020003, nbError = 0

15:34:48.290 [6207] <16> nbemmcmd: (-) Translating EMM_ERROR_FragmentsStillExist(2020003) to 199 in the Media context

15:34:48.318 [6207] <16> nbemmcmd: (-) Translating EMM_ERROR_FragmentsStillExist(2020003) to 199 in the Media context

15:34:48.328 [6207] <16> nbemmcmd: Run time failure: The function returned the following failure status:

the media is allocated for use (199)

I then ran the nbd_unlaod -dbn to check any media or backup id associate with the old media server /

#usr/openv/db/bin/nbdb_unload -dbn NBDB /tmp/unload

]# grep ouragan *

721.dat:'1000007',0x51F32B5E487311E2800086DCBB22CCCD,'ouragan','ouragan','1000002','0','0','1','9','23','','4','5','750000','0','','

722.dat:'ouragan','1','1000007','2012-12-17 13:58:10.821672','2012-12-17 09:58:10.821749'

839.dat:'37','1000002','ouragan'

840.dat:'1000002','ouragan','37'

844.dat:'19','1000002','ouragan','PL_BACKUPHOST_OURAGAN','1355813779','0','0','1','1','0','/nbu_test','/nbu_test/ouragan_1355813779'

846.dat:'1368','1000002',,,,,,'ouragan','37','PL_SYS_UNIX_CIE_HC','1335841221','0','0','0','2','0','0','0','17495104','1','1','0',''

846.dat:'1369','1000002',,,,,,'ouragan','37','PL_SYS_UNIX_CIE_HC','1338279074','0','0','0','2','0','0','0','17489760','1','1','0',''

846.dat:'1370','1000002',,,,,,'ouragan','37','PL_SYS_UNIX_CIE_HC','1340705823','0','0','0','2','0','0','0','17494976','1','1','0',''

I tried to expire the backupid with : #bpexpdate -backupid <backupid> -d 0

Are you SURE you want to delete xxxxx_1355813779 y/n (n)? y

no entity was found

Have I missed something ?  or there is any bug in NBU 7.5 ? 

I cannot upgrade to NBU 7.5.0.XX  until It has been  tested and valided by our enginieering Team

any help will be much appreciated

PS: the old vserver is now turned off and physically moved out.

Regards

Antho05

[Mark_Solutions]

The nbu_test part maybe a clue?

Did you have a test disk storage unit at some point that no longer exists but had valid images on it

If these are disk images the bpexpdate command will fail as it cannot access them to delete them - try adding the -force switch to get them out of the catalog and see if that helps

Also run a catalog verify query against the old media server (as far back as you can) to see if any images still exist

[Nicolai]

Describe the use of nbstlutil remove_all command (I think you will need it):

http://www.symantec.com/docs/TECH173076

Another example how to send a disk pool to the eternal bit fields.

http://www.symantec.com/docs/TECH187294

[D_Flood]

I ran into a similar issue and (after a call to Support) it turns out I had to run the script twice.  Once for the short name and one for the fully qualified name.  You might try that to see if some images were assigned under the full name.

[Anth105]

Hi all,

Many thanks for your all replies.

I will test each of the suggested solutions and keep you posted.

[Anth105]

Hi all,

I have tried all without much sucess

#nbstlutil remove_all -backupid ouragan_1355813779
This operation will has been deprecated and is no longer available.

Extract of the admin log

17:06:53.505 [18127] <2> nbstlutil: INITIATING: NetBackup 7.5 created: 0
17:06:53.505 [18127] <2> nbstlutil: nbstlutil remove_all -backupid ouragan_1355813779
17:06:53.533 [18127] <2> Orb::init: Enabling ORBNativeCharCodeSet UTF-8(Orb.cpp:594)
17:06:53.534 [18127] <2> Orb::init: initializing ORB EMMlib_Orb with: nbstlutil -ORBSvcConfDirective "-ORBDottedDecimalAddresses 0" -ORBSvcConfDirective "static Resource_Factory '-ORBNativeCharCodeSet UTF-8'" -ORBSvcConfDirective "static PBXIOP_Factory '-enable_keepalive'" -ORBSvcConfDirective "static EndpointSelectorFactory ''" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory PBXIOP_Factory'" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory IIOP_Factory'" -ORBDefaultInitRef '' -ORBSvcConfDirective "static PBXIOP_Evaluator_Factory '-orb EMMlib_Orb'" -ORBSvcConfDirective "static Resource_Factory '-ORBConnectionCacheMax 1024 '" -ORBSvcConf /dev/null -ORBSvcConfDirective "static Server_Strategy_Factory '-ORBMaxRecvGIOPPayloadSize 268435456'"(Orb.cpp:825)
17:06:53.537 [18127] <2> Orb::init: caching EndpointSelectorFactory(Orb.cpp:840)
17:06:53.537 [18127] <2> Orb::setOrbConnectTimeout: timeout seconds: 60(Orb.cpp:1481)
17:06:53.537 [18127] <2> Orb::setOrbRequestTimeout: timeout seconds: 1800(Orb.cpp:1490)
17:06:53.550 [18127] <2> Orb::init: Enabling ORBNativeCharCodeSet UTF-8(Orb.cpp:594)
17:06:53.550 [18127] <2> Orb::init: initializing ORB Default_CLIENT_Orb with: Unknown -ORBSvcConfDirective "-ORBDottedDecimalAddresses 0" -ORBSvcConfDirective "static Resource_Factory '-ORBNativeCharCodeSet UTF-8'" -ORBSvcConfDirective "static PBXIOP_Factory '-enable_keepalive'" -ORBSvcConfDirective "static EndpointSelectorFactory ''" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory PBXIOP_Factory'" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory IIOP_Factory'" -ORBDefaultInitRef '' -ORBSvcConfDirective "static PBXIOP_Evaluator_Factory '-orb Default_CLIENT_Orb'" -ORBSvcConfDirective "static Resource_Factory '-ORBConnectionCacheMax 1024 '" -ORBSvcConf /dev/null -ORBSvcConfDirective "static Server_Strategy_Factory '-ORBMaxRecvGIOPPayloadSize 268435456'"(Orb.cpp:825)
17:06:53.551 [18127] <2> Orb::init: caching EndpointSelectorFactory(Orb.cpp:840)
17:06:53.551 [18127] <2> nbstlutil: EXITING, status= 0
17:06:59.393 [18139] <8> vnet_same_host_and_update: [vnet_addrinfo.c:2820] name2 is empty 0 0x0

# bpexpdate -backupid ouragan_1355813779 -d 0 -force
no entity was found

There is no way the catalog verify query can be carried out against the old media server. there is no option to select a media server. Nevertheless I ran it as far back as I can and I found no backup images associated with the old server media

I run also the script twice one on short name and one for the fully qualified name and this made no difference. still the same "Could not connect to vmd on host ouragan (70) " and " UNRESTRICTED_SHARING_GROUP " message

BTW,  the backup were no disk pool involved

I think I have no choice but to contact support

[Nicolai]

The UNRESTRICTED_SHARING_GROP is only a informational message.

[Mark_Solutions]

So what is left of the original server?

It is nice to run the nbdecomission tool to do it for you but you don't have to.

Run it with the -list_ref just to show what exists and then manually remove everything associated with it (SLPs, Storage Units, Disk Pools, Tapes Drives etc., - in that order i believe)

If anything won't delete then that is probably the clue to what is wrong