08-03-2012 02:19 AM
Master: Linux RHEL
Media servers: Mix of various OS's
NBU version: 7.5.0.3 on master and all media servers
This is an 'inherited' environment and nobody knows anything about media09_bck.
How media server was removed with valid images in the catalog is a mystery.
Image cleanup fails with following details:
08/02/2012 15:03:42 - Info nbdelete (pid=6628) deleting expired images. Media Server: media09_bck Media: /dd/09 08/02/2012 15:03:42 - Error nbdelete (pid=6628) Fragments were not removed. Server media09_bck is invalid (37 ) 08/02/2012 15:03:43 - Info nbdelete (pid=6628) deleting expired images. Media Server: media09_bck Media: /dd_co/09_dd 08/02/2012 15:03:43 - Error nbdelete (pid=6628) Fragments were not removed. Server media09_bck is invalid (37 )
nbemmcmd -listhosts NBEMMCMD, Version: 7.5.0.3 The following hosts were found: server master master master media media02 media media01 media media04 foreign_media media09_bck media media03 Command completed successfully.
nbemmcmd -deletehost -machinename media09_bck -machinetype foreign_media NBEMMCMD, Version: 7.5.0.3 The function returned the following failure status: the media is allocated for use (199) Command did not complete successfully.
Unable to find evidence of allocated media:
nbrbutil -dump
MDS allocations in EMM: MdsAllocation: allocationKey=25267 jobType=1 mediaKey=0 mediaId= driveKey=0 driveName= drivePath= stuName=media01_disk_backup2 masterServerName=master mediaServerName=media01 ndmpTapeServerName= diskVolumeKey=60 mountKey=109 linkKey=0 fatPipeKey=0 scsiResType=0 serverStateFlags=0 MdsAllocation: allocationKey=25268 jobType=1 mediaKey=0 mediaId= driveKey=0 driveName= drivePath= stuName=media01_disk_backup3 masterServerName=master mediaServerName=media01 ndmpTapeServerName= diskVolumeKey=71 mountKey=120 linkKey=0 fatPipeKey=0 scsiResType=0 serverStateFlags=0
nbdecommission fails with :
unable to obtain machine record for media09_bck, machine type media server
Please help!
How do we get rid of this 'ghost' media server?
Solved! Go to Solution.
08-17-2012 05:15 PM
This happens because the images you have imported EMM has been inherited from this server.
Try # bpimage - newserver <> -oldserver <>
then try to delete the server
nbemmcmd -deletehost -machinename media09_bck -machinetype foreign_media
08-03-2012 02:45 AM
Ouch,
"nobody knows anything about media09_bck" - neither dows NBU it would appear ...
Manual SQL script, or NBCC I suspect - log call time.
BUT .. it seems the media server is gone, which is why it is "foregin"
Martin
08-03-2012 02:55 AM
Hmm,
Did you try nbdelete -allvolumes -force
M
08-03-2012 03:10 AM
Yes, we have tried nbdelete -allvolumes -force.
Not sure if nbdelete shows up as Image Cleanup in Activity Monitor or if it was coincidence that we saw failed Image Cleanup job shortly after running nbdelete.
No idea if PID 6628 (nbdelete (pid=6628) in Activity Monitor) was the same as our nbdelete command or another one called by 'normal' Image Cleanup.
I guess admin log (if enabled) will tell us but I think that will be purely academical and not really help...
08-03-2012 03:29 AM
OK, can't get to the GUI at the sec, but nbdelete shows in the trylogs, so should be in the gui.
I'm wondering if NBCC can pick this up, it does some disk stuff now, and runs some sql I believe if necessary. Could be worth trying that.
M
08-03-2012 03:43 AM
Thanks Martin
Will suggest that. This means that we need to log a Support call, right??
08-03-2012 04:08 AM
Yep, looking that way I think ... Might want to include a nbdb_unload output, bpimagelist and nbstlutil list + nbsu -c -t ...
M
08-03-2012 04:42 AM
Will update here when we have response from Support.
Since nothing is 'broken', customer and colleagues will probably wait till Monday to log a call.
Thanks for all your support, Martin!
08-06-2012 05:57 AM
Hi Marianne ,
You can try below steps to fix this issue,
Step 1 : Check what all media still assigned to this host i.e. media09_bck
bpmedialist -l -h media09_bck
Step 2 : Move any tapes assigned to this host to any working media server ,
bpmedia -movedb -m <tape_number> -newserver <working_media_server_name> -oldserver <decommission_media_server_name>
Step 3 : If there are any image referrring to disk volume in bpmedialist output in Step1, run below command against each volume,
nbdelete -purge_deletion_list -media_id 'Disk_Path' -force
Step 4 : Run below command ,
nbdelete -allvolumes -force
Step 5 : Remove problematic media server from EMM by using below command
nbemmcmd -deletehost -machinename media09_bck -machinetype foreign_media
Let me know if it fixed this issue .
Regards,
Praveen
08-10-2012 05:32 AM
Thanks for your suggestions Praveen.
Sincere apologies for not updating this post...
There have been other issues at this particular customer that received priority.
We still need to get rid of this 'ghost media server' - hopefully next week some time.
I will discuss suggestions in this thread with the customer and report back.
08-17-2012 05:15 PM
This happens because the images you have imported EMM has been inherited from this server.
Try # bpimage - newserver <> -oldserver <>
then try to delete the server
nbemmcmd -deletehost -machinename media09_bck -machinetype foreign_media
08-17-2012 05:17 PM
I encountered this issue when we did catalgo manipulation for 7.5 and after an hour of research I could resolve it .
08-17-2012 09:50 PM
Thanks for all the ideas and advice.
There are unfortunaly issues with failed backups that is taking up everyone's time and energy (logs uploaded to Symantec support, tests, more logs, etc...)
We will get back to this non-critical issue once the 'dust has settled'....
08-21-2012 08:15 AM
Hi Marianne - when you get chance to look at this again and if the above do not help (most should do unless the images for that server were on disk) then run an All Log Entries report - especially either side of midnight when the cleanup runs as it will actually list the image IDs of the problem images
From that, if they are on disk, they can be forcibly deleted against each image id after which you should be able to get rid of the rogue media server
the oldserver newserver usually sorts most things out and then just remove all reference to that server from the installation
Hope this helps
08-21-2012 12:37 PM
Marianne - instead of specifying media use allvolumes.
bpmedia -movedb -allvolumes -newserver newservername -oldserver oldservername
It also runs much quicker than individual media id...
08-21-2012 12:38 PM
Once all volumes are off the old server, you should be able to remove or decommission it,
10-09-2012 09:47 PM
Hi Marianne,
Thanks to Genericus, I was able to remove the "Ghost" media server I had already decomissioned but could not rid it from residing within the EMM DB tablespace.
$ bpmedia -movedb -allvolumes -newserver <valid server> -oldserver <ghost media server>
$ nbemmcmd -deletehost -machinename <ghost media server> -machinetype <media or foreign_media>
Thanks again to all in this post that fixed my headache...
Gnur
10-09-2012 10:04 PM
I need to get back to the customer about this one... have not had any feedback since August...
Apologies to all for leaving this thread unsolved.