cancel
Showing results for 
Search instead for 
Did you mean: 

Image Cleanup status 37

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Master: Linux RHEL 
Media servers: Mix of various OS's
NBU version: 7.5.0.3 on master and all media servers

This is an 'inherited' environment and nobody knows anything about media09_bck.

How media server was removed with valid images in the catalog is a mystery.

Image cleanup fails with following details:

 

08/02/2012 15:03:42 - Info nbdelete (pid=6628) deleting expired images. Media Server: media09_bck Media: /dd/09
08/02/2012 15:03:42 - Error nbdelete (pid=6628) Fragments were not removed. Server media09_bck is invalid (37 )
08/02/2012 15:03:43 - Info nbdelete (pid=6628) deleting expired images. Media Server: media09_bck Media: /dd_co/09_dd
08/02/2012 15:03:43 - Error nbdelete (pid=6628) Fragments were not removed. Server media09_bck is invalid (37 )

 

nbemmcmd -listhosts
NBEMMCMD, Version: 7.5.0.3
The following hosts were found:
server           master
master           master
media            media02
media            media01
media            media04
foreign_media    media09_bck
media            media03

Command completed successfully.

 

 

nbemmcmd -deletehost -machinename media09_bck -machinetype foreign_media
NBEMMCMD, Version: 7.5.0.3
The function returned the following failure status:
the media is allocated for use (199)

Command did not complete successfully.

Unable to find evidence of allocated media:

nbrbutil -dump

 

MDS allocations in EMM:

        MdsAllocation: allocationKey=25267 jobType=1 mediaKey=0 mediaId= driveKey=0 driveName= drivePath= stuName=media01_disk_backup2 masterServerName=master mediaServerName=media01 ndmpTapeServerName= diskVolumeKey=60 mountKey=109 linkKey=0 fatPipeKey=0 scsiResType=0 serverStateFlags=0

        MdsAllocation: allocationKey=25268 jobType=1 mediaKey=0 mediaId= driveKey=0 driveName= drivePath= stuName=media01_disk_backup3 masterServerName=master mediaServerName=media01 ndmpTapeServerName= diskVolumeKey=71 mountKey=120 linkKey=0 fatPipeKey=0 scsiResType=0 serverStateFlags=0

 

nbdecommission fails with :

unable to obtain machine record for media09_bck, machine type media server

 

Please help!

How do we get rid of this 'ghost' media server?

1 ACCEPTED SOLUTION

Accepted Solutions

Radhakrishnan
Level 4
Employee Accredited Certified

This happens because the images you have imported EMM has been inherited from this server.

Try # bpimage - newserver <> -oldserver <>

then try to delete the server

nbemmcmd -deletehost -machinename media09_bck -machinetype foreign_media

View solution in original post

17 REPLIES 17

mph999
Level 6
Employee Accredited

Ouch,

"nobody knows anything about media09_bck" - neither dows NBU it would appear ...

Manual SQL script, or NBCC I suspect - log call time.

BUT ..  it seems the media server is gone, which is why it is "foregin"

Images may include the names of media servers that do not exist in the current
domain. These media servers are known as foreign media servers.
5 Chapter
The following describes when a foreign media server may appear in a backup
image:
■ When a backup image is migrated from one domain to another.
■ When a media server is decommissioned.
Starting with NetBackup 7.5, foreign media servers also appear in the output for
the following command:
nbemmcmd -listhosts
The output displays the foreign media server as follows:
foreign_media server
NetBackup 7.5 operational behavior changes
About foreign media servers
36
 
So the question may be more ' how to get rid of the images in the DB .. presuming the images were on a disk STU that is now gone " ?
 

Martin

mph999
Level 6
Employee Accredited

Hmm,

Did you try nbdelete -allvolumes -force

M

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Yes, we have tried nbdelete -allvolumes -force.

Not sure if nbdelete shows up as Image Cleanup in Activity Monitor or if it was coincidence that we saw failed Image Cleanup job shortly after running nbdelete.
No idea if PID 6628  (
nbdelete (pid=6628) in Activity Monitor) was the same as our nbdelete command or another one called by 'normal' Image Cleanup.
I guess admin log (if enabled) will tell us but I think that will be purely academical and not really help...

 

mph999
Level 6
Employee Accredited

OK, can't get to the GUI at the sec, but nbdelete shows in the trylogs, so should be in the gui.

I'm wondering if NBCC can pick this up, it does some disk stuff now, and runs some sql I believe if necessary.  Could be worth trying that.

M

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Thanks Martin

Will suggest that.  This means that we need to log a Support call, right?? wink

 

mph999
Level 6
Employee Accredited

Yep, looking that way I think ...  Might want to include a nbdb_unload output, bpimagelist and nbstlutil list + nbsu -c -t ...

M

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Will update here when we have response from Support.

Since nothing is 'broken', customer and colleagues will probably wait till Monday to log a call.

Thanks for all your support, Martin!

Praveen_Sundr1
Not applicable
Accredited

Hi Marianne ,

You can try below steps to fix this issue,

Step 1 : Check what all media still assigned to this host i.e. media09_bck

bpmedialist -l -h media09_bck

Step 2 : Move any tapes assigned to this host to any working media server ,

bpmedia -movedb -m <tape_number> -newserver <working_media_server_name> -oldserver <decommission_media_server_name>

Step 3 : If there are any image referrring to disk volume in bpmedialist output in Step1, run below command against each volume,

nbdelete -purge_deletion_list -media_id  'Disk_Path' -force

Step 4 : Run below command ,

nbdelete -allvolumes -force

Step 5 : Remove problematic media server from EMM by using below command

nbemmcmd -deletehost -machinename media09_bck -machinetype foreign_media

Let me know if it fixed this issue .

Regards,
Praveen

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Thanks for your suggestions Praveen.

Sincere apologies for not updating this post...

There have been other issues at this particular customer that received priority.

We still need to get rid of this 'ghost media server' - hopefully next week some time.

I will discuss suggestions in this thread with the customer and report back.

Radhakrishnan
Level 4
Employee Accredited Certified

This happens because the images you have imported EMM has been inherited from this server.

Try # bpimage - newserver <> -oldserver <>

then try to delete the server

nbemmcmd -deletehost -machinename media09_bck -machinetype foreign_media

Radhakrishnan
Level 4
Employee Accredited Certified

I encountered this issue when we did catalgo manipulation for 7.5 and after an hour of research I could resolve it .

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Thanks for all the ideas and advice.

There are unfortunaly issues with failed backups that is taking up everyone's time and energy (logs uploaded to Symantec support, tests, more logs, etc...)

We will get back to this non-critical issue once the 'dust has settled'....

Mark_Solutions
Level 6
Partner Accredited Certified

Hi Marianne - when you get chance to look at this again and if the above do not help (most should do unless the images for that server were on disk) then run an All Log Entries report - especially either side of midnight when the cleanup runs as it will actually list the image IDs of the problem images

From that, if they are on disk, they can be forcibly deleted against each image id after which you should be able to get rid of the rogue media server

the oldserver newserver usually sorts most things out and then just remove all reference to that server from the installation

Hope this helps

Genericus
Moderator
Moderator
   VIP   

Marianne - instead of specifying media use allvolumes.

 

bpmedia -movedb -allvolumes -newserver newservername -oldserver oldservername

 

It also runs much quicker than individual media id...

 

NetBackup 9.1.0.1 on Solaris 11, writing to Data Domain 9800 7.7.4.0
duplicating via SLP to LTO5 & LTO8 in SL8500 via ACSLS

Genericus
Moderator
Moderator
   VIP   

Once all volumes are off the old server, you should be able to remove or decommission it,

 

NetBackup 9.1.0.1 on Solaris 11, writing to Data Domain 9800 7.7.4.0
duplicating via SLP to LTO5 & LTO8 in SL8500 via ACSLS

Gnur
Level 3

Hi Marianne,

Thanks to Genericus, I was able to remove the "Ghost" media server I had already decomissioned but could not rid it from residing within the EMM DB tablespace. 

$ bpmedia -movedb -allvolumes -newserver <valid server> -oldserver <ghost media server>

$ nbemmcmd -deletehost -machinename <ghost media server> -machinetype <media or foreign_media>

Thanks again to all in this post that fixed my headache...

Gnur

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

I need to get back to the customer about this one... have not had any feedback since August...

Apologies to all for leaving this thread unsolved.