Image Cleanup status 37

Master: Linux RHEL
Media servers: Mix of various OS's
NBU version: 7.5.0.3 on master and all media servers

This is an 'inherited' environment and nobody knows anything about media09_bck.

How media server was removed with valid images in the catalog is a mystery.

Image cleanup fails with following details:

08/02/2012 15:03:42 - Info nbdelete (pid=6628) deleting expired images. Media Server: media09_bck Media: /dd/09
08/02/2012 15:03:42 - Error nbdelete (pid=6628) Fragments were not removed. Server media09_bck is invalid (37 )
08/02/2012 15:03:43 - Info nbdelete (pid=6628) deleting expired images. Media Server: media09_bck Media: /dd_co/09_dd
08/02/2012 15:03:43 - Error nbdelete (pid=6628) Fragments were not removed. Server media09_bck is invalid (37 )

nbemmcmd -listhosts
NBEMMCMD, Version: 7.5.0.3
The following hosts were found:
server           master
master           master
media            media02
media            media01
media            media04
foreign_media    media09_bck
media            media03

Command completed successfully.

nbemmcmd -deletehost -machinename media09_bck -machinetype foreign_media
NBEMMCMD, Version: 7.5.0.3
The function returned the following failure status:
the media is allocated for use (199)

Command did not complete successfully.

Unable to find evidence of allocated media:

nbrbutil -dump

MDS allocations in EMM:

        MdsAllocation: allocationKey=25267 jobType=1 mediaKey=0 mediaId= driveKey=0 driveName= drivePath= stuName=media01_disk_backup2 masterServerName=master mediaServerName=media01 ndmpTapeServerName= diskVolumeKey=60 mountKey=109 linkKey=0 fatPipeKey=0 scsiResType=0 serverStateFlags=0

        MdsAllocation: allocationKey=25268 jobType=1 mediaKey=0 mediaId= driveKey=0 driveName= drivePath= stuName=media01_disk_backup3 masterServerName=master mediaServerName=media01 ndmpTapeServerName= diskVolumeKey=71 mountKey=120 linkKey=0 fatPipeKey=0 scsiResType=0 serverStateFlags=0

nbdecommission fails with :

unable to obtain machine record for media09_bck, machine type media server

Please help!

How do we get rid of this 'ghost' media server?

Radhakrishnan
13 years ago
This happens because the images you have imported EMM has been inherited from this server.

Try # bpimage - newserver <> -oldserver <>

then try to delete the server

nbemmcmd -deletehost -machinename media09_bck -machinetype foreign_media

mph999
Level 6
13 years ago
Ouch,

"nobody knows anything about media09_bck" - neither dows NBU it would appear ...

Manual SQL script, or NBCC I suspect - log call time.

BUT .. it seems the media server is gone, which is why it is "foregin"

Images may include the names of media servers that do not exist in the current

domain. These media servers are known as foreign media servers.

5 Chapter

The following describes when a foreign media server may appear in a backup

image:

■ When a backup image is migrated from one domain to another.

■ When a media server is decommissioned.

Starting with NetBackup 7.5, foreign media servers also appear in the output for

the following command:

nbemmcmd -listhosts

The output displays the foreign media server as follows:

foreign_media server

NetBackup 7.5 operational behavior changes

About foreign media servers

36

So the question may be more ' how to get rid of the images in the DB .. presuming the images were on a disk STU that is now gone " ?

Martin
mph999
Level 6
13 years ago
Hmm,

Did you try nbdelete -allvolumes -force

M
Marianne
Level 6
13 years ago
Yes, we have tried nbdelete -allvolumes -force.

Not sure if nbdelete shows up as Image Cleanup in Activity Monitor or if it was coincidence that we saw failed Image Cleanup job shortly after running nbdelete.
No idea if PID 6628 (nbdelete (pid=6628) in Activity Monitor) was the same as our nbdelete command or another one called by 'normal' Image Cleanup.
I guess admin log (if enabled) will tell us but I think that will be purely academical and not really help...
mph999
Level 6
13 years ago
OK, can't get to the GUI at the sec, but nbdelete shows in the trylogs, so should be in the gui.

I'm wondering if NBCC can pick this up, it does some disk stuff now, and runs some sql I believe if necessary. Could be worth trying that.

M
Marianne
Level 6
13 years ago
Thanks Martin

Will suggest that. This means that we need to log a Support call, right??
mph999
Level 6
13 years ago
Yep, looking that way I think ... Might want to include a nbdb_unload output, bpimagelist and nbstlutil list + nbsu -c -t ...

M
Marianne
Level 6
13 years ago
Will update here when we have response from Support.

Since nothing is 'broken', customer and colleagues will probably wait till Monday to log a call.

Thanks for all your support, Martin!
Praveen_Sundr1
13 years ago
Hi Marianne ,

You can try below steps to fix this issue,

Step 1 : Check what all media still assigned to this host i.e. media09_bck

bpmedialist -l -h media09_bck

Step 2 : Move any tapes assigned to this host to any working media server ,

bpmedia -movedb -m <tape_number> -newserver <working_media_server_name> -oldserver <decommission_media_server_name>

Step 3 : If there are any image referrring to disk volume in bpmedialist output in Step1, run below command against each volume,

nbdelete -purge_deletion_list -media_id 'Disk_Path' -force

Step 4 : Run below command ,

nbdelete -allvolumes -force

Step 5 : Remove problematic media server from EMM by using below command

nbemmcmd -deletehost -machinename media09_bck -machinetype foreign_media

Let me know if it fixed this issue .

Regards,
Praveen
Marianne
Level 6
13 years ago
Thanks for your suggestions Praveen.

Sincere apologies for not updating this post...

There have been other issues at this particular customer that received priority.

We still need to get rid of this 'ghost media server' - hopefully next week some time.

I will discuss suggestions in this thread with the customer and report back.
Radhakrishnan
Level 4
13 years ago
This happens because the images you have imported EMM has been inherited from this server.

Try # bpimage - newserver <> -oldserver <>

then try to delete the server

nbemmcmd -deletehost -machinename media09_bck -machinetype foreign_media