image Cleanup did not complete

[A_M_Ash_]

Hi,

i have a problem that the image Cleanup process started and take a long time -more than one day- & did not complete then give me error number 50

i need help to solve that problem as it is first time to face it ...  thank .

master server data:

Symantic Product Name: Symantec Netbackup

Product Version: 7.0.1

Operating System: HP-UX 11iv3

[mph999]

Run bpdb -consistency 2

Might take some time - redirect the output to a file.

Martin

[A_M_Ash_]

Hi Martin ,

1- is the command " bpdb -consistency 2 "  impact any running backups as there is running backup now

2- is number 2 in command " bpdb -consistency 2 " indicate somthing

thanks

[A_M_Ash_]

hi,

also did you recommend to :

To initiate manual cleanup, perform the following:

# /usr/openv/netbackup/bin/admincmd/bpimage -cleanup -allclients
 

[Marianne]

Please show us all text in Job Details?

Do you see 'hung' bpdbm child process in bpps output?

Do you have bpdbm logging enabled?

[mph999]

It'll read the catalog, so no more effect than an image cleanup would have - just slow things down a bit, but you'll probably not notice.  You can always cancel it.

[mph999]

No I didn't ...  This runs as part of the image cleanup job anyway.  You can run it if you like, but it's not going to tell you much.

Martin

[A_M_Ash_]

hi,

1-i have initiated the command  bpdbm -consistency 2

2-kindly find attached screenshots of Job Details

3- how to know that bpdbm logging enabled

[Marianne]

You can actually copy text in Details tab - simply highlight everything, then do <Ctrl>-C on your keyboard to copy the text.
You can then paste the copied text in your forum post (<Ctrl>-V).

Can you see if cleanup is getting stuck on on particular client image? 
Is it the same client every time?
Perhaps GRT client?

You can see that the bpdbm process ID that is doinfg the cleanup is 10022.
Does bpps show bpdbm process with PID 10022?

To know if bpdbm logging is enabled, cd to /usr/openv/netbackup/logs on the master.

See if there is a bpdbm folder. 
If there is, you will find log files in this folder.
If not, create the folder. NBU needs to be restarted to enable this log.

[A_M_Ash_]

Hi ,

1- bpdb -consistency 2 is finished .. & redirect the output to a file...Now what to do or find in with this output

2-bpdbm logging is enabled

3- for  copy text in Details tab ...  ctrl -c & ctrl -v does not work with symantec

4-  cleanup is getting stuck on on particular client image,it is the image on master server every time

5-i have run the command  bpps -a & bpps -x ..  what you want to see from its output
 

[A_M_Ash_]

hi,

i just want to run this command to force NB to expire the image that must expire ...  is this command do that

# /usr/openv/netbackup/bin/admincmd/bpimage -cleanup -allclients
 

[Marianne]

Does PID 10022 show up in bpps output?

Can you extract all info in bpdbm log file for this PID? 
Try this:
grep "\[10022\]"  log.082112

This will hopefully tell us if cleanup is getting stuck on a specific image.

I agree with Martin - run 'bpdbm -consistency 2'

Extract from Commands Reference Guide:

-consistency [0-2]

Runs the consistency checks on the catalog. The following are the three

consistency levels:

0 - A quick check of the NetBackup image database (the default).

1 - Performs more checks than the default check.

2 - The most in-depth consistency check. In addition to the level 0 and 1

checks, this level checks that the media that is mentioned in the image exists.

(That is, it cross-references the media servers databases.) On a large

NetBackup installation, the process takes much longer to complete than the

other checks.

PS - I have just tried 'Ctrl-C' again in Details Tab of Java and Windows Admin Console - it works....

[mph999]

And there is the issue ...

"  cleanup is getting stuck on on particular client image,it is the image on master server every time"

Look for that image in the consistency output ...

Or even better, go find that image in the catalog, look at the header file

Martin

[mph999]

The image header file will be in ...

/usr/openv/netbackup/db/images/client/<ctime>/

Look for the files with the ctime in the filesname, the header file is the one that does not end with .f

(For <ctime> in dir path, it is the first four digits of the backup ctime followed by six 0s.)

Martin

[A_M_Ash_]

Hi,

-i want to clarify something that the importance of image cleanup is to free expired image from catalogue

is that right??

- this issue with image cleanup started 4 days ago

- after every backup the image cleanup process is initiated and take 2 days then failed

-i tried 'Ctrl-C' again in Details Tab of Java and Windows Admin Console ,, still not working

-PID 10022 didnot exist in bpps output and also all other PID of image cleanup processes

[Mark_Solutions]

You are right - the image cleanup exires and cleans up the catalog as welll as removing images from disk.

If there is a corrupt image that is stopping the process getting past a certain point then it will stop sussequent images from getting cleaned up.

Try running an All Log Entries report and sort by Severity - do this for the time period when the image cleanup fails - also useful for the period from a few minutes before midnight until an hour or so after midnight

From this we may be able to see which image is causing the issue

You can export this report out as a text file which you can attach on here

[mph999]

Did you follow my advice ...

And there is the issue ...

"  cleanup is getting stuck on on particular client image,it is the image on master server every time"

Look for that image in the consistency output ...

Or even better, go find that image in the catalog, look at the header file

The image header file will be in ...

/usr/openv/netbackup/db/images/client/<ctime>/

Look for the files with the ctime in the filesname, the header file is the one that does not end with .f

(For <ctime> in dir path, it is the first four digits of the backup ctime followed by six 0s.)

Martin

[A_M_Ash_]

Is there an alternative way to do the same task of image cleanup till we  repair the problem

[mph999]

No, no other way.

You need to look at the header file as I have suggested.  If necessary, you check every header file for the client in that 'ctime' directory.

Martin

[Mark_Solutions]

As you seem to know which client has the issue you could run a cleanup against all other clients manually at regular intervals

Create a script that calls a list of all of your clients (all the named under netbackup/db/images/ apart from the one having the issue) and then run the following against each of them:

bpimage -cleanup -client clientname

Hope this helps - once you have spotted the affected image in the All Log Entries report it should resolve it for you anyway once that one os manulay removed