cancel
Showing results for 
Search instead for 
Did you mean: 

bpimagelist results are not complete. (missing entries)

Tape_Robot
Level 3

My setup:

I am using Netbackup 6.5 running on an AIX master server. I am running bpimagelist on the backup server directly (local query). I am using a Windows version of Netbackup 6.5 to issue restore commands from the Backup, Archive, and Restore GUI program. 

My goal:

I want to make a list of all the tapes needed to restore every bit of data ever backed up from a server $1. I also need the date correlation to the media ID.

My method:

bpimagelist -media -U -client $1 -d 01/01/1970 > tapelist.txt

($1 = the name of the server that was backed up)

The problem:

At first glance the data appears to be what I want. So I have my list and I open up the GUI "Backup, Archive and Restore" because this is what I am using to restore said data. There is a clear relationship between the green check marks, the dates above them and the Media ID reported by "Preview Media" and the results of my bpimagelist command. However there are a significant number of backups available to restore within the GUI that are not on the list created by the bpimagelist command. There are no inconsistencies with the data that the two lists share but they are not identical in that the GUI has more total backups than the command line admits to. The missing entries are not at the begining or the end but they are randomly missing from the middle. (ex. day 1, 2, 3, 5, 7, 8, 10)

The Question:

Can anyone tell me why my bpimagelist command would return an incomplete list of days and tapes? I promise that all of the backups occured after 1970 so they should all be within the scope of my search, right?

This kind of inconsistancy completely destroys my faith in this application. How can I count on this software to protect my data?

1 ACCEPTED SOLUTION

Accepted Solutions

mph999
Level 6
Employee Accredited

Here you are ...

Not the neastest way, but need to dash out ...

 

bpmedialist -l |awk '{print $1}' |while read TAPE
do
bpimmedia -mediaid $TAPE |grep IMAGE |awk '{print $4}' |while read LINE
do
DATE=$(echo $LINE |awk -F_ '{print $2}' |awk '{print $1}')
REALDATE=$(bpdbm -ctime $DATE |awk -F= '{print $2}')
echo $TAPE $LINE $REALDATE
done
done

 

Gives this output ...

TAPE04   womble_1305812840    Thu May 19 14:47:20 2011
TAPE04   womble_1305811981    Thu May 19 14:33:01 2011
TAPE05   womble_1305814567    Thu May 19 15:16:07 2011
TAPE05   womble_1305814384    Thu May 19 15:13:04 2011

 

Martin

View solution in original post

15 REPLIES 15

Glo
Level 4

Try the -idonly option instead of the -media option if you want to compare bpmedialist with the BAR GUI.

Tape_Robot
Level 3

Thanks for your help and prompt reply!

It's very interesting but not the output I am looking for. That does give me a list of dates that appears to include the dates missing from my first attempt. It does not give me a correlating Media ID to match up with that date.

What I want is very simple. I need to know what tape to put in the robot for a range of dates. Currently my scope is every tape ever written for this server and a record of what that covers. Traceability. This tape was made on this day. There should be a nice auto-generating report for something as common as this. Don't we all want to have a list of what tapes were made on what day? Even if you have a hand written log, I would want to correlate my log with what is actually in the Netbackup catalog. Wouldn't you? Verification? Traceability? Isn't this an important feature?

This also doesn't explain why some of the dates are missing from the -media command.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

I am wondering if the GUI is probably Case INsensitive where cmd is definately case sensitive.

Check the /usr/openv/netbackup/db/images to see if there is more than one folder name for the client.

Andy_Welburn
Level 6

an incomplete list of days and tapes?"

Is it just the fact that the bpimagelist is showing the LAST time it was used?

e.g.

Media ID   Last Written      
--------   ----------------   ----------
300176     05/18/2011 18:33  
300577     05/18/2011 18:00  
300651     05/18/2011 08:00  
300114     05/17/2011 18:30  
300112     05/17/2011 08:00  

from the above output 300651 is "apparently" missing from the output for 17th May as it was also used on the 18th. GUI shows correct tapes for 18th as above and also 300112, 300114 and 300651 for the 17th.

Or have you already taken that into account?

mph999
Level 6
Employee Accredited

Andy is correct,

I cannot see any inconsistancy in the application.  I have tested this left, right and centre today.  I can however see that the command, although I believe it is  workling correctly, is not meeting your expectations. 

So, how about this ...The way I think to achive what you want ...

 

bpmedialist -l |awk '{print $1}' |while read LINE
do
echo $LINE
bpimmedia -mediaid $LINE |grep IMAGE |awk '{print $4}'
done
 

Giving me the correct results ...

TAPE04
womble_1305812840
womble_1305811981
TAPE05
womble_1305814567
womble_1305814384

Martin

Tape_Robot
Level 3

This is a good explanation and is consistent with the entries I am missing. However this behaviour is certainly undesireable. The information written to that tape on the different days could be completely different. My understanding of the command is that it should look through all images and report Media IDs and dates within the range specified but that would be a literal interpretation of the command syntax. Which is foolish I suppose.

I want all entries. Nothing should be omitted.   

Andy_Welburn
Level 6

is to run the command over several iterations for each individual day, but that could also be undesirable if you're looking at a long period of time!!

I did have a script that did something similar, have to check when I'm back at work tomorrow (it probably does contain the odd 'awk' tho'!)

Again, as I'm not at work & can't remember the exact format of them, are there any of the "Reports" that cover what you want? They can be exported to a s/sheet....I think, or is that somewhere else?

....hmmm, could be....

from

DOCUMENTATION: The command line equivalents for running the NetBackup reports available in the NetBackup user interface
http://www.symantec.com/business/support/index?page=content&id=TECH20462

Media Written Report
The Media Written report identifies volumes that were used for backups within the specified time period.  
# cd /usr/openv/netbackup/bin/admincmd
# ./bpimagelist -A -media [-d <start_date> <start_time> -e <end_date> <end_time>]

no apparent client option here, but maybe there is a drop-down available on the GUI. Maybe worth checking all the media-type reports out?

mph999
Level 6
Employee Accredited

What about OpsCenter - this is the 'NetBackup Reporting tool' - I cannot say I'm an expert in OpsCenter, but I'm goig to be very very brave and suggest that I would image it can do what you want.

Certaining, OpsCenter with Analytics (which is the bit that you pay for) is very powerful with the ability to create custom SQL reports.  Although charable, this is in line with other vendors.

The base OpsCenter is free.

Martin

Tape_Robot
Level 3

Okay so,

There are 3 ways I have used so far to get information about what the catalog has availble to restore. All of them give slightly different results.

1.  If you go to the BAR GUI you can click on each image in each day one by one and select "preview media" and it will show you the tapes you need but you can't copy that data out to text. If you select multiple images by holding shift then the media ID shown in the preview is often the last one you clicked on and does not show all the tapes required to actually restore the data you selected. This is not only inefficient but completely impractical once you have run over 800 backups. However, I have always assumed that the list of available backups here is complete. Until now.

2. You can use the various outputs from bpimagelist to get different degrees of completeness, some of which are compatible with a database and some of which would require extensive reformatting to be usefull for any sort of records keeping. Information about tapes that have been written to multiple times may be missing.

3. (Admin Console) NetBackup Management > Reports > Tape Reports > Images on tape. This is where it gets strange. If you query a server name and leave the media ID blank you get a good combination of customizable data regarding backups in the catalog. This is the closest to being useful but raises more questions. It is a far more complete list than using #2 (bpimagelist) however there are some inconsistancies again. There are entries that bpimagelist shows that are not on the images on tape list and that is disturbing.

What is far worse is that there is one entry that shows up in the "images on tape" list that is not available from the BAR GUI or bpimagelist. This is unacceptable because if you were using the GUI you would not be able to recover this data. This image is not expired and should be the same Policy type as the others.

On a related note. There is a column for "policy type" in the images on tape list that should correspond to the "policy type" dropdown selector in the BAR GUI. This feild is blank. The policy type is Lotus Notes but you couldn't tell that from this report. Just blank. It was a good idea but nobody bothered to make it work.

The bottom line is that none of these methods can agree with eachother about what tapes and what days are available. Making tapes is great but eventually someone will want to get the data off of the tapes, right? That is why we make them. Perhaps 100% accountability and traceability is asking too much. I know, I am a pickey bastard and I complain too much.

Tape_Robot
Level 3

I don't speak awk. Perhaps I should. The output I really need is just like the output that the command I issued gives, except complete. I need the Media ID in one column and the Date right next to it, in another column. This is functional to restoration. What tapes do you need to pull for what days? There are ways I can do this with the output from multiple commands and lots of scripting but it is rediculous that I should have to. Especially when the output from the existing commands is so close. bpimagelist has access to all the information I need but it simply won't display it all at the same time. It's frustrating. We pay good money for a backup solution and in order to provide simple data traceability the answer I keep finding is "Write some code to do that". Isn't that what we pay Symantec to do? Write the code?

The information in the BAR GUI? Yeah, I want a text version of it for records that would import easily in to a proper database(or spreadsheet) because the program itself is lacking a proper database. Seems so simple.

Tape_Robot
Level 3

I'm sure it could do exactly what I am asking for if I were running Netbackup 7 (and possibly show a 4th different set of data). We actually own copies of NB7 and are running 7 on few of the servers here but I am working with this old Lotus Notes AIX server that doesn't seem to play well with NB7. I need to get all of it's data out of it so I can retire the server itself. We are talking about years of mail backups that we are on legal obligation to preserve. Due to court orders I need to provide good end-to-end data traceability to prove that I have not lost any data. The server itself may not live long enough to see the end of the lawsuit but the data it created needs to. Even if I am to use a different server to restore this data I would still like the old one to tell me what it thinks it has in the catalog.

mph999
Level 6
Employee Accredited

Here you are ...

Not the neastest way, but need to dash out ...

 

bpmedialist -l |awk '{print $1}' |while read TAPE
do
bpimmedia -mediaid $TAPE |grep IMAGE |awk '{print $4}' |while read LINE
do
DATE=$(echo $LINE |awk -F_ '{print $2}' |awk '{print $1}')
REALDATE=$(bpdbm -ctime $DATE |awk -F= '{print $2}')
echo $TAPE $LINE $REALDATE
done
done

 

Gives this output ...

TAPE04   womble_1305812840    Thu May 19 14:47:20 2011
TAPE04   womble_1305811981    Thu May 19 14:33:01 2011
TAPE05   womble_1305814567    Thu May 19 15:16:07 2011
TAPE05   womble_1305814384    Thu May 19 15:13:04 2011

 

Martin

Tape_Robot
Level 3

Martin wins!

Well done sir! Your Kung Fu is strong. I certainly should be able to get what I need out of this script. This awkward jumble of awk commands is crippling to my server. It cries blood as it displays one line every 2 seconds. Perhaps it is the nested "do". . .

I will continue to hate Netbackup but this Martin guy is alright. Perhaps Symantec should pay him to stop by with some Industrial strength turd polish and take a look at their code. Worst job ever.

Thank you all for playing. See you next time.

mph999
Level 6
Employee Accredited

I needed the nested do, as the grep to get the backupid display multiple lines at once, and I needed to process them individually.

The killer is probably the bpdbm -ctime - perhaps best not to run if the server is busy.

Thank you for your kind comments.

Martin .

Andy_Welburn
Level 6

as Martin's done all the leg work! yes