11-21-2011 03:14 AM
Hi
We are using the following Environment : Netbackup Version : Netbackup 7.0.1 Master-Server : ( Solaris 10.0 ) on SUN-Hardware. Client : Windows Server 2003 Standard on Intel x86-Hardware
The Environment for storing the Catalogs is a Mounted Filesystem with an defined amount of space. The last time, we had problems, to handle NBU-Catalog; as it failed with the error
1: (87) media close error
or
1: (2) none of the requested files were backed up
To Clarify : The File-System hosts more data than this mounted NBU-Catalogs, so that I could not say, how much space is availiable at a special moment.
What I can see is, that from all failed Catalogs there remains *.info and *.img- Job-files for each Catalog.
Primarily the *.img-file growths to great dimensions. If the NBU-Catalog-Procedure failes more than one time the file system will be filled up.
What may I do against this ?
Thanks for your help
Thomas
Solved! Go to Solution.
11-23-2011 03:37 AM
Looks fine - I presume you've amended the FULL_Disk schedule to go to tape for the time being due to the issues you are currently encountering?
As I read it, what should be happening is a catalog backup is carried out daily @10.00 to disk & retained for 4 days. A further daily catalog is carried out to tape @16:00 & retained for a week.
DR file in both cases is copied to "/export/VERITAS_NetBackup" - is that where your catalog backups were going to also as it looks like there could be both in the output you provided earlier?
I agree with Marianne in her post "This DSU should be a dedicated lun/filesystem and not shared with anything else."
If your DR files (e.g. NBU-Catalog_12345678910_FULL etc etc) are also going there they will probably need to be cleaned up manually. As you have a retention of 4 days for the disk backups you will need at the very least 5x the size of your catalog backup to ensure they'll complete successfully.
***EDIT***
Have you got some RDP session open to my PC Mark or is there a hidden camera somewhere!!!
11-21-2011 03:48 AM
It seems that you are referring to images written by Disk Storage Units and not NetBackup catalogs (stored on the master under /usr/openv/netbackup/db/images)?
Disk Storage Units should OWN the entire filesystem where the DSU resides. It should NOT share the filesystem with any other data.
Schedule disk staging to tape often to free up space. Otherwise backup images will remain on disk until they naturally expire.
Carefully go through this section in NBU Admin Guide I:
Disk staging storage unit size and capacity
11-21-2011 03:50 AM
Hi,
When you write "The Environment for storing the Catalogs is a Mounted Filesystem", do you mean it's a filesystem for your offline catlog backups ?
And could you tell the size of your catalog ?
11-21-2011 05:54 AM
Hi Frederic,
As i understand the configuration of our system, my colleagues has set up two forms ob Scripts. One time a copy of the NBU-Catalog will be written to disk and the other time directly to tape.
The Scrpt itsself was was - as i knew - set up with a previous version of Netbackup.
Peak to peak Values for the Files differs between 86 and 96 GB ( If there is enough space to store it. Here are some Examples :
= > Failed
11-21-2011 06:07 AM
Seems you have a hot catalog backup that is backing up to DSU.
This DSU should be a dedicated lun/filesystem and not shared with anything else.
If this is not possible, backup the NBU catalogs to tape only or else find alternative disk space that can be dedicate for exclusive use by the DSU.
No need to use scripts - use the GUI to configure the Hot Catalog policy.
11-21-2011 06:51 AM
Ok. I've the right policy. And there are two shedules within. . Whatabout i wonder, is the Retention-time. It Is set to 4 Days and i have a long list of
NBU-Catalog_Files beginning from Aug 30. Is there any Logic within, or ar are these the rests of the broken jobs ?
11-21-2011 07:19 AM
The files you are mentioning sound like they are the DR information files - the setting for these is found on the Disater Recovery tab - these are just small files that hold the information needed to do a catalog restore.
The actual catalog backup will be directed to the storage unit defined in the Attributes tab of the policy or this may be over ridden by a setting in a schedule
If the catalog backup is failing it is this storage unit you need to look at
Hope this helps
11-22-2011 06:52 AM
Ok. Looking deeper to the Protokolls offers some more detailed information. It seems to me, that the Retention Time for the NBU-Catalog-Files respectively the DR-files will not be observed.
About which Time Period should these DR-files be retained ?
Kind regards
Thomas
11-22-2011 08:56 AM
Hi Thomas
One thing to note is that the catalog backups have changed across Netbackup versions - netbackup 5.x supported only hot backups, 6.x supported hot and cold, and 7.x supports only hot backups. It was quite typical to use scripts to schedule cold backups in the past but seems somewhat redundant these days as hot backups are scheduled via Policy (NBU_Catalog type). I mention this as it ties in with your comments that colleagues set up scripts in the past with a previous version of Netbackup however this is unlikely to be causing your issue.
With the old cold catalog backups you used to be able to configure them to work to 2 separate locations on the same schedule, often disk and tape. It's possible that your colleagues set up a script to initiate different catalog backup policies or modify a hot backup policy to change it's storage policy on the fly but this is unlikely as you can now simply use 2 schedules which you did mention and get one to override policy storage selection.
If there are 2 schedules to write to 2 separate locations double-check the retention of both schedules is correct - it's easy to overlook a difference between them. Also check whether the Multiple Copies tick-box is selected in one of the schedules - if it is click the Configure box besides that check box and check what retention period is set there as this will not be obvious from the initial view of policies.
Don't be confused between the DR files and the Catalog backup files - the ones you listed in the earlier post which seem to be causing your space issues are going to be the actual catalog backup files - the DR files are pretty small and you can email these to yourself rather than have them on the backup server if you wish - I do this and archive them within exchange so I have a good long-term view of the catalog backup details away from the backup server.
Best of luck,
Nick.
11-23-2011 02:20 AM
Hi Nick,
To reply to your information, i will amend some details from the Policy : There exists one Policy with two Schedules. On one of this Schedule the Settings will be overwritten, so that the Catalog-Files are only written to disk. On the other Schedule, the Setting is overwritten, so that the Catalog-Filess will be written to tape ( At the moment, it is not clear to me wether the file are intermittend written to to the Storage on the Disk before they are written to Tape ).
What is absolutely confusing me is the following :
Retention Period : 4 Days
Retention Level : 11
In my understanding, Retention Level 11 is one of the "Never Expire" Levels ? Is this correct , and what would this mean ?
Thanks
Thomas
11-23-2011 02:40 AM
can you provide the output from:
bppllist your_catalog_policy -U
As far as the retention periods/levels are concerned ALL retention levels can be customised (except infinity level 9) to suit your needs. Someone has obviously changed level 11 to 4 days to suit business requirements. These levels can be seen/amended via the GUI Host Properties/Master Servers/master_server/Retention Periods tab.
11-23-2011 03:13 AM
11-23-2011 03:36 AM
Ok - looking at this you run a Full catalog backup to tape every afternoon and a full catalog back to disk every morning
Tape is kept for 1 week, disk is kept for 4 days
The catalog DR file is always written to /export/Veritas_NetBackup and in view of the name of the disk storage unit used i get the feeling that this is actually the same path as used for the disk storage unit
I would reccomend changing this so that the DR files go to a specific location and not to this disk storage unit area
I do not believe that there is a specific retention on the DR files - as far as i know they do not expire or get deleted so are kept forever. As they are only a few kb in size this is not generally considered an issue and in your case you are safe to delete any older than 8 days anyway
The main confusion here comes from the disk storage unit and the dr file path being one and the same file system location
Hope this helps
11-23-2011 03:37 AM
Looks fine - I presume you've amended the FULL_Disk schedule to go to tape for the time being due to the issues you are currently encountering?
As I read it, what should be happening is a catalog backup is carried out daily @10.00 to disk & retained for 4 days. A further daily catalog is carried out to tape @16:00 & retained for a week.
DR file in both cases is copied to "/export/VERITAS_NetBackup" - is that where your catalog backups were going to also as it looks like there could be both in the output you provided earlier?
I agree with Marianne in her post "This DSU should be a dedicated lun/filesystem and not shared with anything else."
If your DR files (e.g. NBU-Catalog_12345678910_FULL etc etc) are also going there they will probably need to be cleaned up manually. As you have a retention of 4 days for the disk backups you will need at the very least 5x the size of your catalog backup to ensure they'll complete successfully.
***EDIT***
Have you got some RDP session open to my PC Mark or is there a hidden camera somewhere!!!
11-23-2011 04:21 AM
Mark, Andy - GREAT minds think alike!!
11-23-2011 05:44 AM
Thats right. What i presume is, that Files, for which the Retention-Time was expired were marked for Deletion or directly deleted. Are the _C1_HDR and _C1_F1 Files from the following Excerpt "Zombies" ? They are older than than the the Retention Time ? :
Catalog Backup Status
the requested operation was successfully completed (status 0).
11-23-2011 05:53 AM
This command should cleanup the image fragments:
/usr/openv/netbackup/bin/admincmd/nbdelete -allvolumes -force
The command is safe and will not delete any valid backups.
Please create 'admin' and 'bpdm' folders on the master under /usr/openv/netbackup/logs before you run this command.
Cleanup attempts/actions will be logged in these 2 folders.
11-23-2011 07:02 AM
@Mark and Andy
It is very sophisticated, to decide, which to mark answer as solution, if both are so nearly congruent with.
I keep in mind to arrange something with my collegue to set up a routine, what to do with the Dedicated Lun/File-System and the manually Cleanup of the DR-Files .
@Marianne
Thank you for your tip with the nbdelete-command. It does not show any effect. I cleaned up the files manually.
Kind regards
Thomas