Problems with NBU_Catalog

[Stormchaser]

Hi

We are using the following Environment : Netbackup Version : Netbackup 7.0.1 Master-Server :   ( Solaris 10.0 ) on SUN-Hardware. Client : Windows Server 2003 Standard on Intel x86-Hardware

The Environment for storing the Catalogs is a Mounted Filesystem  with an defined amount of space. The last time, we had problems, to handle NBU-Catalog; as it failed with the error

1: (87) media close error 

or

1: (2) none of the requested files were backed up 

To Clarify : The File-System hosts more data than this mounted NBU-Catalogs, so that I could not say, how much space is availiable at a special moment.

What I can see is, that from all failed Catalogs there remains *.info and *.img- Job-files for each Catalog.

Primarily the *.img-file growths to great dimensions. If the NBU-Catalog-Procedure failes more than one time the file system will be filled up.

What may I do against this ?

Thanks for your help

Thomas

[Marianne]

It seems that you are referring to images written by Disk Storage Units and not NetBackup catalogs (stored on the master under /usr/openv/netbackup/db/images)?

Disk Storage Units should OWN the entire filesystem where the DSU resides. It should NOT share the filesystem with any other data.

Schedule disk staging to tape often to free up space. Otherwise backup images will remain on disk until they naturally expire.

Carefully go through this section in NBU Admin Guide I:

Disk staging storage unit size and capacity

[Frederic_GERARD]

Hi,

When you write "The Environment for storing the Catalogs is a Mounted Filesystem", do you mean it's a filesystem for your offline catlog backups ?

And could you tell the size of your catalog ?

[Stormchaser]

Hi Frederic,

As i understand the configuration of our system, my colleagues has set up two forms ob Scripts. One time a copy of the NBU-Catalog will be written to disk and the other time directly to tape.

The Scrpt itsself was was - as i knew - set up with a  previous version of Netbackup.

Peak to peak Values for the Files differs between 86 and 96 GB ( If there is enough space to store it.  Here are some Examples :

-rw-------   1 root     root     250904576 Nov 19 10:49 [servername]_1321696144_C1_F1.1321696144.img

-rw-------   1 root     root        1024 Nov 19 10:49 [servername]_1321696144_C1_F1.1321696144.info

= > Failed

-rw-------   1 root     root        1024 Nov 21 10:43 [servername]_1321866065_C1_F1.1321866065.info

-rw-------   1 root     root     4688896 Nov 21 10:43 [servername]_1321866065_C1_TIR.1321866065.img

-rw-------   1 root     root        1024 Nov 21 10:43[servername]_1321866065_C1_TIR.1321866065.info

-rw-------   1 root     root        1953 Nov 21 10:43 NBU-Catalog_1321866065_FULL

= >Successfully Completed

[Marianne]

Seems you have a hot catalog backup that is backing up to DSU.

This DSU should be a dedicated lun/filesystem and not shared with anything else.

If this is not possible, backup the NBU catalogs to tape only or else find alternative disk space that can be dedicate for exclusive use by the DSU.

No need to use scripts - use the GUI to configure the Hot Catalog policy.

[Stormchaser]

Ok. I've the right policy.  And there are two shedules within. . Whatabout i wonder, is the Retention-time. It Is set to 4 Days and i  have a long list of

NBU-Catalog_Files beginning from Aug 30. Is there any Logic within, or ar are these the rests of the broken jobs ?

Kind Regards

Thomas

[Mark_Solutions]

The files you are mentioning sound like they are the DR information files - the setting for these is found on the Disater Recovery tab - these are just small files that hold the information needed to do a catalog restore.

The actual catalog backup will be directed to the storage unit defined in the Attributes tab of the policy or this may be over ridden by a setting in a schedule

If the catalog backup is failing it is this storage unit you need to look at

Hope this helps

[Stormchaser]

Ok.  Looking deeper to the Protokolls offers some more detailed information. It seems to me, that the Retention Time for the NBU-Catalog-Files respectively the DR-files will not be observed. 

About which Time Period should these DR-files  be retained ?

Kind regards

Thomas

[Nick_M]

Hi Thomas

One thing to note is that the catalog backups have changed across Netbackup versions - netbackup 5.x supported only hot backups, 6.x supported hot and cold, and 7.x supports only hot backups.  It was quite typical to use scripts to schedule cold backups in the past but seems somewhat redundant these days as hot backups are scheduled via Policy (NBU_Catalog type).  I mention this as it ties in with your comments that colleagues set up scripts in the past with a previous version of Netbackup however this is unlikely to be causing your issue.

With the old cold catalog backups you used to be able to configure them to work to 2 separate locations on the same schedule, often disk and tape.  It's possible that your colleagues set up a script to initiate different catalog backup policies or modify a hot backup policy to change it's storage policy on the fly but this is unlikely as you can now simply use 2 schedules which you did mention and get one to override policy storage selection.

If there are 2 schedules to write to 2 separate locations double-check the retention of both schedules is correct - it's easy to overlook a difference between them.  Also check whether the Multiple Copies tick-box is selected in one of the schedules - if it is click the Configure box besides that check box and check what retention period is set there as this will not be obvious from the initial view of policies.

Don't be confused between the DR files and the Catalog backup files - the ones you listed in the earlier post which seem to be causing your space issues are going to be the actual catalog backup files - the DR files are pretty small and you can email these to yourself rather than have them on the backup server if you wish - I do this and archive them within exchange so I have a good long-term view of the catalog backup details away from the backup server.

Best of luck,

  Nick.

[Stormchaser]

Hi Nick,

To reply to your information, i will amend some details from the Policy : There exists one Policy with two Schedules. On one of this Schedule the Settings will be overwritten, so that the Catalog-Files are only written to disk. On the other Schedule, the Setting is overwritten, so that the Catalog-Filess will be  written to tape ( At the moment, it is not clear to me wether the file are intermittend written to to the Storage on the Disk before they are written to Tape ).

What is absolutely confusing me is  the following :

Retention Period :  4 Days

Retention Level  :   11

In my understanding, Retention Level 11 is one of the "Never Expire" Levels ?  Is this correct , and what would this mean ?

Thanks

Thomas

[Andy_Welburn]

can you provide the output from:

bppllist your_catalog_policy -U

As far as the retention periods/levels are concerned ALL retention levels can be customised (except infinity level 9) to suit your needs. Someone has obviously changed level 11 to 4 days to suit business requirements. These levels can be seen/amended via the GUI Host Properties/Master Servers/master_server/Retention Periods tab.

[Stormchaser]

Yes, it seems to me, that the Retention Level was set earlier from my collegue. The Policy was updated with the last Software-Upgrade from 6.5 to 7.0 .

Here is the Output from bpplist :

root@ [servername] # bppllist NBU-Catalog -U

------------------------------------------------------------

Policy Name:       NBU-Catalog

  Policy Type:         NBU-Catalog

  Active:              yes

  Effective date:      08/25/2010 15:45:07

  Mult. Data Streams:  no

  Client Encrypt:      no

  Checkpoint:          no

  Policy Priority:     0

  Max Jobs/Policy:     1

  Disaster Recovery:   0

  Collect BMR info:    no

  Residence:           SL500

  Volume Pool:         CatalogBackup

  Server Group:        *ANY*

  Keyword:             NBU Catalog

  Data Classification:       -

  Residence is Storage Lifecycle Policy:    no

Granular Restore Info:  no

Ignore Client Direct:  no

  HW/OS/Client:  Solaris       SunOS         [servername]

  Include:  CATALOG_DRIVEN_BACKUP

  Schedule:          FULL_Disk

    Type:            Full Backup

    Frequency:       every 1 day

    Maximum MPX:     1

    Synthetic:       0

    PFI Recovery:    0

    Retention Level: 11 (4 days)

    Number Copies:   1

    Fail on Error:   0

    Residence:       VERITAS_NetBackup

    Volume Pool:     (same as policy volume pool)

    Server Group:    (same as specified for policy)

    Residence is Storage Lifecycle Policy:     0

    Daily Windows:

          Sunday     10:00:00  -->  Sunday     12:00:00

          Monday     10:00:00  -->  Monday     12:00:00

          Tuesday    10:00:00  -->  Tuesday    12:00:00

          Wednesday  10:00:00  -->  Wednesday  12:00:00

          Thursday   10:00:00  -->  Thursday   12:00:00

          Friday     10:00:00  -->  Friday     12:00:00

          Saturday   10:00:00  -->  Saturday   12:00:00

  Schedule:          Full_Tape

    Type:            Full Backup

    Frequency:       every 1 day

    Maximum MPX:     1

    Synthetic:       0

    PFI Recovery:    0

    Retention Level: 0 (1 week)

    Number Copies:   1

    Fail on Error:   0

    Residence:       SL500

    Volume Pool:     CatalogBackup

    Server Group:    (same as specified for policy)

    Residence is Storage Lifecycle Policy:     0

    Daily Windows:

          Sunday     16:00:00  -->  Sunday     18:00:00

          Monday     16:00:00  -->  Monday     18:00:00

          Tuesday    16:00:00  -->  Tuesday    18:00:00

          Wednesday  16:00:00  -->  Wednesday  18:00:00

          Thursday   16:00:00  -->  Thursday   18:00:00

          Friday     16:00:00  -->  Friday     18:00:00

          Saturday   16:00:00  -->  Saturday   18:00:00

Catalog Disaster Recovery Configuration:

  Email Address:  admin1@domain, admin2@domain

  Disk Path:       /export/VERITAS_NetBackup

  User Name:       (none specified)

  Pass Word:       (none specified)

  Critical policy: (none specified)

root@ [servername] #

Thanks Thomas

[Mark_Solutions]

Ok - looking at this you run a Full catalog backup to tape every afternoon and a full catalog back to disk every morning

Tape is kept for 1 week, disk is kept for 4 days

The catalog DR file is always written to /export/Veritas_NetBackup and in view of the name of the disk storage unit used i get the feeling that this is actually the same path as used for the disk storage unit

I would reccomend changing this so that the DR files go to a specific location and not to this disk storage unit area

I do not believe that there is a specific retention on the DR files - as far as i know they do not expire or get deleted so are kept forever. As they are only a few kb in size this is not generally considered an issue and in your case you are safe to delete any older than 8 days anyway

The main confusion here comes from the disk storage unit and the dr file path being one and the same file system location

Hope this helps

[Andy_Welburn · Accepted solution]

Looks fine - I presume you've amended the FULL_Disk schedule to go to tape for the time being due to the issues you are currently encountering?

As I read it, what should be happening is a catalog backup is carried out daily @10.00 to disk & retained for 4 days. A further daily catalog is carried out to tape @16:00 & retained for a week.

DR file in both cases is copied to "/export/VERITAS_NetBackup" - is that where your catalog backups were going to also as it looks like there could be both in the output you provided earlier?

I agree with Marianne in her post "This DSU should be a dedicated lun/filesystem and not shared with anything else."

If your DR files (e.g. NBU-Catalog_12345678910_FULL etc etc) are also going there they will probably need to be cleaned up manually. As you have a retention of 4 days for the disk backups you will need at the very least 5x the size of your catalog backup to ensure they'll complete successfully.

***EDIT***

Have you got some RDP session open to my PC Mark or is there a hidden camera somewhere!!!

[Marianne]

Mark, Andy - GREAT minds think alike!!

[Stormchaser]

Thats right. What i presume is, that Files, for which the Retention-Time was expired were marked for Deletion or directly deleted. Are the _C1_HDR and _C1_F1 Files from the following Excerpt "Zombies" ? They are older than than the the Retention Time ? :

-rw-------   1 root     root        1024 Nov  7 10:00 [servername]_1320656418_C1_HDR.1320656418.info

-rw-------   1 root     root         512 Nov  7 10:00 [servername]_1320656418_C1_HDR.1320656418.img

-rw-------   1 root     root     291766272 Nov  7 10:00 [servername]_1320656418_C1_F1.1320656418.img

-rw-------   1 root     root        1024 Nov  7 10:00 [servername]_1320656418_C1_F1.1320656418.info

-rw-------   1 root     root        1024 Nov  7 10:01 [servername]_1320656460_C1_HDR.1320656460.info

-rw-------   1 root     root         512 Nov  7 10:01 [servername]_1320656460_C1_HDR.1320656460.img

-rw-------   1 root     root     95478120448 Nov  7 10:33 [servername]_1320656460_C1_F1.1320656460.img

-rw-------   1 root     root        1024 Nov  7 10:33 [servername]_1320656460_C1_F1.1320656460.info

-rw-------   1 root     root        1024 Nov  7 10:33 [servername]_1320656460_C1_TIR.1320656460.info

-rw-------   1 root     root     5109248 Nov  7 10:33 [servername]_1320656460_C1_TIR.1320656460.img

-rw-------   1 root     root        1954 Nov  7 10:33 NBU-Catalog_1320656460_FULL

Catalog Backup Status

      the requested operation was successfully completed (status 0).

[Marianne]

This command should cleanup the image fragments:

/usr/openv/netbackup/bin/admincmd/nbdelete -allvolumes -force

The command is safe and will not delete any valid backups.

Please create 'admin' and 'bpdm' folders on the master under /usr/openv/netbackup/logs before you run this command.

Cleanup attempts/actions will be logged in these 2 folders.

[Stormchaser]

@Mark and Andy 

It is very sophisticated, to decide, which to mark answer as solution, if both are so nearly congruent with.

I keep in mind to arrange something with my collegue to set up a routine, what to do with the Dedicated Lun/File-System and the manually Cleanup of the DR-Files .

@Marianne

Thank you for your tip with the nbdelete-command. It does not show any effect. I cleaned up the files manually.

Kind regards

Thomas