Catalog Backup Failing with 2,50 and 96

[skrp]

Hi All,

In our environment, cummulative incremental backup for catalog is failing with 2, 50 and 96. But the full backup is completing successfully, do we have any problem if the catalog incremental backup fails continuously as it has been failing since 4 days.

Master server is itself the media server and using netbackup 7.6.1
It is a windows 7 server.

Can you please help me in getting a solution for this.

Parent Job Details:
10/26/2015 4:03:31 PM - Info nbjm(pid=4620) starting backup job (jobid=220226) for client kthydbak101.kt.group.local, policy Pol_Catalog, schedule Cummulative-Incremental 
10/26/2015 4:03:31 PM - Info nbjm(pid=4620) requesting CATALOG_BACKUP_RESOURCE resources from RB for backup job (jobid=220226, request id:{E358B87F-8B22-4385-A642-31B75BFBFC30}) 
10/26/2015 4:03:31 PM - requesting resource kthydbak101.kt.group.local.NBU_CATALOG.MAXJOBS
10/26/2015 4:03:31 PM - granted resource kthydbak101.kt.group.local.NBU_CATALOG.MAXJOBS
10/26/2015 4:03:32 PM - estimated 0 Kbytes needed
10/26/2015 4:03:32 PM - begin Parent Job
10/26/2015 4:03:32 PM - begin Catalog Backup, Start Notify Script
10/26/2015 4:03:32 PM - Info RUNCMD(pid=22656) started           
10/26/2015 4:03:32 PM - Info RUNCMD(pid=22656) exiting with status: 0        
Status 0
10/26/2015 4:03:32 PM - end Catalog Backup, Start Notify Script; elapsed time: 0:00:00
10/26/2015 4:03:32 PM - begin Catalog Backup, DBM Query

Child Job Details:
10/26/2015 4:03:32 PM - Info bpdbm(pid=14680) staging relational database files for catalog backup     
10/26/2015 4:03:32 PM - Info bpdbm(pid=14680) staging NBAZDB backup to E:\Program Files\Veritas\NetBackupDB\staging      
10/26/2015 4:03:32 PM - Info bpdbm(pid=14680) done staging NBAZDB backup to E:\Program Files\Veritas\NetBackupDB\staging     
10/26/2015 4:03:32 PM - Info bpdbm(pid=14680) staging NBDB backup to E:\Program Files\Veritas\NetBackupDB\staging      
10/26/2015 4:03:34 PM - Info bpdbm(pid=14680) done staging NBDB backup to E:\Program Files\Veritas\NetBackupDB\staging     

Second Child Job:
10/26/2015 4:03:35 PM - Info nbjm(pid=4620) starting backup job (jobid=220228) for client kthydbak101.kt.group.local, policy Pol_Catalog, schedule Cummulative-Incremental 
10/26/2015 4:03:35 PM - Info nbjm(pid=4620) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=220228, request id:{3779599E-D83D-41DE-8B53-7AA580A2B8A8}) 
10/26/2015 4:03:35 PM - requesting resource kthydbak101-hcart-robot-tld-0
10/26/2015 4:03:35 PM - requesting resource kthydbak101.kt.group.local.NBU_CLIENT.MAXJOBS.kthydbak101.kt.group.local
10/26/2015 4:03:35 PM - requesting resource kthydbak101.kt.group.local.NBU_POLICY.MAXJOBS.Pol_Catalog
10/26/2015 4:03:35 PM - Error nbjm(pid=4620) NBU status: 96, EMM status: No media is available   
unable to allocate new media for backup, storage unit has none available (96)

[Marianne]

Firstly - NBU is not supported as master/media server on Windows 7 - only as client.

You need to have Win2008 or Win2012 as master/media server.

See the OS SCL: http://www.veritas.com/docs/000040842

About the Catalog Backup failure:

You need a minimum of 2 tapes in CatalogBackup pool in order to perform Full and Incremental backups with different retentions.

NetBackup does not mix retentions on the same tape.
It can be configured but is NOT recommended.

[skrp]

Hi Marianne,

But as of now we only have one catalog tape. What are we supposed to do now.

[Marianne]

The choice is 100% yours:

1. Get another tape

2. Change retention of incremental to the same as full backup so that the same tape can be used for all backups.

[mph999]

Just run a full catalog backup ....

Is this really really true ....

Master server is itself the media server and using netbackup 7.6.1
It is a windows 7 server.

If so, as the wonderful Marianne has pointed out it is not supported.  OK, it might work, or work so far, but if you hit any issues, for example can't restore something ....  you're going to have a very big problem.

I'll put it directly, it is madness to run a master server in an unsupported configeration.

It needs moviong to a supported OS, quickly.

[skrp]

Hi All,

I'm very sorry it is a Windows 2008 server. But yes we are running both master and media on the same server.

But I had no issues running the incremental the full catalog onto the same tape earlier. We've had a similar problem earlier with one tape which was used for more than three months, so had replaced it with a new tape which worked fine for one month and now suddenly have this problem.

[revarooo]

skrp, show us the policy info for your catalog backup: bppllist <policyname> -U

Also run netbackup\bin\goodies\available_media from the cmd line and post the contents.

[Marianne]

I agree with above request. 

The output will help us understand why you are getting status 96 for the Incr catalog backup.

bppllist is in ..netbackup\bin\admincmd

Please also run this command for the catalog backup tape (command is also in admincmd) and post the text output:

nbemmcmd -listmedia -mediaid <media-id>

(Running NBU master and media server on W2008 is perfectly fine.)

[Marianne]

Some reading matter while you are collecting output:

How NetBackup selects media :
About selecting media in robots: http://www.veritas.com/docs/000043840 
About selecting media in standalone drives : http://www.veritas.com/docs/000043842

[skrp]

bppllist <policyname> -U Output

E:\Program Files\Veritas\NetBackup\bin\admincmd>bppllist Pol_Catalog -U
------------------------------------------------------------

Policy Name:       Pol_Catalog

  Policy Type:         NBU-Catalog
  Active:              yes
  Effective date:      09/08/2011 16:50:33
  Mult. Data Streams:  no
  Client Encrypt:      no
  Checkpoint:          no
  Policy Priority:     0
  Max Jobs/Policy:     1
  Disaster Recovery:   0
  Collect BMR info:    no
  Residence:           kthydbak101-hcart-robot-tld-0
  Volume Pool:         CatalogBackup
  Server Group:        *ANY*
  Keyword:             (none specified)
  Data Classification:       -
  Residence is Storage Lifecycle Policy:    no
  Application Discovery:      no
  Discovery Lifetime:      0 seconds
  ASC Application and attributes: (none defined)

  Granular Restore Info:  no
  Ignore Client Direct:  no
  Enable Metadata Indexing:  no
  Index server name:  NULL
  Use Accelerator:  no
  HW/OS/Client:  PC-x64        Windows2008   kthydbak101.kt.group.local

  Include:  CATALOG_DRIVEN_BACKUP

  Schedule:              Full
    Type:                Full Backup
    Calendar sched: Enabled
     Included Dates-----------
        Wednesday, Week 1
        Wednesday, Week 2
        Wednesday, Week 3
        Wednesday, Week 4
        Wednesday, Week 5
     Excluded Dates----------
        No specific exclude dates entered
        No exclude days of week entered
    PFI Recovery:        0
    Maximum MPX:         1
    Retention Level:     4 (2 months)
    Number Copies:       1
    Fail on Error:       0
    Residence:           (specific storage unit not required)
    Volume Pool:         (same as policy volume pool)
    Server Group:        (same as specified for policy)
    Residence is Storage Lifecycle Policy:         0
    Schedule indexing:   0
    Daily Windows:
          Wednesday  10:00:00  -->  Wednesday  19:00:00

  Schedule:              Cummulative-Incremental
    Type:                Cumulative Incremental Backup
    Calendar sched: Enabled
     Included Dates-----------
        Monday, Week 1
        Tuesday, Week 1
        Thursday, Week 1
        Friday, Week 1
        Monday, Week 2
        Tuesday, Week 2
        Thursday, Week 2
        Friday, Week 2
        Monday, Week 3
        Tuesday, Week 3
        Thursday, Week 3
        Friday, Week 3
        Monday, Week 4
        Tuesday, Week 4
        Thursday, Week 4
        Friday, Week 4
        Monday, Week 5
        Tuesday, Week 5
        Thursday, Week 5
        Friday, Week 5
     Excluded Dates----------
        No specific exclude dates entered
        No exclude days of week entered
    PFI Recovery:        0
    Maximum MPX:         1
    Retention Level:     3 (1 month)
    Number Copies:       1
    Fail on Error:       0
    Residence:           (specific storage unit not required)
    Volume Pool:         (same as policy volume pool)
    Server Group:        (same as specified for policy)
    Residence is Storage Lifecycle Policy:         0
    Schedule indexing:   0
    Daily Windows:
          Monday     10:00:00  -->  Monday     19:00:00
          Tuesday    10:00:00  -->  Tuesday    19:00:00
          Thursday   10:00:00  -->  Thursday   19:00:00
          Friday     10:00:00  -->  Friday     19:00:00

Catalog Disaster Recovery Configuration:
  Email Address:   kihyddatacenteroperationsteam@kantaritp.com
  Disk Path:       E:\NBUCatalogDR_DONOTEDIT
  User Name:       (none specified)
  Pass Word:       xxxx
  Critical policy: (none specified)

netbackup\bin\goodies\available_media Output is attached.

[Marianne]

So, you have changed retention levels in the Catalog policy at some point, right? You can see in available_media output that all tapes in the CatalogBackup pool have been written with Retention level 10. The retention levels are now 4 and 3 for the two schedules. NBU cannot append to any of the retention level 10s. I suggest you expire the oldest Catalog tape (bpexpdate ), put it back in the robot and make retention levels the same for both schedules. Level 3 (1 month ) will be fine as you will never need to restore old catalog info.

[skrp]

Marianne,

Iam unable to understand this 'NBU cannot append to any of the retention level 10s. '

[Marianne · Accepted solution]

As I've explained before - NetBackup does not mix retention levels on the same tape. All of the CatalogBackup tapes contain valid, unexpired images. NBU can only append to these tapes if schedules are level 10 as before. Any other retention level can only be written to these tapes (overwrite) once all images on them have expired.

[skrp]

Thanks Marianne.

I will use a daily tape to check it. Erase the data on it and use it as a catalog tape to check if it works. Will get back to you with the result.

[Marianne]

You don't need to erase the tape. Simply use bpexpdate to expire all images on the tape.

[skrp]

Hi All,

The problem was that we the retention levels have been changed to multiple retentions on a single tape which is not recommended by netbackup. Now changed it and the backups ran fine. Thank you all for your valuable time.

Special thanks to Marianne. I feel like I can solve any problem when I'm around you people.

[Marianne]

Have a look at this blog and subsequent comments to understand why multiple retentions is not recommended : http://forum.support.veritas.com/connect/blogs/understanding-how-netbackup-writes-tape