Forum Discussion

SYMAJ's avatar
SYMAJ
Level 6
13 years ago

Netbackup Catalog Backup Failure After Upgrade to 7.5.0.4

Having just upgraded my Master Server and 3 Media Servers from 7.0.1. to 7.5.0.4 all regular backups and general system operation appears to be running OK.  However - I have just attempted a Catalog Backup and it is failing.

The parent job starts and spawns two more jobs as per usual.  The first job stages the NBAZDB and NBDB to disk, and the second one writes the DARS_DATA.db to tape.  Both of these jobs work OK.  Then just as I expect to see the last job be spawned to copy the main catalog files to tape (290GB) the parent job fails with the following:

System Call Failed(11)

The final copy job never starts.

I did have NBAC configured prior to the upgrade, but this is now set to PROHIBITED as it was causing the upgrade to fail.  I am investigating this as a seperate issue.

Any pointers ?

AJ

7.5
Backing Up
Backup and Recovery
Error messages
Installing
NetBackup
Tip-How to
Troubleshooting
Upgrade
  • SYMAJ's avatar
    SYMAJ
    13 years ago

    OK - Update on this issue.

    Things got worse before they got better.......  I ended up in a situation - after following Tech190214 - whereby the databases would not start due to  SQLAny 11.0 errors.  This was caused by the transaction log having being used since the databases were last used - thereby stopping the databases from starting (taken from error logs / server logs).  Netbackup would not start therefore I had no ability to backup or restore.

    This was caused - I believe - by one of the steps in the TECH190214 prior to step 5.  It only became apparrent when I was doing the 7504 update in step 6 - when it could not update the database as it was unavailable.

    After a lot of investigation, in order to get round this we deleted the transaction log (in conjunction with support) and then the databases all started OK. 

    I was now back in a position whereby the system would backup/restore OK but I could still not perform a catalog backup - job was running the first two of three steps (ignoring the parent job) OK then failing status 11 without ever starting the Image library backup job.

    Whilst support were convinced this was down to NBAC, hence following the technote to remove NBAC functions, it appears not to be the case. 

    We redirect ALL the images to an alternate location using the registry entry IMAGESDB_PATH as opposed to the ALTPATH touchfile.  While attempting to run the catalog backup we noticed that some images were being written to the original images location (default) and some to the path in the registry entry IMAGESDB_PATH.  Support could not understand why the regsitry entry location was not being honoured for all of these files, as it was for regular backups of clients.  On the off-chance I put an ALTPATH touch-file in the original images location for the master server and hey presto - the catalog backup works - writing only to the alternate location and submitting the job to backup the images correctly.  Support are at a loss why this should work, but it does.  They are going to continue to look into this (they have all the logs etc.) and we will pick up again this evening.

    Summary - this appears not to be NBAC related, but due to the IMAGESDB_PATH registry entry not being honoured during the catalog backup.  Resolved by placing an ALTPATH touchfile in the original images location for the master server.

    AJ

  • Mark_Solutions's avatar
    Mark_Solutions
    Level 6
    13 years ago

    The authentication service is now used by NBU 7.5 so if that is disabled it may be causing the issue

    Check the bpdbm, bptm and admin logs on the Master as they may give more information

    It maybe that the upgrade has not included NBAC to leave authentication in an unstable / unmatched state

  • SYMAJ's avatar
    SYMAJ
    Level 6
    13 years ago

    Relevant log content below:

    BPDBM

    change_file_modified_date: CreateFile() failed

    <pathname to catalog images location>CO_Catalog_Backup_1352472612_Full (2)

    ImageCatalogBackup::export_image_and_previous_images: change_file_modified_date(1352472605, <pathname to catalog images location>CO_Catalog_Backup_1352472612_Full) failed (11)

    ImageCatalogBackup::executeQuery: Error exporting NBDB backup metadata (11)

    BPTM

    Looks OK

    ADMIN

    Looks OK

     

    I have checked the completion of the Phase 1 and Phase 2 catalog conversion jobs and they are both completed, however I still have the Image Catalog Cleanup job running.  It has imported all of the images OK but is now 'deleting images which expire before Fri Nov 09......'.  It has been sitting here for some time, using CPU but doesn't appear to be progressing.

    AJ

     

  • Mark_Solutions's avatar
    Mark_Solutions
    Level 6
    13 years ago

    When did you do the upgrade?

    Did you upgrade to 7.5 then shut down NBU, patch to 7.5.0.4 and then let the upgrade complete?

    Until the image cleanup has finished it is not totally complete - wait for that to finish first as it imports the images and does a tidy up - you should not run anything until it is all up to date

  • SYMAJ's avatar
    SYMAJ
    Level 6
    13 years ago

    I raised a support call over this one and it appears that the upgrade was successfull (I did let all the catalog migrations etc. complete and check the phase 1 & phase 2 steps were complete) however the NBAC issue during installation appears to be the issue.

    Support talked me thorugh removing NBAC but this did not resolve the issue.  They have since sent me through an updated procedure for removing NBAC:

    http://www.symantec.com/business/support/index?page=content&id=TECH190214

    I will try this on Monday and update you as to whether it fixes my problem.

    AJ

  • SYMAJ's avatar
    SYMAJ
    Level 6
    13 years ago

    OK - Update on this issue.

    Things got worse before they got better.......  I ended up in a situation - after following Tech190214 - whereby the databases would not start due to  SQLAny 11.0 errors.  This was caused by the transaction log having being used since the databases were last used - thereby stopping the databases from starting (taken from error logs / server logs).  Netbackup would not start therefore I had no ability to backup or restore.

    This was caused - I believe - by one of the steps in the TECH190214 prior to step 5.  It only became apparrent when I was doing the 7504 update in step 6 - when it could not update the database as it was unavailable.

    After a lot of investigation, in order to get round this we deleted the transaction log (in conjunction with support) and then the databases all started OK. 

    I was now back in a position whereby the system would backup/restore OK but I could still not perform a catalog backup - job was running the first two of three steps (ignoring the parent job) OK then failing status 11 without ever starting the Image library backup job.

    Whilst support were convinced this was down to NBAC, hence following the technote to remove NBAC functions, it appears not to be the case. 

    We redirect ALL the images to an alternate location using the registry entry IMAGESDB_PATH as opposed to the ALTPATH touchfile.  While attempting to run the catalog backup we noticed that some images were being written to the original images location (default) and some to the path in the registry entry IMAGESDB_PATH.  Support could not understand why the regsitry entry location was not being honoured for all of these files, as it was for regular backups of clients.  On the off-chance I put an ALTPATH touch-file in the original images location for the master server and hey presto - the catalog backup works - writing only to the alternate location and submitting the job to backup the images correctly.  Support are at a loss why this should work, but it does.  They are going to continue to look into this (they have all the logs etc.) and we will pick up again this evening.

    Summary - this appears not to be NBAC related, but due to the IMAGESDB_PATH registry entry not being honoured during the catalog backup.  Resolved by placing an ALTPATH touchfile in the original images location for the master server.

    AJ