catalog backup failing with a none of the requested files were backed up (2)

[contra04]

Hi this has failed for over 2 weeks now. 100 percent of the time

30/09/2011 09:40:29 - Info bpdbm(pid=7668) staging relational database files for catalog backup      
30/09/2011 09:40:29 - Info bpdbm(pid=7668) staging NBDB backup to C:\Program Files\VERITAS\NetBackupDB\staging       
30/09/2011 09:48:11 - Error bpdbm(pid=7668) error staging NBDB backup to C:\Program Files\VERITAS\NetBackupDB\staging      
none of the requested files were backed up(2)

All of the technotes and forums talk about database consistancy - but we do not get that error.

I have no idea what I am looking for in the bpdbm logs:

I tried a manual Full backup at 9:40 this morning  I have attached the log for the collective experts perusal.

As always any help would be extremely appreciated !

 ParseConfigExA: Unknown configuration option on line 93: RenameIfExists = 0
09:48:09.573 [2872.5580] <2> image_db: Q_IMAGE_ADD_FILES (locking)
09:48:09.589 [2872.5580] <2> db_get_image_info: Job in progress, found image file D:\NetbackupDB\CLIENTS\grub-bak\1317000000\tmp\grub-bak_D_Excl_DFS_1317369638_INCR
09:48:10.761 [2872.5580] <2> process_request: request complete: exit status 0  ; query type: 78
09:48:11.714 [7668.2088] <16> backup_dbspaces: Execute SQL failed: backup database directory 'C:\Program Files\VERITAS\NetBackupDB\staging' transaction log truncate;
09:48:11.714 [7668.2088] <16> backup_dbspaces: ErrMsg [Sybase][ODBC Driver][SQL Anywhere]Error during backup/restore: Error writing backup file "C:\Program Files\VERITAS\NetBackupDB\staging\NBDB.log", ErrCode -1, Sqlstate HY000
09:48:11.730 [7668.2088] <16> do_online_nbdb_backup: error staging NBDB backup to C:\Program Files\VERITAS\NetBackupDB\staging
09:48:11.730 [7668.2088] <16> do_online_nbdb_backup: Error completing online backup of:  NBDB
09:48:11.730 [7668.2088] <2> ParseConfigExA: Unknown configuration option on line 92: VssOrphanage = {62eadb2e-8ebe-4b9a-ba3f-8460ecc8eb88}
09:48:11.730 [7668.2088] <2> ParseConfigExA: Unknown configuration option on line 93: RenameIfExists = 0
09:48:11.745 [7668.2088] <2> job_monitoring_exex: ACK disconnect
09:48:11.745 [7668.2088] <2> job_disconnect: Disconnected
09:48:11.761 [7668.2088] <2> ParseConfigExA: Unknown configuration option on line 92: VssOrphanage = {62eadb2e-8ebe-4b9a-ba3f-8460ecc8eb88}
09:48:11.761 [7668.2088] <2> ParseConfigExA: Unknown configuration option on line 93: RenameIfExists = 0
09:48:11.761 [7668.2088] <4> empty_staging_dir: Emptying staging directory:  C:\Program Files\VERITAS\NetBackupDB\staging
09:48:13.558 [7668.2088] <16> exec_catalog_backup: exec_asa_database_backup() failed (2)
09:48:13.573 [7668.2088] <2> exec_catalog_DR_protection: classname=Catalog_Backup, fqDRImagePath=<NULL>, status=2
09:48:13.589 [7668.2088] <2> exec_catalog_DR_protection: Sending DR image info via email
09:48:13.589 [7668.2088] <4> db_DR_generate_report: ?
09:48:13.620 [7668.2088] <4> db_DR_send_report: ?
09:48:13.620 [7668.2088] <2> sendViaScript: attempting to send mail
09:48:13.620 [7668.2088] <2> sendViaScript: C:\Program Files\VERITAS\NetBackup\bin\nbmail.cmd "geeks@robertwalters.com" "NetBackup Catalog Backup FAILED on host harry status 2" "C:\WINDOWS\TEMP\nb_1A.tmp"
09:48:13.667 [7668.2088] <2> image_db: sending status = 2 back to jobmgr on host harry for job 10786
09:48:13.667 [7668.2088] <4> jmcomm_UpdateActionStatus: Sending status: 2 back to parent JobID: 10786
09:48:13.714 [7668.2088] <2> ParseConfigExA: Unknown configuration option on line 92: VssOrphanage = {62eadb2e-8ebe-4b9a-ba3f-8460ecc8eb88}
09:48:13.714 [7668.2088] <2> ParseConfigExA: Unknown configuration option on line 93: RenameIfExists = 0
09:48:13.714 [7668.2088] <2> Orb::init: Created anon service name: NB_7668_142834866(Orb.cpp:612)
09:48:13.714 [7668.2088] <2> Orb::init: endpointvalue is : pbxiop://1556:NB_7668_142834866(Orb.cpp:630)
09:48:13.714 [7668.2088] <2> Orb::init: initializing ORB Default_DAEMON_Orb with: Unknown -ORBSvcConfDirective "-ORBDottedDecimalAddresses 0" -ORBSvcConfDirective "static PBXIOP_Factory '-enable_keepalive'" -ORBSvcConfDirective "static EndpointSelectorFactory ''" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory PBXIOP_Factory'" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory IIOP_Factory'" -ORBSvcConfDirective "static PBXIOP_Evaluator_Factory '-orb Default_DAEMON_Orb'" -ORBSvcConfDirective "static Resource_Factory '-ORBConnectionCacheMax 1024 '" -ORBEndpoint pbxiop://1556:NB_7668_142834866 -ORBSvcConf nul -ORBSvcConfDirective "static Server_Strategy_Factory '-ORBMaxRecvGIOPPayloadSize 268435456'"(Orb.cpp:741)
09:48:13.745 [7668.2088] <2> EndpointSelector::select_endpoint: performing call with the only endpt available!(Endpoint_Selector.cpp:437)
09:48:13.776 [7668.2088] <2> jmcomm_UpdateActionStatus: returning
09:48:13.776 [7668.2088] <2> image_db: done sending status back to jobmgr for job 10786
09:48:13.776 [7668.2088] <2> process_request: request complete: exit status 2 none of the requested files were backed up; query type: 160
09:48:13.839 [2936.6892] <2> logconnections: BPDBM ACCEPT FROM 10.1.104.78.4787 TO 10.1.104.78.13724
09:48:13.839 [2936.6892] <2> ParseConfigExA: Unknown configuration option on line 92: VssOrphanage = {62eadb2e-8ebe-4b9a-ba3f-8460ecc8eb88}
09:48:13.839 [2936.6892] <2> ParseConfigExA: Unknown configuration option on line 93: RenameIfExists = 0

[Andy_Welburn · Accepted solution]

All the T/N's I've found indicate the same thing (in relation to the error "error staging NBDB"):

http://www.symantec.com/business/support/index?page=content&id=TECH57150
http://www.symantec.com/business/support/index?page=content&id=TECH64987
http://www.symantec.com/business/support/index?page=content&id=TECH85209
http://www.symantec.com/business/support/index?page=content&id=TECH160590

***EDIT***

That could also explain your comment yesterday about services not restarting

[Mark_Solutions]

Dont know where my last post went but i'll try again ...

You seem to have the error : Sqlstate HY000
indicating there is an issue staging the database

This may be a read / write issue on the staging directory or a problem with the database.

In the netbackup\bin\ directory is NbDbAdmin.exe

Run this and use the password nbusql to log in.

This shows the state of your databases so you can see what is happening.

This tech note covers more:

http://www.symantec.com/business/support/index?page=content&id=TECH126758

Hope this helps

[contra04]

Thanks guys, I actually got a netbackup support case opened, and apparently when the catalogue backup fails, it begins to grow in size. I had 3.5 gig free on the C:\ and the database had grown to 7 gigs.  I mistakenly thought it was only 12mb as that is what I saw in the db directory. 

We are not able to celar enough space on the C:\ - I understand moving the database is a difficult task? Ill see if I can have a poke around trying to find out how to do this.

[Mark_Solutions]

Hi

Moving the DB is not too bad a job and there is a tech note out there on how to do it.

Just needs NBU shutting down, the data moving, the registry editing and NBU starting up again.

If you need help then do let us know - we could be engaged to do this for you (remotely if neccesary)

[contra04]

The symantec chap talked about 7 gigs woth of staging required for a catalog backup that has been failing for this long.

My error points staging on :

C:\Program Files\VERITAS\NetBackupDB\staging\ (No slash between Netbackup and DB)

the C:\ only has 3 gig of space.

However isnt the database under C:\Program Files\VERITAS\NetBackup\DB\ somewhere?(Where is it? what is it called?)

If the backup is staging here - when I move the database will there not still be a problem because the staging files will still be on the C Drive?

But the entire C:\Program Files\VERITAS\NetBackupDB\ directory and subfolders are only 52 megabytes? can the database really be that small and have 7 gig worth of metadata/logs etc?

[contra04]

OK all the folders under

C:\Program Files\VERITAS\NetBackup\DB\images\myservers......\ALTPATH

already have an "altpath" file pointing to : D:\NetbackupDB\CLIENTS\

which points each folder, for each client to:

D:\NetbackupDB\CLIENTS\servername

What is strange is that there is no registry key pointing there ?? how does it know to go there?

Does it just read the file and doesnt need the reg key in newer versions of netbackup ?

Which contains 131GB size and 58.4 GB size on disk

How it expect to stage on 3 gigs is crazy....

Should the database be so big?

how do I change the staging area?

my vxdbms.conf is located here .....D:\NetbackupDB\DATA contains the below information, but has no mention of staging.  Do I just add the staging line as specified in Andy's Technotes?

VXDBMS_NB_SERVER = NB_HARRY
VXDBMS_NB_PORT = 13785
VXDBMS_NB_DATABASE = NBDB
VXDBMS_NB_DATA = D:\NetbackupDB\DATA
VXDBMS_NB_INDEX = D:\NetbackupDB\INDEX
VXDBMS_NB_TLOG = D:\NetbackupDB\TLOG
VXDBMS_NB_MLOG = D:\NetbackupDB\MLOG
VXDBMS_NB_PASSWORD = f431dbe5b4f4136fb7ebbd483f9d13bd3c95f1309ab37499
VXDBMS_ODBC_DRIVER = NB SQL Anywhere
VXDBMS_NB_FULL_KEYWORD = NBDB:9630:1316127611:F
VXDBMS_NB_INCREMENTAL = NBDB.log.1

[contra04]

[contra04]

I changed the file with C:\program files\VERITAS\netbackup\bin>nbdb_admin.exe -VXDBMS_NB_STAGING S:\catalog_db_sataging

Now the job starts and creates these files:

Activity monitor:

05/10/2011 11:37:31 - begin Catalog Backup, DBM Query and the status is stuck at mounting. I imagine it is going to fail.....

[Mark_Solutions]

Hi

Your issue is the size of the NBDB log files - this has been caused due to the lack of catalog backups.

If you can get this backup through it will truncate the logs and suddenly you will get 6GB back (or 12GB as you have a mirror of the log)

Keep us updated

If you do need professional services drop me a mail (can be carried out remotely if you are desperate)

[contra04]

getting there !

05/10/2011 11:37:35 - Info bpdbm(pid=6292) staging relational database files for catalog backup      
05/10/2011 11:37:35 - Info bpdbm(pid=6292) staging NBDB backup to S:\catalog_db_staging        
05/10/2011 12:28:35 - Info bpdbm(pid=6292) done staging NBDB backup to S:\catalog_db_staging       
05/10/2011 12:38:40 - Info bpdbm(pid=6292) validating NBDB backup in S:\catalog_db_staging   

and:

the log file now in staging directory - we never got this far before. feeling better...

[contra04]

success.  log file now at 10 meg. took 12 hours to complete the Job.

Thankyou for your help everyone.

A shame symantec support engineer that we pay for wasnt any good.

[Mark_Solutions]

contra04 - we offer our own support contract and get a lot of take up on it due to how quickly we can respond - we wouldn't do so well if there wasnt a demand - i will say no more!