cancel
Showing results for 
Search instead for 
Did you mean: 

NB_CATALOG error

turguns
Level 5

 

Hi ALL,

Netbackup 7.1.

Linux Redhat 6.1

1.NB_catalog failing with this error:

Jan 14, 2016 8:00:13 PM - Info bpdbm (pid=31399) staging relational database files for catalog backup
Jan 14, 2016 8:00:13 PM - Info bpdbm (pid=31399) staging NBAZDB backup to /usr/openv/db/staging
Jan 14, 2016 8:00:13 PM - Error bpdbm (pid=31399) error staging NBAZDB backup to /usr/openv/db/staging
Jan 14, 2016 8:00:13 PM - Info bpdbm (pid=31399) staging NBDB backup to /usr/openv/db/staging
Jan 14, 2016 8:00:15 PM - Error bpdbm (pid=31399) error staging NBDB backup to /usr/openv/db/staging
none of the requested files were backed up  (2)

2. After that I've pinged the DB

/usr/openv/db/bin/nbdb_ping -dbn NBDB
Database [NBDB] is alive and well on server [NB_nbumedia].

nbdb_ping -dbn NBAZDB
bash: nbdb_ping: command not found

 

3. I've started the NBAZDB. 

 /usr/openv/db/bin/nbdb_ping -dbn NBAZDB
Database [NBAZDB] is alive and well on server [NB_nbumedia].

4.  But now error on staging NBDB

Jan 15, 2016 5:48:31 PM - Info bpdbm (pid=17979) staging relational database files for catalog backup
Jan 15, 2016 5:48:31 PM - Info bpdbm (pid=17979) staging NBAZDB backup to /usr/openv/db/staging
Jan 15, 2016 5:48:32 PM - Info bpdbm (pid=17979) done staging NBAZDB backup to /usr/openv/db/staging
Jan 15, 2016 5:48:32 PM - Info bpdbm (pid=17979) staging NBDB backup to /usr/openv/db/staging
Jan 15, 2016 5:48:34 PM - Error bpdbm (pid=17979) error staging NBDB backup to /usr/openv/db/staging
the requested operation was partially successful  (1)

 

5. I've validated both of db, No error:

/usr/openv/db/bin/nbdb_admin -validate NBAZDB
SQL Anywhere Validation Utility Version 11.0.1.2475
VALIDATE TABLE "DBA"."acle" WITH EXPRESS CHECK 

      :

No errors reported
Database [NBAZDB] validation successful.

_______________________________
[root@nbumedia ~]# /usr/openv/db/bin/nbdb_admin -validate NBDB
SQL Anywhere Validation Utility Version 11.0.1.2475
VALIDATE TABLE "ADTR_MAIN"."ADTR_AuditRecord" WITH EXPRESS CHECK

      :

No errors reported
Database [NBDB] validation successful.

 

6.  Three days ago I manually started NBAZDB. But it stopped after the following process.

Jan 14 05:26:05 nbumedia SQLAnywhere(nb_nbumedia): Starting database "NBDB" (/usr/openv/db/data/NBDB.db) at Thu Jan 14 2016 05:26
Jan 14 05:26:06 nbumedia SQLAnywhere(nb_nbumedia): Opening dbspace 'EMM_DATA' in file '/usr/openv/db/data/EMM_DATA.db' for database 'NBDB'
Jan 14 05:26:06 nbumedia SQLAnywhere(nb_nbumedia): Opening dbspace 'EMM_INDEX' in file '/usr/openv/db/data/EMM_INDEX.db' for database 'NBDB'
Jan 14 05:26:06 nbumedia SQLAnywhere(nb_nbumedia): Opening dbspace 'DBM_DATA' in file '/usr/openv/db/data/DBM_DATA.db' for database 'NBDB'
Jan 14 05:26:06 nbumedia SQLAnywhere(nb_nbumedia): Opening dbspace 'DBM_INDEX' in file '/usr/openv/db/data/DBM_INDEX.db' for database 'NBDB'
Jan 14 05:26:06 nbumedia SQLAnywhere(nb_nbumedia): Opening dbspace 'DARS_DATA' in file '/usr/openv/db/data/DARS_DATA.db' for database 'NBDB'
Jan 14 05:26:06 nbumedia SQLAnywhere(nb_nbumedia): Opening dbspace 'DARS_INDEX' in file '/usr/openv/db/data/DARS_INDEX.db' for database 'NBDB'
Jan 14 05:26:06 nbumedia SQLAnywhere(nb_nbumedia): Transaction log: /usr/openv/db/data/NBDB.log

Today I started NABZDB  again.

 

7. Conf file is as following:

more  /usr/openv/var/global/databases.conf 
"/usr/openv/db/data/NBDB.db" -n NBDB

 

Any suggestions?

BR,

Turgun

 

 

 

 

 

 

 

 

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
Another TN that may help with getting rid of problematic log: http://www.veritas.com/docs/000089830 PS: Any reason why you are still on such an old version? You can see from the previous TN that 7.5 and later handels log files a lot better.

View solution in original post

11 REPLIES 11

revarooo
Level 6
Employee

Ok so in your databases.conf there is no nbazdb database so don't worry about nbazdb, lets worry about nbdb (the NetBackup Database).

what happens if you run this command, which is the staging part of the catalog backup jobs prior to backing up the database from the staging directory.

/usr/openv/db/bin/nbdb_backup -dbn NBDB -online /usr/openv/db/staging

does it succeed?

 

Have you checked disk space?

 

Have you checked the bpdbm log? search for PID [17979] and <16> or <32>

turguns
Level 5

Hi Revarooo,

1.The command output is :

# /usr/openv/db/bin/nbdb_backup -dbn NBDB -online /usr/openv/db/staging

Online backup of database to /usr/openv/db/staging failed.

2.Disk space is enough( 35G free)

3 .I've activated bpdbm log file. From attachment you can find   it. 

There many read_info_file and  I couldn't find any error except the following.

more log.011516 | grep error
19:32:24.787 [24331] <2> error_db: Q_ERRADD
19:32:24.787 [24331] <4> db_error_add_to_file: VBRT 1 12441 1 1 hcart2040 E818L5 0 1 0 0 0 
19:33:00.020 [24481] <2> error_db: Q_ERRADD

I couldn't find that pid in this log.

On ps-ef ouptut

UID        PID  PPID  C STIME TTY          TIME CMD

root        16     2  0 Jan09 ?        00:00:00 [migration/3]

root        32     2  0 Jan09 ?        00:00:00 [migration/7]

No processes with pid 17979

Thanks,

Turgun

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
To find info for bpdbm pid 17979, you had to have bpdbm logging enabled at the time that the backup was attempted and produced the error. Jan 15, 2016 5:48:31 PM -

turguns
Level 5

Hi Marianne and Revarooo.

I have allready enabled bpdbm logging. But just now I got the logic about pid )).

1.The following is NBU catalog backup error from GUI:

Jan 16, 2016 2:36:39 AM - Info bpdbm (pid=11399) staging relational database files for catalog backup
Jan 16, 2016 2:36:39 AM - Info bpdbm (pid=11399) staging NBAZDB backup to /usr/openv/db/staging
Jan 16, 2016 2:36:39 AM - Info bpdbm (pid=11399) done staging NBAZDB backup to /usr/openv/db/staging
Jan 16, 2016 2:36:39 AM - Info bpdbm (pid=11399) staging NBDB backup to /usr/openv/db/staging
Jan 16, 2016 2:36:42 AM - Error bpdbm (pid=11399) error staging NBDB backup to /usr/openv/db/staging
the requested operation was partially successful  (1)

2.Now I've searched from bpdbm according to that PID.

I've listed here odd messages from there. Full messages is at the attachment.

02:36:39.554 [11399] <4> do_online_nbdb_backup: done staging NBAZDB backup to /usr/openv/db/staging
02:36:39.554 [11399] <4> do_online_nbdb_backup: Online backup of NBAZDB successful
02:36:39.554 [11399] <4> do_online_nbdb_backup: Server nbumedia, Policy NBU-Catalog, Schedule Full, parentJobID 1043339
02:36:39.554 [11399] <4> do_online_nbdb_backup: staging NBDB backup to /usr/openv/db/staging
02:36:39.569 [11399] <4> backup_dbspaces: Executing:  backup database directory '/usr/openv/db/staging' transaction log truncate;
02:36:42.210 [11399] <16> backup_dbspaces: Execute SQL failed: backup database directory '/usr/openv/db/staging' transaction log truncate;
02:36:42.210 [11399] <16> backup_dbspaces: ErrMsg [Sybase][ODBC Driver][SQL Anywhere]Error during backup/restore: Error writing backup file "/usr/openv/db/staging/NBDB.log", ErrCode -1, Sqlstate HY000
02:36:42.223 [11399] <16> do_online_nbdb_backup: error staging NBDB backup to /usr/openv/db/staging
02:36:42.223 [11399] <16> do_online_nbdb_backup: Error completing online backup of:  NBDB

02:36:57.826 [11399] <16> exec_catalog_backup: unexpected result from exec_asa_database_backup(): check_point is 0
02:36:57.827 [11399] <16> exec_catalog_backup: get_prev_catalog_backup(parentid=NBU-Catalog_1452897399_FULL) failed (227) (retry:2)
02:36:59.827 [11399] <16> exec_catalog_backup: get_prev_catalog_backup(parentid=NBU-Catalog_1452897399_FULL) failed (227) (retry:1)
02:37:01.828 [11399] <16> exec_catalog_backup: get_prev_catalog_backup(parentid=NBU-Catalog_1452897399_FULL) failed (227) (retry:0)

 

Thanks,

Turgun

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
There seems to be a problem with the transaction log. What is the size of the log? When last did a full catalog backup run? Try to manually backup and truncate using steps in this TN : http://www.veritas.com/docs/000039913 Specify another location, not NBU staging directory. /tmp maybe.

turguns
Level 5

1. Yes, As you mentioned the size of log file is 6.3GB.

6.3G Jan 16 15:16 NBDB.log

2. last all full backups are also failed. for approx. more than 1 week we have issue.

3. When I'm trying to backup and truncate getting error:

# /usr/openv/db/bin/nbdb_backup -dbn NBDB -online /nbdblog  -truncate_tlog
Online backup of database to /nbdblog failed.

 

I've checked inside of /nbdblog directory. Created some files

# ls -lh /nbdblog/
total 229M
-rw-rw-rw-. 1 root root  26M Jan 16 15:20 DARS_DATA.db
-rw-rw-rw-. 1 root root  26M Jan 16 15:20 DARS_INDEX.db
-rw-rw-rw-. 1 root root  26M Jan 16 15:20 DBM_DATA.db
-rw-rw-rw-. 1 root root  26M Jan 16 15:20 DBM_INDEX.db
-rw-rw-rw-. 1 root root  66M Jan 16 15:20 EMM_DATA.db
-rw-rw-rw-. 1 root root  26M Jan 16 15:20 EMM_INDEX.db
-rw-rw-rw-. 1 root root 3.6M Jan 16 15:20 NBDB.db
-rw-rw-rw-. 1 root root  34M Jan 16 15:20 NBDB.log

But why failing and not truncating the transaction log ((.

# ls -lh /usr/openv/db/data/NBDB.log 
-rw-------. 1 root root 6.4G Jan 16 15:31 /usr/openv/db/data/NBDB.log

 

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
Another TN that may help with getting rid of problematic log: http://www.veritas.com/docs/000089830 PS: Any reason why you are still on such an old version? You can see from the previous TN that 7.5 and later handels log files a lot better.

turguns
Level 5

Okay, Now there big backup data is processing. I will try it by tomorrow evening.

1.

@Marianne, there on Note 7, like this command : 

/usr/openv/db/bin/dbeng9 -f /usr/openv/db/data/NBDB.db

But on my system, 

# ls -lh /usr/openv/db/bin/dbeng*
-r-x--x---. 1 root daemon 65K Feb  4  2011 /usr/openv/db/bin/dbeng11

 
Is there  no difference dbeng9 and dbeng11 ? 
 
2. Is this huge transaction log impact the system performance?
We had performance issue last two week on each saturday. 
Regarding this issue I've created another post. (https://www.veritas.com/community/forums/slowness-netbackup-java-console-high-io-only-saturday) 
 
Thank you,
Turgun
 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
I have no idea. And honestly no idea what else to suggest. The large transaction log is probably because of other issues in your environment and now no idea what caused it or what came first... The large log is because of repeated unsuccessful catalog backups, but what caused it in the first place? Seems we are not going to find out what came first. The chicken or the egg? Just wondering if OS patches are as far out of date as NBU?

turguns
Level 5

Hi All,

Thank you for your valuable feedback.

@ Marianne, your suggested link was very helpfull. (http://www.veritas.com/docs/000089830).

Thanks.

Turgun

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

PLEASE check OS patch levels on your NBU master and upgrade NBU as a matter of urgency....

You may need to upgrade hardware on your NBU master as well - NBU 7.5 and later needs more memory.

Please upgrade straight to 7.7.1 as all versions up to 7,6,x will reach EOSL in a year from now.

Use SORT to prepare for upgrade: https://sort.veritas.com/assessment/install_upgrade