01-15-2016 06:09 AM
Hi ALL,
Netbackup 7.1.
Linux Redhat 6.1
1.NB_catalog failing with this error:
Jan 14, 2016 8:00:13 PM - Info bpdbm (pid=31399) staging relational database files for catalog backup
Jan 14, 2016 8:00:13 PM - Info bpdbm (pid=31399) staging NBAZDB backup to /usr/openv/db/staging
Jan 14, 2016 8:00:13 PM - Error bpdbm (pid=31399) error staging NBAZDB backup to /usr/openv/db/staging
Jan 14, 2016 8:00:13 PM - Info bpdbm (pid=31399) staging NBDB backup to /usr/openv/db/staging
Jan 14, 2016 8:00:15 PM - Error bpdbm (pid=31399) error staging NBDB backup to /usr/openv/db/staging
none of the requested files were backed up (2)
2. After that I've pinged the DB
/usr/openv/db/bin/nbdb_ping -dbn NBDB
Database [NBDB] is alive and well on server [NB_nbumedia].
nbdb_ping -dbn NBAZDB
bash: nbdb_ping: command not found
3. I've started the NBAZDB.
/usr/openv/db/bin/nbdb_ping -dbn NBAZDB
Database [NBAZDB] is alive and well on server [NB_nbumedia].
4. But now error on staging NBDB
Jan 15, 2016 5:48:31 PM - Info bpdbm (pid=17979) staging relational database files for catalog backup
Jan 15, 2016 5:48:31 PM - Info bpdbm (pid=17979) staging NBAZDB backup to /usr/openv/db/staging
Jan 15, 2016 5:48:32 PM - Info bpdbm (pid=17979) done staging NBAZDB backup to /usr/openv/db/staging
Jan 15, 2016 5:48:32 PM - Info bpdbm (pid=17979) staging NBDB backup to /usr/openv/db/staging
Jan 15, 2016 5:48:34 PM - Error bpdbm (pid=17979) error staging NBDB backup to /usr/openv/db/staging
the requested operation was partially successful (1)
5. I've validated both of db, No error:
/usr/openv/db/bin/nbdb_admin -validate NBAZDB
SQL Anywhere Validation Utility Version 11.0.1.2475
VALIDATE TABLE "DBA"."acle" WITH EXPRESS CHECK
:
No errors reported
Database [NBAZDB] validation successful.
_______________________________
[root@nbumedia ~]# /usr/openv/db/bin/nbdb_admin -validate NBDB
SQL Anywhere Validation Utility Version 11.0.1.2475
VALIDATE TABLE "ADTR_MAIN"."ADTR_AuditRecord" WITH EXPRESS CHECK
:
No errors reported
Database [NBDB] validation successful.
6. Three days ago I manually started NBAZDB. But it stopped after the following process.
Jan 14 05:26:05 nbumedia SQLAnywhere(nb_nbumedia): Starting database "NBDB" (/usr/openv/db/data/NBDB.db) at Thu Jan 14 2016 05:26
Jan 14 05:26:06 nbumedia SQLAnywhere(nb_nbumedia): Opening dbspace 'EMM_DATA' in file '/usr/openv/db/data/EMM_DATA.db' for database 'NBDB'
Jan 14 05:26:06 nbumedia SQLAnywhere(nb_nbumedia): Opening dbspace 'EMM_INDEX' in file '/usr/openv/db/data/EMM_INDEX.db' for database 'NBDB'
Jan 14 05:26:06 nbumedia SQLAnywhere(nb_nbumedia): Opening dbspace 'DBM_DATA' in file '/usr/openv/db/data/DBM_DATA.db' for database 'NBDB'
Jan 14 05:26:06 nbumedia SQLAnywhere(nb_nbumedia): Opening dbspace 'DBM_INDEX' in file '/usr/openv/db/data/DBM_INDEX.db' for database 'NBDB'
Jan 14 05:26:06 nbumedia SQLAnywhere(nb_nbumedia): Opening dbspace 'DARS_DATA' in file '/usr/openv/db/data/DARS_DATA.db' for database 'NBDB'
Jan 14 05:26:06 nbumedia SQLAnywhere(nb_nbumedia): Opening dbspace 'DARS_INDEX' in file '/usr/openv/db/data/DARS_INDEX.db' for database 'NBDB'
Jan 14 05:26:06 nbumedia SQLAnywhere(nb_nbumedia): Transaction log: /usr/openv/db/data/NBDB.log
Today I started NABZDB again.
7. Conf file is as following:
more /usr/openv/var/global/databases.conf
"/usr/openv/db/data/NBDB.db" -n NBDB
Any suggestions?
BR,
Turgun
Solved! Go to Solution.
01-16-2016 06:38 AM
01-15-2016 06:52 AM
Ok so in your databases.conf there is no nbazdb database so don't worry about nbazdb, lets worry about nbdb (the NetBackup Database).
what happens if you run this command, which is the staging part of the catalog backup jobs prior to backing up the database from the staging directory.
/usr/openv/db/bin/nbdb_backup -dbn NBDB -online /usr/openv/db/staging
does it succeed?
Have you checked disk space?
Have you checked the bpdbm log? search for PID [17979] and <16> or <32>
01-15-2016 08:14 AM
Hi Revarooo,
1.The command output is :
# /usr/openv/db/bin/nbdb_backup -dbn NBDB -online /usr/openv/db/staging
Online backup of database to /usr/openv/db/staging failed.
2.Disk space is enough( 35G free)
3 .I've activated bpdbm log file. From attachment you can find it.
There many read_info_file and I couldn't find any error except the following.
more log.011516 | grep error
19:32:24.787 [24331] <2> error_db: Q_ERRADD
19:32:24.787 [24331] <4> db_error_add_to_file: VBRT 1 12441 1 1 hcart2040 E818L5 0 1 0 0 0
19:33:00.020 [24481] <2> error_db: Q_ERRADD
I couldn't find that pid in this log.
On ps-ef ouptut
UID PID PPID C STIME TTY TIME CMD
root 16 2 0 Jan09 ? 00:00:00 [migration/3]
root 32 2 0 Jan09 ? 00:00:00 [migration/7]
No processes with pid 17979
Thanks,
Turgun
01-15-2016 10:10 AM
01-15-2016 04:31 PM
Hi Marianne and Revarooo.
I have allready enabled bpdbm logging. But just now I got the logic about pid )).
1.The following is NBU catalog backup error from GUI:
Jan 16, 2016 2:36:39 AM - Info bpdbm (pid=11399) staging relational database files for catalog backup
Jan 16, 2016 2:36:39 AM - Info bpdbm (pid=11399) staging NBAZDB backup to /usr/openv/db/staging
Jan 16, 2016 2:36:39 AM - Info bpdbm (pid=11399) done staging NBAZDB backup to /usr/openv/db/staging
Jan 16, 2016 2:36:39 AM - Info bpdbm (pid=11399) staging NBDB backup to /usr/openv/db/staging
Jan 16, 2016 2:36:42 AM - Error bpdbm (pid=11399) error staging NBDB backup to /usr/openv/db/staging
the requested operation was partially successful (1)
2.Now I've searched from bpdbm according to that PID.
I've listed here odd messages from there. Full messages is at the attachment.
02:36:39.554 [11399] <4> do_online_nbdb_backup: done staging NBAZDB backup to /usr/openv/db/staging
02:36:39.554 [11399] <4> do_online_nbdb_backup: Online backup of NBAZDB successful
02:36:39.554 [11399] <4> do_online_nbdb_backup: Server nbumedia, Policy NBU-Catalog, Schedule Full, parentJobID 1043339
02:36:39.554 [11399] <4> do_online_nbdb_backup: staging NBDB backup to /usr/openv/db/staging
02:36:39.569 [11399] <4> backup_dbspaces: Executing: backup database directory '/usr/openv/db/staging' transaction log truncate;
02:36:42.210 [11399] <16> backup_dbspaces: Execute SQL failed: backup database directory '/usr/openv/db/staging' transaction log truncate;
02:36:42.210 [11399] <16> backup_dbspaces: ErrMsg [Sybase][ODBC Driver][SQL Anywhere]Error during backup/restore: Error writing backup file "/usr/openv/db/staging/NBDB.log", ErrCode -1, Sqlstate HY000
02:36:42.223 [11399] <16> do_online_nbdb_backup: error staging NBDB backup to /usr/openv/db/staging
02:36:42.223 [11399] <16> do_online_nbdb_backup: Error completing online backup of: NBDB
02:36:57.826 [11399] <16> exec_catalog_backup: unexpected result from exec_asa_database_backup(): check_point is 0
02:36:57.827 [11399] <16> exec_catalog_backup: get_prev_catalog_backup(parentid=NBU-Catalog_1452897399_FULL) failed (227) (retry:2)
02:36:59.827 [11399] <16> exec_catalog_backup: get_prev_catalog_backup(parentid=NBU-Catalog_1452897399_FULL) failed (227) (retry:1)
02:37:01.828 [11399] <16> exec_catalog_backup: get_prev_catalog_backup(parentid=NBU-Catalog_1452897399_FULL) failed (227) (retry:0)
Thanks,
Turgun
01-15-2016 08:44 PM
01-16-2016 03:32 AM
1. Yes, As you mentioned the size of log file is 6.3GB.
6.3G Jan 16 15:16 NBDB.log
2. last all full backups are also failed. for approx. more than 1 week we have issue.
3. When I'm trying to backup and truncate getting error:
# /usr/openv/db/bin/nbdb_backup -dbn NBDB -online /nbdblog -truncate_tlog
Online backup of database to /nbdblog failed.
I've checked inside of /nbdblog directory. Created some files
# ls -lh /nbdblog/
total 229M
-rw-rw-rw-. 1 root root 26M Jan 16 15:20 DARS_DATA.db
-rw-rw-rw-. 1 root root 26M Jan 16 15:20 DARS_INDEX.db
-rw-rw-rw-. 1 root root 26M Jan 16 15:20 DBM_DATA.db
-rw-rw-rw-. 1 root root 26M Jan 16 15:20 DBM_INDEX.db
-rw-rw-rw-. 1 root root 66M Jan 16 15:20 EMM_DATA.db
-rw-rw-rw-. 1 root root 26M Jan 16 15:20 EMM_INDEX.db
-rw-rw-rw-. 1 root root 3.6M Jan 16 15:20 NBDB.db
-rw-rw-rw-. 1 root root 34M Jan 16 15:20 NBDB.log
But why failing and not truncating the transaction log ((.
# ls -lh /usr/openv/db/data/NBDB.log
-rw-------. 1 root root 6.4G Jan 16 15:31 /usr/openv/db/data/NBDB.log
01-16-2016 06:38 AM
01-16-2016 07:33 AM
Okay, Now there big backup data is processing. I will try it by tomorrow evening.
1.
@Marianne, there on Note 7, like this command :
/usr/openv/db/bin/dbeng9 -f /usr/openv/db/data/NBDB.db
But on my system,
# ls -lh /usr/openv/db/bin/dbeng*
-r-x--x---. 1 root daemon 65K Feb 4 2011 /usr/openv/db/bin/dbeng11
01-16-2016 08:37 AM
01-21-2016 02:07 AM
Hi All,
Thank you for your valuable feedback.
@ Marianne, your suggested link was very helpfull. (http://www.veritas.com/docs/000089830).
Thanks.
Turgun
01-21-2016 02:23 AM
PLEASE check OS patch levels on your NBU master and upgrade NBU as a matter of urgency....
You may need to upgrade hardware on your NBU master as well - NBU 7.5 and later needs more memory.
Please upgrade straight to 7.7.1 as all versions up to 7,6,x will reach EOSL in a year from now.
Use SORT to prepare for upgrade: https://sort.veritas.com/assessment/install_upgrade