cancel
Showing results for 
Search instead for 
Did you mean: 

Netbackup 7.5.0.6 Catalog recovery failing to restore.

snarayanaswamy
Level 3

We are trying to restore catalog on new host inorder to migrate to new host.

/usr/openv/netbackup/logs/user_ops/root/logs/Recover1570086112.log
 
00:20:53 (8.001) INF - TAR EXITING WITH STATUS = 0
00:20:53 (8.001) INF - TAR RESTORED 722 OF 78458 FILES SUCCESSFULLY
00:20:53 (8.001) INF - TAR KEPT 0 EXISTING FILES
00:20:53 (8.001) INF - TAR PARTIALLY RESTORED 77736 FILES
 
00:20:54 (8.001) Status of restore from copy 1 of image created Wed Oct  2 22:41:48 2019 = the requested operation was partially successful
 
00:20:54 ERR - Failed to execute command /usr/openv/netbackup/bin/bprestore -w -T -X -C tape001 -t 35 -p hot_catalog_backup_disk_oneoff -e 1570081308 -L "/usr/openv/netbackup/logs/user_ops/root/logs/Recover1570086112.log" / on host tape001 (1)
 
00:20:54 (8.xxx) INF - Status = the requested operation was partially successful.

 

Anyone faced similar issue, Please let me know if we need to try something else.

Thanks

Sreedhar 

14 REPLIES 14

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

@snarayanaswamy 

Please give us more info -
- Is hostname exactly the same as old host?
- Have you patched the installation to same 7.5.0.6 version before starting catalog recovery?
- Have you installed NBU to exact same path? 
- Can you please upload the Recover log here? (/usr/openv/netbackup/logs/user_ops/root/logs/Recover1570086112.log)

PS: 
Any reason why you are still on such an old, unsupported NBU version? 

 

- Is hostname exactly the same as old host? - Hostname is the same.
- Have you patched the installation to same 7.5.0.6 version before starting catalog recovery? - It is patched to 7.5.0.6 before attempting to perform catalog reboot.
- Have you installed NBU to exact same path?  - Yes
- Can you please upload the Recover log here? (/usr/openv/netbackup/logs/user_ops/root/logs/Recover1570086112.log)

Logs attached.

PS: 
Any reason why you are still on such an old, unsupported NBU version? 

  - We are migrated to 7.5 to 8.0, As there is no direct path for update to 8.1, we have go to 7.7.1 first and then to 8.1.

Thanks

Sreedhar 

We are running on Oracle  X7 h/w

# uname -a
Linux tape001 4.1.12-124.26.12.el6uek.x86_64 #2 SMP Wed May 8 22:12:22 PDT 2019 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/*release
Oracle Linux Server release 6.9
NAME="Oracle Linux Server"
VERSION="6.9"
ID="ol"
VERSION_ID="6.9"
PRETTY_NAME="Oracle Linux Server 6.9"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:6:9:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://bugzilla.oracle.com/"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 6"
ORACLE_BUGZILLA_PRODUCT_VERSION=6.9
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=6.9
Red Hat Enterprise Linux Server release 6.9 (Santiago)
Oracle Linux Server release 6.9

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

@snarayanaswamy 

Please double-check the installation path? 
On Linux, the default installation path is /usr/openv.
Please check contents of /usr - is openv a directory or symbolic link to /opt/openv?

We can see in the Recover log that the flat file catalogs were restored to /opt/openv.

In /usr we have openv symbolic link to /opt/openv

lrwxrwxrwx 1 root root 10 Apr 10 21:59 openv -> /opt/openv

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

@snarayanaswamy 

Please show us contents of these 2 files that were recovered first:

00:02:01 (8.001) /opt/openv/netbackup/bp.conf.bkp
00:02:01 (8.001) /opt/openv/volmgr/vm.conf

I am also wondering if this was a clean NBU installation on a new server ?
I see entries like these all over the log:

Directory /opt/openv/netbackup/xxx/xxx already exists.

Please also tell us the exact command that was used to recover the catalog. 
I do not see any reference in the log to recover NBDB.

$ cat vm.conf
ENABLE_AUTOPATH_CORRECTION
MEDIA_ID_BARCODE_CHARS = 0 8 1:2:3:4:5:6
DEVICE_HOST = tape001
VERBOSE
MAP_CONTINUE_TIMEOUT = 900
AUTO_UPDATE_ROBOT

$ less bp.conf.bkp

##########################################################
##
## This file is pushed to this server via puppet.
## any changes you make here will be overwritten
##
###########################################################
# updated by: netbackup module
#

CLIENT_NAME = tape001
CLIENT_NAME = tape001.svale.netledger.com
EMMSERVER = tape001
EMMSERVER = tape001.svale.netledger.com
OPS_CENTER_SERVER_NAME = opscenter.adm.svale.netledger.com

SERVER = tape001
SERVER = tape001x
SERVER = tape001.svale.netledger.com
SERVER = tape001x.svale.netledger.com

SERVER = tape-med001.adm
SERVER = tape-med001x.adm
SERVER = tape-med001.adm.svale.netledger.com
SERVER = tape-med001x.adm.svale.netledger.com

SERVER = tape-med002.adm
SERVER = tape-med002x.adm
SERVER = tape-med002.adm.svale.netledger.com
SERVER = tape-med002x.adm.svale.netledger.com

SERVER = tape-med003.adm
SERVER = tape-med003x.adm
SERVER = tape-med003.adm.svale.netledger.com
SERVER = tape-med003x.adm.svale.netledger.com

SERVER = tape-med004.adm
SERVER = tape-med004x.adm
SERVER = tape-med004.adm.svale.netledger.com
SERVER = tape-med004x.adm.svale.netledger.com

SERVER = tape-med005.adm
SERVER = tape-med005x.adm
SERVER = tape-med005.adm.svale.netledger.com
SERVER = tape-med005x.adm.svale.netledger.com

SERVER = tape-med006.adm

MEDIA_SERVER = tape-med006x.adm
MEDIA_SERVER = tape-med006.adm.svale.netledger.com
MEDIA_SERVER = tape-med006x.adm.svale.netledger.com


ALLOW_MEDIA_OVERWRITE = TAR
#VERBOSE = 5
#CRYPT_KIND = NONE
MEGABYTES_OF_MEMORY = 2048
VXDBMS_NB_DATA = /usr/openv/db/data
# BPDBM_VERBOSE = 5
USE_VXSS = PROHIBITED
CRYPT_OPTION = DENIED

MPX_RESTORE_DELAY = 120
INCOMPLETE_JOB_CLEAN_INTERVAL = 2
INCOMPLETE_BKUP_JOB_CLEAN_INTERVAL = 6
NFS_ACCESS_TIMEOUT = 10
RE_READ_INTERVAL = 900
MEDIA_UNMOUNT_DELAY = 0
SERVER_CONNECT_TIMEOUT = 60
CLIENT_CONNECT_TIMEOUT = 900
CLIENT_READ_TIMEOUT = 1800
FORCE_RESTORE_MEDIA_SERVER = tape-med006x.adm.svale.netledger.com tape-med004x.adm.svale.netledger.com
LIST_FILES_TIMEOUT = 600
TELEMETRY_UPLOAD = NO
(END)

/usr/openv/netbackup/bin/admincmd/bprecover -wizard ==> command used to recover the catalog db.

While checking OS compatibility for 7.5.0.6, We see RH 6.9 is not supported, Can this cause 

any issue with cat restore. 

I did fresh install, didn't get any errors at the time of installation.

Thanks

Sreedhar 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

It would be nice if you could show us all output and your answers to the questions that followed the 'bprecover -wizard' command. 

Did you see these questions? 

Do you want to recover the entire NetBackup catalog? (Y/N)

Do you want to startup the NetBackup relational database (NBDB) after the recovery?(Y/N)

As per my previous post - I did not see any attempt to recover NBDB. 

About your choice of OS - I was wondering about why you chose Oracle Linux. 
I know that support for this OS was dropped at some point in time. 

Please check 7.x and 8.x OS SCLs and ask your management for new server with OS that is supported for NBU master server. 

By the way - you can upgrade directly to 8.0 from 7.5.x.
Actually - you can upgrade to 8.1.x. There is just no documentation for this upgrade. You will need to consult both 7.7 and 8.1.x upgrade documentation. 

See 8.x Upgrade portal :
https://www.veritas.com/support/en_US/article.000115678

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Something else that is worrying me - this is what I see in the bp.conf version that was restored:

VXDBMS_NB_DATA = /usr/openv/db/data

I am wondering why the path would be /usr/openv if NBU was installed in /opt/openv ?

@Marianne Here is wizard output.

# sudo /usr/openv/netbackup/bin/admincmd/bprecover -wizard

Welcome to the NetBackup Catalog Recovery Wizard!

Please make sure the devices and media that contain catalog disaster recovery
data are available
Are you ready to continue?(Y/N)
Y

Please specify the full pathname to the catalog disaster recovery file:
/gfsnet1/tape001_dsu/dr_file/hot_catalog_backup_disk_oneoff_1567602621_FULL
tape001_1567602621
All media resources were located

Do you want to recover the entire NetBackup catalog? (Y/N)
Y

Do you want to startup the NetBackup relational database (NBDB) after the recovery?(Y/N)
Y

Catalog recovery is in progress. Please wait...
EXIT STATUS 1: the requested operation was partially successful
ERR - Failed to execute command /usr/openv/netbackup/bin/bprestore -w -T -X -C tape001 -t 35 -p hot_catalog_backup_disk_oneoff -e 1567602621 -L "/usr/openv/netbackup/logs/user_ops/root/logs/Recover1568094993.log" / on host tape001 (1)
Failed to recover catalog (1)

 

Even though path is show /usr/opev/db/data this is simbolic link to /opt/openv.

From the log of bprecover -wizard command we see it is existing at tar. 

Let me recheck on upgrade documention on available upgrade path for 7.5.0.6 to either 7.7 or 8.0.

We have currently Netbackup 7.5.0.6 running on oracle linux 6.6. Not sure if downgrading the current server to 6.6 will help.

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

@snarayanaswamy 

I honestly don't know what else to suggest. 
It seems that nbdb is not attempted because of Status 1 for the flat file restore.

I'm trying to find errors in the Recover log  I don't see anything that can be seen as failure.

All of these seem 'normal':  LOTS of these:
/opt/openv/xx/xxx/ is not in the true image list. Skipping.

And then LOTS of these :
Directory /opt/openv/var/xxx/xxx already exists.
We should not see this if you did a new, clean installation.

But none of theses look like errors, rather 'Info' messages, because tar exists with status 0.

00:20:53 (8.001) INF - TAR EXITING WITH STATUS = 0
00:20:53 (8.001) INF - TAR RESTORED 722 OF 78458 FILES SUCCESSFULLY
00:20:53 (8.001) INF - TAR KEPT 0 EXISTING FILES
00:20:53 (8.001) INF - TAR PARTIALLY RESTORED 77736 FILES

There should be something else in another log that caused the status 1 (partially successful)

00:20:54 (8.001) Status of restore from copy 1 of image created Wed Oct 2 22:41:48 2019 = the requested operation was partially successful

This is the problem with waiting way too long to upgrade. 
All support for your NBU version ended on February 1, 2017.

What you really need is to have a Veritas Support engineer to assist you to dig into logs.

Your OS is still supported up to NBU 7.6.x.
Page 38 of the 7.6 OS SCL shows that Oracle Linux 6 is supported as from NBU 7.1. 
(Or maybe not, as I see that for the RHEL equivalent, support for version 6.8 started from NBU 7.6.0.4)

Your real problem is that NO version of Oracle Linux is supported as Master server as from NBU 8.x. 

So, this is my suggestion: 

Speak to your management to purchase a Linux OS version that is supported by both NBU 7.x and 8.x. 
(Something lower than Red Hat Enterprise Linux 6 update 8).

Try the catalog recovery again on RHEL. 

If the catalog recovery still fails, and you have active maintenance with Veritas, then log a Support call with a topic that looks like this:

'Need assistance with upgrade to NBU 8.0'.

Hopefully they will accept the call when you explain that you first need to migrate to a supported OS before you can upgrade to 8.x. 

@Marianne 

Thanks for assistance, Issue was /opt/openv was xfs, Had to recreate filesystem with ext4 and it worked.

We are downgrading our OS to 6.6, as we have netbackup working fine. 

Thanks for all your assistance.

-Sreedhar 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Please remember that you need to migrate to RHEL (or another supported Linux OS) before you upgrade to NBU 8.x.