Restore after disk pool crash

[Seburau]

I've lost deduplication disk pool (RAID group crash). The system disk is OK. I've built new RAID group and mount it to server. Now I can't see catalog, policies, clients and etc. There was a policy for catalog backup to tape. I tried recover the catalogs wizards, but I have not disaster recovery file. I see a media in CatalogBackup pool. How can I restore the catalog, policies and etc from tape?

I have tried import image but in results page I see "Import not allowed. cannot connect on socket (25)".

What additional Information can I provide?

[Seburau]

I tried this command on master server:

nbu-s1:~ # /usr/openv/netbackup/bin/bpclntcmd -pn
nbu-s1:~ #

But I not see expected information. Any ideas?

[Marianne]

There is something very wrong here - loss of dedupe pool should NOT result in loss of your catalogs, unless your NBU catalogs were configured on the same drive letter / location as the dedupe pool. 
This goes against every single NBU document.

If you added email notification to NBU Catalog policy, you will have emails that contain all relevant info for Catalog recovery - The media-id that was used, the steps for Catalog recovery plus the DR-file as attachment.
Dr-file should also be stored on a different volume or network share so that the DR-file is not lost when catalogs are lost.

You can recreate a DR-file using this TN: 
https://www.veritas.com/support/en_US/article.100037884

[Amol_Nair]

import not allowed. Cannot connect on socket (25)

Looking at this error I would say that you seem to have nbac configured on the master server.. Are you sure catalog, policies are actually missing and not something like within nbac configuration or auth.conf settings your user does not have sufficient rights to list stuff..

Do you see any folders present under the netbackup\db\class folder and in netbackup\db\images folder

[Seburau]

Thank you Marianne for answer.

This Configuration was done by other person. I cannot reach him now. Dedup pool was configured to folder /dedup. Is /usr/openv/netbackup/db usually catalog folder? /usr/openv/netbackup/db was a symbol link to folder /dedup/db. So I have lost /usr/openv/netbackup/db too.

I have not receive any notifications from this system.

Now there is volume pool "CatalogBackup". And there is one volume in this volume pool. I hope this volume contains backup of the catalog.

I have tried to recreate DR-file:

nbu-s1:~ # /usr/openv/netbackup/bin/admincmd/bpimport -drfile -id 7012L6 -drfile_dest /usr/openv/netbackup/dr
import not allowed: cannot connect on socket (25)
import not allowed: cannot connect on socket (25)

[Seburau]

Thank you Amol_Nair for your answer.

I have root permissions for this server.

nbu-s1:~ # ls -al /usr/openv/netbackup/db
lrwxrwxrwx 1 root root 9 Июл 22 2015 /usr/openv/netbackup/db -> /dedup/db

nbu-s1:~ # ls -al /dedup/db
drwxr-xr-x 5 root root 96 Май 14 11:01 .
drwxr-xr-x 5 root root 96 Май 11 17:28 ..
drwxr-xr-x 2 root root 8192 Май 15 14:59 discovery
-rw-r--r-- 1 root root 2 Май 11 17:34 IDIRSTRUCT
drwxr-xr-x 2 root root 96 Май 11 17:34 images
drwxr-xr-x 4 root root 96 Май 11 18:05 jobs

nbu-s1:~ # ls -al /dedup/db/images/
drwxr-xr-x 2 root root 96 Май 11 17:34 .
drwxr-xr-x 5 root root 96 Май 14 11:01 ..

[Marianne]

 /usr/openv/netbackup/db was a symbol link to folder /dedup/db. So I have lost /usr/openv/netbackup/db too.

Worst possible mistake.... Hope you have not paid this guy.....  
Email notification must be configured - it does not happen automatically.

Please check if all NBU daemons/processes are running:

bpps -x 

[Seburau]

nbu-s1:~ # /usr/openv/netbackup/bin/bpps -x
NB Processes
------------
root 9064 41469 0 16:01 ? 00:00:00 /usr/openv/netbackup/bin/admincmd/bpstsinfo -DPSPROXY
root 10128 41280 0 16:07 ? 00:00:00 /usr/openv/netbackup/bin/bpcompatd
root 40830 1 0 May14 ? 00:00:18 /usr/openv/netbackup/bin/private/nbatd -c /usr/openv/var/global/vxss/eab/data
root 40838 1 0 May14 ? 00:00:00 /usr/openv/netbackup/bin/vnetd -standalone
root 40843 1 0 May14 ? 00:00:00 /usr/openv/netbackup/bin/bpcd -standalone
root 40883 1 0 May14 ? 00:00:41 /usr/openv/netbackup/bin/nbdisco
root 40968 1 1 May14 ? 00:14:33 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 5
root 41074 1 0 May14 ? 00:00:10 /usr/openv/netbackup/bin/nbevtmgr
root 41109 1 0 May14 ? 00:00:10 /usr/openv/netbackup/bin/nbaudit
root 41214 1 0 May14 ? 00:00:55 /usr/openv/netbackup/bin/nbemm
root 41223 1 0 May14 ? 00:00:09 /usr/openv/netbackup/bin/nbrb
root 41280 1 0 May14 ? 00:00:05 /usr/openv/netbackup/bin/bpcompatd
root 41293 1 0 May14 ? 00:00:13 /usr/openv/netbackup/bin/nbjm
root 41469 1 0 May14 ? 00:00:26 /usr/openv/netbackup/bin/nbrmms
root 41517 1 0 May14 ? 00:00:44 /usr/openv/netbackup/bin/nbsl
root 41554 1 0 May14 ? 00:00:06 /usr/openv/netbackup/bin/nbim
root 41584 1 0 May14 ? 00:00:50 /usr/openv/netbackup/bin/nbars
root 41649 1 0 May14 ? 00:00:02 /usr/openv/netbackup/bin/bmrd
root 41690 41293 0 May14 ? 00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbjm
root 41701 1 0 May14 pts/2 00:00:04 /usr/openv/netbackup/bin/nbcssc -a NetBackup
root 41727 1 0 May14 ? 00:00:10 /usr/openv/netbackup/bin/nbsvcmon
root 41747 1 0 May14 ? 00:00:02 /usr/openv/netbackup/bin/bmrbd
root 47564 1 0 May14 ? 00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 en_US /usr/openv/java/auth.conf 1 -1 hpsim.irro.ru 192.168.7.30 47559 noUserCredentialsFile
root 47566 47564 0 May14 ? 00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 en_US /usr/openv/java/auth.conf 1 -1 hpsim.irro.ru 192.168.7.30 47559 noUserCredentialsFile
root 47569 47564 0 May14 ? 00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 en_US /usr/openv/java/auth.conf 1 -1 hpsim.irro.ru 192.168.7.30 47559 noUserCredentialsFile
root 47766 41517 0 May14 ? 00:00:00 /usr/openv/netbackup/bin/nbproxy dblib -mgrIORFile -PolicyManager-2-1-1526303144.ior.mgr -proxyID PolicyManager-2-1-1526303144
root 47780 41517 0 May14 ? 00:00:00 /usr/openv/netbackup/bin/nbproxy dblib -mgrIORFile -StorageService-2-2-1526303153.ior.mgr -proxyID StorageService-2-2-1526303153

MM Processes
------------
root 41237 1 0 May14 pts/2 00:00:03 /usr/openv/volmgr/bin/ltid
root 41250 1 0 May14 pts/2 00:00:02 vmd

Shared Symantec Processes
-------------------------
root 7832 1 0 May14 ? 00:00:01 /opt/VRTSpbx/bin/pbx_exchange

[Marianne]

Neither bprd nor bpdbm are running. Probably because the subfolders under /usr/openv/netbackup/db are gone. 

I am not sure if creating empty folders will be enough, but probably worth a try.

Please remove the symlink and create the db folder in /usr/openv/netbackup/.
Create all of these folders in db:

• class
• client
• config
• error 
• images
• jobs 
• media

In images folder, create empty touch-file: db_marker.txt 

Create bprd and bpdbm log folders under /usr/openv/netbackup/logs.
Stop and restart NBU.

If bprd and bpdbm are still not starting, look at the relevant logs for clues.

Probably best not to spend too much time on this and rather uninstall NBU and re-install. 
You should be able to recreate the dr-file and then recover the catalogs.

PS:
Were all of your MSDP backups duplicated to tape?
If so, Catalog recovery will be good. Further steps will be needed to remove the catalog info for dedupe backups so that only information about tape copies will be left.

[Seburau]

Thank you Marianne.

Please remove the symlink and create the db folder in /usr/openv/netbackup/.

Done.

Stop and restart NBU.

Done.

bprd and bpdbm have started.

Further,

/usr/openv/netbackup/bin/admincmd/bpimport -drfile -id 7012L6 -drfile_dest /usr/openv/netbackup/dr
Import phase 1 started 05/16/2018 10:40:28
INF - Create DB information for media id 7012L6.
INF - Initiation of bptm process to phase 1 import media id 7012L6 was successful.
Import phase 2 started 05/16/2018 10:40:28
INF - Found no images matching the selection criteria that were ready for phase 2 import.

INF - Copying DR files to /usr/openv/netbackup/dr on master server 'nbu-s1'.
INF - Import of DR files from 7012L6 to /usr/openv/netbackup/dr on master server 'nbu-s1' was successful.

I have found a few files in /usr/openv/netbackup/dr. In catalog recovery wizard I entered the name of the last of these files. But recovery has finished with error. In bpdm log I see:

15:56:34.613 [64824] <4> report_client: VBRC 2 64824 0 nbu-s1_1525748864 -1 *NULL* -1 *NULL* 0 1 1
15:56:34.902 [64824] <32> bp_sts_open_image: sts_open_image failed: error 2060013
15:56:34.902 [64824] <32> io_open_disk: image open failed: error 2060013:
15:56:34.902 [64824] <16> read_data_tir: ERR - Cannot open TIR file /dedup/disk/nbu-s1_1525748864_C1_TIR.

DR file contains such line:

# FRAG: c# f# K rem mt den fn id/path host bs off md dwo f_flags desc exp mpx rl chkpt rsm_nbr seq_no media_subtype keep_date copy_date fragment_state data_format slp_index_num resumable expiration_count copy_type
FRAGMENT 1 -1 1415 665 0 0 0 /dedup/disk/nbu-s1_1525748864_C1_TIR nbu-s1 524288 0 0 -1 0 nbu-s1 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0
FRAGMENT 1 1 72562784 0 0 0 0 /dedup/disk/nbu-s1_1525748864_C1_F1 nbu-s1 524288 0 0 -1 0 nbu-s1 1526958464 0 65537 0 0 0 1 0 1525751378 1 1 0 0 0 0
FRAGMENT 2 -1 1416 0 2 20 20 7012L6 nbu-s1 524288 928640 1525180327 1 0 *NULL* 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0
FRAGMENT 2 1 72562784 0 2 20 19 7012L6 nbu-s1 524288 786913 1525180327 1 0 *NULL* 1528427264 0 65539 0 0 0 1 0 1525751378 1 1 0 0 0 0
#DR_MEDIA_REC: ver rtype mtype host c# p# policies# policies kbytes nimg vimg hsize off subtype client_type sched_type run_time id/path frag_id media_desc bcode den
DR_MEDIA_REC 1 0 0 nbu-s1 1 1 0 0 0 0 0 0 1 7 0 1525748503 /dedup/disk/nbu-s1_1525748503_C1_F1 nbu-s1 *NULL* 0
DR_MEDIA_REC 1 0 2 nbu-s1 2 1 0 402875025 18 18 1024 786913 1 7 0 1525748503 7012L6 *NULL* 7012L6 20
DR_MEDIA_REC 1 0 0 nbu-s1 1 1 0 0 0 0 0 0 1 35 0 1525748864 /dedup/disk/nbu-s1_1525748864_C1_F1 nbu-s1 *NULL* 0
DR_MEDIA_REC 1 0 2 nbu-s1 2 1 0 402875025 18 18 1024 786913 1 35 0 1525748864 7012L6 *NULL* 7012L6 20
DR_MEDIA_REC 1 0 0 nbu-s1 1 1 0 0 0 0 0 0 1 35 0 1525748864 /dedup/disk/nbu-s1_1525748864_C1_TIR nbu-s1 *NULL* 0

I see on volume 7012L6 one image nbu-s1_1525180327. Can I edit DR file so that it has only lines with tape ID?

[Marianne]

Try :
bprecover -wizard -copy 2

See: https://www.veritas.com/support/en_US/article.100000915

[Seburau]

nbu-s1:/usr/openv/netbackup/dr # /usr/openv/netbackup/bin/admincmd/bprecover -wizard -copy 2

Welcome to the NetBackup Catalog Recovery Wizard!

Please make sure the devices and media that contain catalog disaster recovery
data are available
Are you ready to continue?(Y/N)
Y

Please specify the full pathname to the catalog disaster recovery file:
/usr/openv/netbackup/dr/nbu-s1-catalog_1525748864_FULL
nbu-s1_1525748864
All media resources were located

Do you want to recover the entire NetBackup catalog? (Y/N)
Y

Do you want to startup the NetBackup relational database (NBDB) after the recovery?(Y/N)
Y

Catalog recovery is in progress. Please wait...
Gathering configuration information.
Waiting for the security services to start operation.
Generating identity for host 'nbu-s1'
Setting up security on target host: nbu-s1
nbatd is successfully configured on NetBackup Master Server.
Operation completed successfully.

Beginning recovery of NBDB. Please wait...
ERR - Failed to recover NBDB on nbu-s1 (5)
Failed to recover catalog (5)

[Marianne]

Please locate the Recover log in this path:
/usr/openv/netbackup/logs/user_ops/root/logs/Recover<date-time>.log

Copy to recover.txt and upload here. 

[Seburau]

I have uploaded this file.

Looking through the file, I noticed that the DB files are being restored to the directory /dedup/db. Maybe I can stop Netbackup, copy files from /dedup/db to /usr/openv/netbackup/db and start netbackup?

[Marianne]

Rather create a symbolic link the other way round (to what it was before) and rerun the catalog restore.
(That is what I would do.)

cd /dedup
mv db db-old   (you can delete this later)
ln -s db /usr/openv/netbackup/db

Please ensure that /usr/openv is on a partition with sufficient space. 

It should NEVER again be on the same partition as Dedupe storage.