NetBackup Catalog Recover

Didi7 · ‎08-25-2022

Hello, I am trying to recover a NetBacukp catalog from an old server on a new one with another OS but same name and different IP address.

I get the following error, when I chose the NetBackup Catalog Recovery Wizard right after successful installation of NetBackup 8.2 to a Windows Server 2016 virtual machine.

The output of the log D:\Veritas\NetBackup\logs\user_ops\SERVERNAME\Administrator\logs is only 2 bytes in size.

Any idea?

quebek · ‎08-25-2022

Hello

Please create log folders in <installdir>\netbackup\logs by running mklogdir.bat file in the same dir.
post the out comes from activity monitor for this failed job - check also the demons logs for pids mentioned in the activity monitor - you can post these logs too. If log folders were not existing earlier on please rerun the job - also you might want to enable verbosity to lvl 5.

Didi7 · ‎08-25-2022

Hi quebek, here is the outcome from the Activity Monitor after running 'mklogdir.bat' ...

Aug 25, 2022 11:03:20 AM - Info bpdm (pid=8072) started
Aug 25, 2022 11:03:20 AM - started process bpdm (pid=8072)
Aug 25, 2022 11:03:20 AM - Info bpdm (pid=8072) reading backup image
Aug 25, 2022 11:03:20 AM - Critical bpdm (pid=8072) sts_get_lsu_prop_byname on LSU E:\NBU_Catalog_Backup failed: 2060013 no more entries
Aug 25, 2022 11:03:20 AM - Critical bpdm (pid=8072) Invalid storage device: E:\NBU_Catalog_Backup no more entries
Aug 25, 2022 11:03:20 AM - Critical bpdm (pid=8072) failure to open disk at path E:\NBU_Catalog_Backup: plug-in reports error 2060013
Aug 25, 2022 11:03:20 AM - Error bpdm (pid=8072) ERR - Cannot open TIR file E:\NBU_Catalog_Backup\OLDSERVERNAME_1635493171_C1_TIR.
media open error (83)

Didi7 · ‎08-25-2022

And here is the outcom from <installdir>\logs\bpdm and the underlying log-file ...

11:03:20.438 [8072.7264] <2> bpdm: INITIATING (VERBOSE = 0): -restoretir -jobid 10 -cn 1 -b OLDSERVERNAME_1635493171 -c OLDSERVERNAME -priority 90000
11:03:20.454 [8072.7264] <2> job_connect: Set SO_KEEPALIVE on socket.
11:03:20.454 [8072.7264] <2> logconnections: BPJOBD CONNECT FROM 172.21.18.61.56982 TO 172.21.18.61.13723 fd = 652
11:03:20.454 [8072.7264] <2> job_authenticate_connection: VxSS authentication check for now...
11:03:20.469 [8072.7264] <2> job_connect: Connected to the host OLDSERVERNAME contype 53 jobid <10> socket <652>
11:03:20.469 [8072.7264] <2> job_connect: Connected on port 56982
11:03:20.469 [8072.7264] <4> JobdSockList::RegisterSocket: Register socket (652).
11:03:20.469 [8072.7264] <4> report_client: VBRC 3 8072 64 OLDSERVERNAME_1635493171 -1 *NULL* -1 *NULL* 0 1 1
11:03:20.485 [8072.7264] <2> ConnectionCache::connectAndCache: Acquiring new connection for host OLDSERVERNAME, query type 81
11:03:20.485 [8072.7264] <2> logconnections: BPDBM CONNECT FROM 172.21.18.61.56983 TO 172.21.18.61.13721 fd = 672
11:03:20.516 [8072.7264] <2> db_end: Need to collect reply
11:03:20.516 [8072.7264] <2> closeConnection: Caching connection for query type 81 for reuse
11:03:20.594 [8072.7264] <16> 10:bptm:8072:OLDSERVERNAME: libsts openp() 22/08/25 11:03:20: stsm_open_module D:\Veritas\NetBackup\bin\\ost-plugins\libstspizapi.dll failed err 2060053 platerr 126: LoadLibrary failed
11:03:20.594 [8072.7264] <2> bp_sts_get_lsu: entry E:\NBU_Catalog_Backup
11:03:20.594 [8072.7264] <32> bp_sts_get_lsu: sts_get_lsu_prop_byname on LSU E:\NBU_Catalog_Backup failed: 2060013
11:03:20.594 [8072.7264] <32> bp_sts_open_disk: Invalid storage device: E:\NBU_Catalog_Backup
11:03:20.594 [8072.7264] <32> io_open_disk: failure to open disk at path E:\NBU_Catalog_Backup: plug-in reports error 2060013
11:03:20.594 [8072.7264] <16> read_data_tir: ERR - Cannot open TIR file E:\NBU_Catalog_Backup\OLDSERVERNAME_1635493171_C1_TIR.
11:03:20.594 [8072.7264] <4> JobdSockList::UnregisterSocket: Unregister socket (652).
11:03:20.641 [8072.7264] <2> job_monitoring_exex: ACK disconnect
11:03:20.641 [8072.7264] <2> job_disconnect: Disconnected
11:03:20.641 [8072.7264] <2> bpdm: EXITING with status 83

Didi7 · ‎08-25-2022

And where do I enable verbosity to lvl 5 ?

quebek · ‎08-25-2022

Hey

Where this catalog backup was written to? ie what storage unit msdp, basic or advanced disk?

Does this folder exist E:\NBU_Catalog_Backup

are there all files needed to recover NBU catalog.

About verbosity - open java gui/Netbackup management/host properties/master servers double click on your master then in logging set global logging level to 5 (maximum).

Didi7 · ‎08-25-2022

Hello quebek,

the catalog was written to E:\NBU_Catalog_Backup, the DR-files were written to a different directory on 
drive E:\ and to be honest currently I cannot tell you to which storage unit it was written. The original 
is available as a virtual machine. We used MSDP and/or basic or advanced disk.

Of course the folder E:\NBU_Catalog_Backup exists. I have several catalog and dr files from different dates.

Is it possible to recover a catalog, without configuring the orignal storage units etc. at first?

quebek · ‎08-25-2022

Hi

if it was written to basic disk then nothing needs to be configured. But this restore looks like it is willing to engage some libraries for either advanced or msdp - I just cant tell. In such case I think the catalog recovery won't be so easy.

for msdp check this out https://www.veritas.com/support/en_US/article.100049485

maybe you can check in recovery file what was used... ie

The storage server attributes must match those of the original storage server as documented in the Catalog backup's disaster recovery file. To determine these attributes, open the DR file and examine the fragment records as shown in the following example:

# FRAG: c# f# K rem mt den fn id/path host bs off md dwo f_flags f_unused1 exp mpx rl chkpt rsm_nbr seq_no media_subtype keep_date copy_date i_unused1
FRAGMENT 1 -1 38 970 0 39 0 @aaaa8 dorna 262144 0 0 -1 0 1;PureDisk;dorna;DiskPool1;dv02;0 0 0 0 0 0 0 6 0 0 0
FRAGMENT 1 1 2912 0 0 0 0 @aaaa8 dorna 262144 0 0 -1 0 1;PureDisk;dorna;DiskPool1;dv02;01226440313 0 65537 0 0 0 6 0 225230725 0

Didi7 · ‎08-25-2022

Hello quebek,

this is an excerpt of the dr file ...

# FRAG: c# f# K rem mt den fn id/path host bs off md dwo f_flags desc exp mpx rl chkpt rsm_nbr seq_no media_subtype keep_date copy_date fragment_state data_format slp_index_num resumable expiration_count copy_type
FRAGMENT 1 -1 9534 494 0 0 0 E:\NBU_Catalog_Backup\OLDSERVERNAME_1635493171_C1_TIR OLDSERVERNAME 1048576 0 0 -1 0 OLDSERVERNAME 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0
FRAGMENT 1 1 51200000 0 0 0 0 E:\NBU_Catalog_Backup\OLDSERVERNAME_1635493171_C1_F1 OLDSERVERNAME 1048576 0 0 -1 0 OLDSERVERNAME 1636097971 0 65536 0 0 0 1 0 1635494147 1 1 0 0 0 0
FRAGMENT 1 2 25816005 0 0 0 0 E:\NBU_Catalog_Backup\OLDSERVERNAME_1635493171_C1_F2 OLDSERVERNAME 1048576 0 0 -1 0 OLDSERVERNAME 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0
#DR_MEDIA_REC: ver rtype mtype host c# p# policies# policies kbytes nimg vimg hsize off subtype client_type sched_type run_time id/path frag_id media_desc bcode den
DR_MEDIA_REC 1 0 0 OLDSERVERNAME 1 1 0 0 0 0 0 0 1 35 0 1635493171 E:\NBU_Catalog_Backup\OLDSERVERNAME_1635493171_C1_F2 OLDSERVERNAME *NULL* 0
DR_MEDIA_REC 1 0 0 OLDSERVERNAME 1 1 0 0 0 0 0 0 1 7 0 1635493146 E:\NBU_Catalog_Backup\OLDSERVERNAME_1635493146_C1_F1 OLDSERVERNAME *NULL* 0
DR_MEDIA_REC 1 0 0 OLDSERVERNAME 1 1 0 0 0 0 0 0 1 35 0 1635493171 E:\NBU_Catalog_Backup\OLDSERVERNAME_1635493171_C1_F1 OLDSERVERNAME *NULL* 0
DR_MEDIA_REC 1 0 0 OLDSERVERNAME 1 1 0 0 0 0 0 0 1 35 0 1635493171 E:\NBU_Catalog_Backup\OLDSERVERNAME_1635493171_C1_TIR OLDSERVERNAME *NULL* 0

Didi7 · ‎08-25-2022

Furthermore, what does it mean, when the 'NetBackup Catalog Recovery Wizard' reports ...

OLDSERVERNAME_1635493171
All media resources were located

Does it mean, that catalog files and dr files are complete and ready for Recovery?

Loglevel is at 5 now!

quebek · ‎08-25-2022

Interesting - looks like basic disk

maybe there are some missing ACLs for this folder... not sure...

Open a case with VRTS.

Didi7 · ‎08-25-2022

How can that be? ACLs missing? I am local administrator to that machine, it is not part of any domain, it's just workgroup!

Didi7 · ‎08-25-2022

Does this say anything to you?

Aug 25, 2022 2:45:22 PM - Info bpdm (pid=4984) started
Aug 25, 2022 2:45:22 PM - started process bpdm (pid=4984)
Aug 25, 2022 2:45:22 PM - Info bpdm (pid=4984) reading backup image
Aug 25, 2022 2:45:23 PM - Critical bpdm (pid=4984) sts_get_lsu_prop_byname on LSU E:\NBU_Catalog_Backup failed: 2060013 no more entries
Aug 25, 2022 2:45:23 PM - Critical bpdm (pid=4984) Invalid storage device: E:\NBU_Catalog_Backup no more entries
Aug 25, 2022 2:45:23 PM - Critical bpdm (pid=4984) failure to open disk at path E:\NBU_Catalog_Backup: plug-in reports error 2060013 
Aug 25, 2022 2:45:23 PM - Error bpdm (pid=4984) ERR - Cannot open TIR file E:\NBU_Catalog_Backup\desg-dc2wi195_1635493171_C1_TIR.
media open error  (83)

Do I need to configure anything in advance, before I can restore the catalog? Why is this so complex? I have the catalog and dr files and according to the 'NetBackup Catalog Recovery Wizard' I have all media resources as the wizard reports ...

OLDSERVERNAME_1635493171
All media resources were located

quebek · ‎08-25-2022

Well NBU is not trivial thing :)

so this file do exist, right?

E:\NBU_Catalog_Backup\desg-dc2wi195_1635493171_C1_TIR

If so why we see this

ERR - Cannot open TIR file

ACL ?? try to all everyone RW to this folder and its children...

Again open a case... they should be able to jump on zoom session with you and possibly sort it out within minutes ;)

quebek · ‎08-25-2022

one more to do - add to etc\hosts entry for this box its IP and its hostname - as it was in original setup (hostname only you said IP did change) so like this

current_IP oldhostname oldhostname_fqdn

Didi7 · ‎08-25-2022

Hello quebek,

Well NBU is not trivial thing :) I know ;)

so this file do exist, right?

E:\NBU_Catalog_Backup\desg-dc2wi195_1635493171_C1_TIR

My bad! The directory name was wrong (_Backup was missing). I must have been blind! So no ACL problems ;)

The catalog restore is working currently. Last time I did this test, it didn't finish though. Let's see, what happens this time.

quebek · ‎08-25-2022

Hey

yea put your glasses on ;) hahaha

good luck...

Didi7 · ‎08-25-2022

Sorry, I could not read your last post. I had to copy your text and convert it into audio :p

*********************************************************************

Seriouly, after many tests this is the first time the catalog was successfully recovered. Now, I can write my own docu and continue with tape drives, so that I will be able to recover VMs into my hypervisor environment.

So far I learned ...

0) Open your eyes or clean your glasses and make sure directory names are correct ;)
1) Using the same server name as the old server is helpful
2) Using the same NetBackup <installdir> as on the old server is helpful

In former tests I used a different server name but also named the old server in the local HOSTS file, but I also installed NetBackup in the default path and tried to trick the 'NetBackup Catalog Recovery Wizard' by manually creating the d:\veritas\netbackup\db\images path etc.

I will have some more tests now but nevertheless thanks for pointing me in the right direction.

Appreciated.

Didi7 · ‎08-25-2022

Ok, I got a lot of things in place, attached a new VMware vCenter to this new Master Server and tried to restore a virtual machine, but it failed telling me ...

NetBackup VMware policy restore error (2820)

Before decommissioning the old Master Server a colleague of me deleted a lot of policies no longer used and only a handful of policies left over. The backup of the selected VM was done with a policy, which no longer exists, even with recovering the catalog.

Do I need to recreate a policy, when it no longer exists to restore a virtual machine into a VMware vCenter environment? Sounds strange to me.

Further note: I must admit this is still a virtual machine and all those tape devices and physical Media Servers (reconfigured by recovering the catalog) do no longer exist. I have to re-create a new tape device configuration for a much smaller tape library, as the original bigger tape library also no longer exists. So this Master Server will be also Media Server in one machine, when this whole environment will be re-installed on a physical server. NetBackup even tells me, which tape media is necessary to restore the virtual machine, but of course NetBackup has no access to this tape media currently, as the new server is only a virtual machine and the old physical media servers no longer exist.

VOX

NetBackup Catalog Recover