Migrating NBU 7.1.0.4 from Solaris to RHEL 5.7

[Denz1217]

Hi All,

We've attempted to move our NBU 7.1.0.4 from a T2000 Sparc to a HP (intel based) RHEL 5.7. Below is the procedure we took.

* From the old solaris, we performed DB catalog backup and saved it to disk (shared NFS). Then, disconnected the old system from the network.

* Installed RHEL 5.7 in the new server. Used the same hostname and IP address.

* Installed NBU 7.1.0.4. Then, performed the catalog recovery. AFAIK, it went really smooth. In fact, the logfile (see excerpt below) and even in the activity monitor, it showed that the recovery was successful.

* The catalog size was almost, if not the same, as the CATDB backup/image files.

************************************************************************************************************************************

14:45:36 (2.001) Status of restore from copy 1 of image created Tue Apr  9 06:10:02 2013 = the requested operation was successfully completed

14:45:36 (2.xxx) INF - Status = the requested operation was successfully completed.

14:45:41 INF - Database restore started

Restore started Tue Apr  9 14:45:42 2013

14:45:42 (3.001) Restoring from copy 1 of image created Tue Apr  9 06:09:20 2013

14:45:43 (3.001) TAR STARTED 20821

14:45:44 (3.001) INF - Beginning restore from server mojito to client mojito.

14:45:46 (3.001) /opt/openv/db/staging/DARS_DATA.db

14:45:46 (3.001) /opt/openv/db/staging/DARS_INDEX.db

14:45:46 (3.001) /opt/openv/db/staging/DBM_DATA.db

14:45:46 (3.001) /opt/openv/db/staging/DBM_INDEX.db

14:45:47 (3.001) /opt/openv/db/staging/EMM_DATA.db

14:45:47 (3.001) /opt/openv/db/staging/EMM_INDEX.db

14:45:53 (3.001) /opt/openv/db/staging/NBAZDB.db

14:45:54 (3.001) /opt/openv/db/staging/NBAZDB.log.1

14:45:54 (3.001) /opt/openv/db/staging/NBDB.db

14:45:54 (3.001) /opt/openv/db/staging/NBDB.log.1

14:46:08 (3.001) /opt/openv/db/staging/databases.conf

14:46:08 (3.001) /opt/openv/db/staging/server.conf

14:46:08 (3.001) /opt/openv/db/staging/vxdbms.conf

14:46:08 (3.001) INF - TAR EXITING WITH STATUS = 0

14:46:08 (3.001) INF - TAR RESTORED 13 OF 13 FILES SUCCESSFULLY

14:46:08 (3.001) INF - TAR KEPT 0 EXISTING FILES

14:46:08 (3.001) INF - TAR PARTIALLY RESTORED 0 FILES

14:46:08 (3.001) Status of restore from copy 1 of image created Tue Apr  9 06:09:20 2013 = the requested operation was successfully completed

14:46:09 INF - Server status = 0

14:46:09 (3.xxx) INF - Status = the requested operation was successfully completed.

14:46:14 INF - Database recovery started

14:46:14 INF - Shutting down databases

14:46:14 INF - Moving database files from /usr/openv/db/staging

14:46:14 INF - Source /usr/openv/db/staging/EMM_DATA.db

14:46:14 INF - Destination /usr/openv/db/data/EMM_DATA.db

14:46:14 INF - Source /usr/openv/db/staging/DBM_DATA.db

14:46:14 INF - Destination /usr/openv/db/data/DBM_DATA.db

14:46:14 INF - Source /usr/openv/db/staging/DARS_DATA.db

14:46:14 INF - Destination /usr/openv/db/data/DARS_DATA.db

14:46:14 INF - Source /usr/openv/db/staging/NBDB.db

14:46:14 INF - Destination /usr/openv/db/data/NBDB.db

14:46:14 INF - Source /usr/openv/db/staging/NBAZDB.db

14:46:14 INF - Destination /usr/openv/db/data/NBAZDB.db

14:46:14 INF - Source /usr/openv/db/staging/EMM_INDEX.db

14:46:14 INF - Destination /usr/openv/db/data/EMM_INDEX.db

14:46:14 INF - Source /usr/openv/db/staging/DBM_INDEX.db

14:46:14 INF - Destination /usr/openv/db/data/DBM_INDEX.db

14:46:14 INF - Source /usr/openv/db/staging/DARS_INDEX.db

14:46:14 INF - Destination /usr/openv/db/data/DARS_INDEX.db

14:46:14 INF - Applying transaction log:  /usr/openv/db/staging/NBDB.log.1

14:46:15 INF - Applying transaction log:  /usr/openv/db/staging/NBDB.log

14:46:15 INF - Applying transaction log:  /usr/openv/db/data/NBDB.log

14:46:15 INF - Applying transaction log:  /usr/openv/db/staging/NBAZDB.log.1

14:46:16 INF - Applying transaction log:  /usr/openv/db/staging/NBAZDB.log

14:46:16 INF - Applying transaction log:  /usr/openv/db/data/NBAZDB.log

14:46:16 INF - Creating /usr/openv/db/data/vxdbms.conf

14:46:16 INF - Creating /usr/openv/var/global/server.conf

14:46:16 INF - Creating /usr/openv/var/global/databases.conf

14:46:16 INF - Starting up databases

14:46:19 INF - Database recovery successfully completed

14:46:19 INF - Recovery successfully completed

14:46:19 INF - Catalog recovery has completed.

14:46:19 WRN - NetBackup will not run scheduled backup jobs until NetBackup is restarted.

************************************************************************************************************************************

* We then proceeded to configure/change the new robotic path.

* All test backups as well as some "pre-selected" test restores went fine.

I thought everything went really smooth until the next morning when we receved a restore request from our user. Apparently, when we searched for the files (under backup, archive, and restore), it didn't list the files we were expecting. In fact, in some parts, it didn't show anything. But when we check/search the catalog, we see a bunch of images. In the end, we ended up rolling back to our old server, which is just, disconnecting the new system and reconnecting the old one to our network. True enough, that when we searched for the same files in the old server, it listed them out as expected.

Any inputs or ideas on this? We plan to perform another cutover next week. By the way, the catalog size we'll dealing with is about 180GB.

Cheers,

Dennis

[Marianne]

Installation on Solaris server was in /opt, right?

Was Linux installation done in /opt as well? 

Ensure installation is done in /opt, or else link /usr/openv to /opt/openv before you start catalog recovery.

[Denz1217]

Hey Marianne,

Thanks for the feedback. Yep, we've done that. We actually hit this issue during a mock upgrade using VMs. the default NBU installation path between Solaris and Linux are different but we made sure this part was covered. Any more suggestions?

Cheers,

Dennis

[Marianne]

Just a good look at the restore look to see what exactly was restored:

 /usr/openv/netbackup/logs/user_ops/root/logs/Recover#####

Also compare size of image folder after the restore.

The details in the opening post is only showing the restore of the Relational database. 
What were your exact catalog recovery command and/or options?

[Denz1217]

The log in the initial post is just a part of the entire log and its just the last segment. The actual log itself is about 44MB :) Mostly hundreds or thousands of line just showing each image file that was untarred or restored. Plus the summary of images xxxxxx of xxxxxx that was successfully restored.

Also, after the restoration was completed the final size of the image folder was about the same size of the backup image/files that was used for the recovery. The funny thing is, its about 8GB smaller compared to the images folder size in the original server. But then again, there was no errors during the FULL CATDB backup that was done prior to the restoration.

Also, the restoration was done thru the GUI. The steps were very similar to http://www.symantec.com/business/support/index?page=content&id=HOWTO33932. We selected  "Recover entire NetBackup catalog". Do note that we staged the CATDB backup in NFS (share)

Anyways, we're doing the test below and see if we can replicate the issue.

* Re-installed the OS on the new server. Used the same hostname but different IP. Not sure if having a different IP would matter though.

* Re-installed NBU 7.1.0.4.

* We're now restoring the same CATDB backup we used in our first cutover attempt. Hopefully, I'll get some results tomorrow.

For the meantime, appreciate if there's any more inputs about this matter.

[Marianne]

I have used the exact same method to migrate a customer from Solaris to Suse Linux. Also different IP address. I assume you have used the nbemmcmd command to update Master server OS in the EMM database? All we can do now is work with what you have after your next attempt and see how it goes...

[Rusty_Major]

I'm sure you already have looked at this, but if not, this technote I had bookmarked for when we plan to go down this road later this year. It looks like you mentioned doing everything its states, except for the last step about updating the emm entry for the master.

http://www.symantec.com/business/support/index?page=content&id=TECH77448

Also, I wonder if there was just a problem showing the restore files in the GUI. Would you still be able to test if bplist can list out the requested files for you? If it is a GUI issue, I wonder if the Windows GUI or a Java GUI would work, whichever one you are not using right now.

[Denz1217]

Hi Rusty and Marianne,

The recent test (using Diff IP but same hostname) seemed to work. There was some minor glitches on the java thingy but managed to resolve it. So, we are now running the old and new system in parallel. The 2nd attempt to restore the CATDB was again successful as far as the log is concern.

We tried the bplist and it didn't give a positive result. It only listed very very very minimal number of folders and files. These are the exact files we also see in the GUI. See below:

/usr/openv/netbackup/bin/bplist -C <Clientname> -t 19 -s 11/11/2011 11:11:11 -e 12/12/2013 11:11:11 -R  /las_vs0
/las_vs0/a12985/
/las_vs0/a12985/eft000

But running the bpimagelist against the same client shows a lot of stuff as seen below:

/usr/openv/netbackup/bin/admincmd/bpimagelist -client <Clientname> -d 11/11/2011 -e 12/12/2013 11:11:11 -U
Backed Up         Expires       Files       KB  C  Sched Type   Policy
----------------  ---------- -------- --------  -  ------------ ------------
03/20/2013 12:33  03/20/2014     3147 490977400  N  Full Backup  1time-ARC-ALD04
03/11/2013 17:00  03/11/2014    65253 509325914  N  Full Backup  1time-ARC-ALD04
02/27/2013 14:30  02/27/2014       18      453  N  Full Backup  1time-ARC-ALD04
02/27/2013 14:29  02/27/2014      398    66106  N  Full Backup  1time-ARC-ALD04
02/27/2013 14:24  02/27/2014      249  1093225  N  Full Backup  1time-ARC-ALD04
02/27/2013 14:24  02/27/2014     1239  3683678  N  Full Backup  1time-ARC-ALD04
02/27/2013 14:24  02/27/2014      683  9436814  N  Full Backup  1time-ARC-ALD04
02/27/2013 14:24  02/27/2014   242526 2211511764  N  Full Backup  1time-ARC-ALD04
02/27/2013 13:59  02/27/2014       18      453  N  Full Backup  1time-ARC-ALD04
02/08/2013 12:47  02/08/2014       73      589  N  Full Backup  1time-ARC-ALD04
02/08/2013 12:46  02/08/2014      439    63999  N  Full Backup  1time-ARC-ALD04
02/08/2013 12:46  02/08/2014      231   924647  N  Full Backup  1time-ARC-ALD04
02/08/2013 12:45  02/08/2014      541  5230952  N  Full Backup  1time-ARC-ALD04
02/08/2013 12:40  02/08/2014     2046 16704330  N  Full Backup  1time-ARC-ALD04
02/08/2013 12:35  02/08/2014     7145 25807690  N  Full Backup  1time-ARC-ALD04
02/08/2013 12:29  02/08/2014     5946 27245956  N  Full Backup  1time-ARC-ALD04
02/08/2013 12:13  02/08/2014     4789 31962304  N  Full Backup  1time-ARC-ALD04
02/08/2013 12:13  02/08/2014     8131 48344958  N  Full Backup  1time-ARC-ALD04
02/08/2013 12:13  02/08/2014     3029 91137622  N  Full Backup  1time-ARC-ALD04
02/08/2013 12:13  02/08/2014   256808 2300701330  N  Full Backup  1time-ARC-ALD04
12/21/2012 15:47  12/21/2013     4597 102731556  N  Full Backup  1time-ARC-ALD04
12/20/2012 10:46  12/20/2013   172946 251191320  N  Full Backup  1time-ARC-ALD04
12/20/2012 10:44  12/20/2013    17052 39708904  N  Full Backup  1time-ARC-ALD04
12/20/2012 10:44  12/20/2013    40670   963008  N  Full Backup  1time-ARC-ALD04
12/20/2012 10:44  12/20/2013   223401 232850434  N  Full Backup  1time-ARC-ALD04
12/20/2012 10:44  12/20/2013     7280  1135256  N  Full Backup  1time-ARC-ALD04
12/20/2012 09:32  12/20/2013     3884   147508  N  Full Backup  1time-ARC-ALD04
12/20/2012 09:31  12/20/2013     2364   207228  N  Full Backup  1time-ARC-ALD04
12/20/2012 09:31  12/20/2013    12224   833472  N  Full Backup  1time-ARC-ALD04
12/20/2012 09:30  12/20/2013    20806  1536630  N  Full Backup  1time-ARC-ALD04
12/20/2012 09:29  12/20/2013    24514  1823734  N  Full Backup  1time-ARC-ALD04
12/20/2012 09:29  12/20/2013    43769  1947780  N  Full Backup  1time-ARC-ALD04
12/20/2012 09:28  12/20/2013     4919  3228018  N  Full Backup  1time-ARC-ALD04
12/20/2012 09:27  12/20/2013    19584  3501972  N  Full Backup  1time-ARC-ALD04
12/20/2012 09:26  12/20/2013    13952  7520640  N  Full Backup  1time-ARC-ALD04
12/20/2012 09:22  12/20/2013     3031  9659364  N  Full Backup  1time-ARC-ALD04
12/20/2012 09:18  12/20/2013    25694 14759554  N  Full Backup  1time-ARC-ALD04
12/20/2012 09:17  12/20/2013    14685 24567622  N  Full Backup  1time-ARC-ALD04
12/20/2012 09:05  12/20/2013    25007 29089440  N  Full Backup  1time-ARC-ALD04
12/20/2012 09:00  12/20/2013    16480 40975006  N  Full Backup  1time-ARC-ALD04
12/20/2012 08:39  12/20/2013    52831 41103946  N  Full Backup  1time-ARC-ALD04
12/20/2012 08:39  12/20/2013    24954 52626248  N  Full Backup  1time-ARC-ALD04
12/20/2012 08:39  12/20/2013    64325 117158256  N  Full Backup  1time-ARC-ALD04
12/20/2012 08:39  12/20/2013    60759 294982922  N  Full Backup  1time-ARC-ALD04
12/19/2012 19:21  12/19/2013    31126 12728596  N  Full Backup  1time-ARC-ALD04
12/19/2012 19:10  12/19/2013     8866 23573170  N  Full Backup  1time-ARC-ALD04
12/19/2012 18:59  12/19/2013    32309 23611516  N  Full Backup  1time-ARC-ALD04
12/19/2012 18:21  12/19/2013   167499 69787654  N  Full Backup  1time-ARC-ALD04
12/19/2012 18:21  12/19/2013    80829 212626168  N  Full Backup  1time-ARC-ALD04
12/19/2012 18:21  12/19/2013   211129 322978872  N  Full Backup  1time-ARC-ALD04
12/19/2012 18:21  12/19/2013    31038 429556356  N  Full Backup  1time-ARC-ALD04
12/19/2012 13:12  12/19/2013      156     1304  N  Full Backup  1time-ARC-ALD04
12/19/2012 13:12  12/19/2013      241    30311  N  Full Backup  1time-ARC-ALD04
12/19/2012 13:11  12/19/2013      216    20194  N  Full Backup  1time-ARC-ALD04
12/19/2012 13:10  12/19/2013      215    29282  N  Full Backup  1time-ARC-ALD04
12/19/2012 13:10  12/19/2013      226    86564  N  Full Backup  1time-ARC-ALD04
12/19/2012 13:10  12/19/2013    36027 93805624  N  Full Backup  1time-ARC-ALD04
12/19/2012 13:10  12/19/2013      245   390501  N  Full Backup  1time-ARC-ALD04
12/19/2012 13:09  12/19/2013      211   654240  N  Full Backup  1time-ARC-ALD04
12/19/2012 13:09  12/19/2013      239   335332  N  Full Backup  1time-ARC-ALD04
12/19/2012 13:08  12/19/2013      317   298863  N  Full Backup  1time-ARC-ALD04
12/19/2012 13:08  12/19/2013      205   861408  N  Full Backup  1time-ARC-ALD04
12/19/2012 13:05  12/19/2013        2      451  N  Full Backup  1time-ARC-ALD04
12/19/2012 13:05  12/19/2013   171014 474871948  N  Full Backup  1time-ARC-ALD04
12/19/2012 13:05  12/19/2013     7732  2460436  N  Full Backup  1time-ARC-ALD04
12/19/2012 13:05  12/19/2013      129     1045  N  Full Backup  1time-ARC-ALD04
11/27/2012 11:56  06/01/2013    80374 498123752  N  Full Backup  1time-ARC-ALD04
 

Also, I found this https://www-secure.symantec.com/connect/forums/cant-restore-bplist-status-227-images-exist-bpimageli... and was hoping it'll help since we have a "similar" situation. The <Clientname> we had actually started with an Uppercase. So, i tried the suggested solution but the issue persists.

Hi Marianne,

Can you please elaborate on the nbemmcmd? I don't remember we ran this command. Thanks.

Cheers.

[Marianne]

It is point 9 in the TN quoted by Rusty: http://www.symantec.com/docs/TECH77448 
(I have simply assumed that you were using the same TN.)

If the operating system has changed (for example from Solaris to AIX) run the command "nbemmcmd -updatehost -machinename <master server name> -machinetype master -operatingsystem <new O/S>".  Running this command updates the operating system field in the EMM master server record to reflect the new operating system.  A list of valid operating systems can be found in the on-line help for the nbemmcmd -updatehost command. 

Have you tried bplist with shorter periods? Maybe not more than a month at a time?
it is possible that "Browse timeout" is experienced due to length of period.
bpimagelist simply reads header info, bplist needs to drill down into '.f' files.
If you have database compression enabled, this prosess will even take longer.

Maybe double-check if image compression is enabled on source server?
If so, it may be a good idea to uncompress before doing the migration.

[Denz1217]

I believe we may have missed that point #9. Anyways, i ran it accordingly but the issue persists :(

With regards to the bplist command, the odd thing is, if I ran the exact command in the old server, it'll display more data. Anyways, I still tried your suggestion of running a shorter date range. See below:

/usr/openv/netbackup/bin/bplist -C <Client-2> -t 19 -s 02/25/2013 11:11:11 -e 03/01/2013 11:11:11 -R  /las_vs0
/las_vs0/a14204/
/las_vs0/a14204/restore/
/las_vs0/a14204/restore_symboltable

/usr/openv/netbackup/bin/admincmd/bpimagelist -client <Client-2> -d 02/25/2013 -e 03/01/2013 11:11:11 -U
Backed Up         Expires       Files       KB  C  Sched Type   Policy
----------------  ---------- -------- --------  -  ------------ ------------
02/27/2013 14:30  02/27/2014       20     4678  N  Full Backup  1time-ARC-ALD03
02/27/2013 14:30  02/27/2014        3     4547  N  Full Backup  1time-ARC-ALD03
02/27/2013 14:30  02/27/2014       80    31054  N  Full Backup  1time-ARC-ALD03
02/27/2013 14:29  02/27/2014      415    39996  N  Full Backup  1time-ARC-ALD03
02/27/2013 14:24  02/27/2014     6230  3846676  N  Full Backup  1time-ARC-ALD03
02/27/2013 14:24  02/27/2014      479 10589662  N  Full Backup  1time-ARC-ALD03
02/27/2013 14:24  02/27/2014     2060 23637884  N  Full Backup  1time-ARC-ALD03
02/27/2013 14:24  02/27/2014    11303 180909900  N  Full Backup  1time-ARC-ALD03
02/27/2013 13:59  02/27/2014       20     4678  N  Full Backup  1time-ARC-ALD03

Again, the bplist output was very very minimal compared to the result of the running the same command on the original server.

As for DB compression, it's indeed enabled (compress after 14 weeks). It would be nice if we can try doing a backup/recovery of the uncompress state but I'm afraid the current LUN holding the DB in the original server might not have enough space to accommodate the uncompressed version.

We now have opened a ticket with Symantec. I'm hoping they can help on this matter :)

But please don't stop providing your valuable insights and same goes to the other readers of this topic :)

[Denz1217]

While waiting for Symantec to call us. We'll likely perform a new CATDB backup and try to do the restoration again. Maybe we got super unlikely with the CATDB backup we used during the migration :)

[Marianne · Accepted solution]

So, it seems we managed to pinpoint the problem: catalog compression on source server. I would try the following: Use nbu commands to uncompress one client. Delete contents of client folder on new server, then use scp to copy uncompressed images from source server. Try bplist again.

[Denz1217]

Hi Marianne,

I personally didn't get a chance to do your suggestion but we did show this thread to the tech support. But your conclusion is very much true. The tech guy tried to run the bpimage -uncompress on the new server. It didn't error out but it also didn't do anything :) In the end, we needed to install a linux package to make this work. Odd thing is, we have a different NBU environment running on rhel 5.5 and the same nbu version but the image uncompression is working even without having the package in question. So maybe this is something to do with the difference between rhel 5.7 and 5.5. Not really sure. But the good news is, we're now able to browse the files for restoration properly. As usual, thanks Marianne for the prompt and accurate responses.

Cheers,

Dennis

[Marianne]

Thanks for the feedback!

Please tell us the name of the package that was installed on the new master?
Someone else with a similar problem may stumble upon this post....

[rickylimsc]

Hi Marianne,

The package comes with the RHEL5.7 ISO but is not pre-installed with the OS installation. Our linux admin installed this package manually in the new server when the tech support requested it.

Just had a call with the tech support that they have found a tech note relevant to this case and it indicates that this uncompress needs to be installed to work.

I'll post the technote here when it's available to us.

[rickylimsc]

This is the TechNote.

http://www.symantec.com/business/support/index?page=content&id=TECH153677&profileURL=https%3A%2F%2Fs...

[rickylimsc]

This is the technote given Netbackup Compression and decompression