04-10-2013 01:07 AM
Hi All,
We've attempted to move our NBU 7.1.0.4 from a T2000 Sparc to a HP (intel based) RHEL 5.7. Below is the procedure we took.
* From the old solaris, we performed DB catalog backup and saved it to disk (shared NFS). Then, disconnected the old system from the network.
* Installed RHEL 5.7 in the new server. Used the same hostname and IP address.
* Installed NBU 7.1.0.4. Then, performed the catalog recovery. AFAIK, it went really smooth. In fact, the logfile (see excerpt below) and even in the activity monitor, it showed that the recovery was successful.
* The catalog size was almost, if not the same, as the CATDB backup/image files.
************************************************************************************************************************************
14:45:36 (2.001) Status of restore from copy 1 of image created Tue Apr 9 06:10:02 2013 = the requested operation was successfully completed
14:45:36 (2.xxx) INF - Status = the requested operation was successfully completed.
14:45:41 INF - Database restore started
Restore started Tue Apr 9 14:45:42 2013
14:45:42 (3.001) Restoring from copy 1 of image created Tue Apr 9 06:09:20 2013
14:45:43 (3.001) TAR STARTED 20821
14:45:44 (3.001) INF - Beginning restore from server mojito to client mojito.
14:45:46 (3.001) /opt/openv/db/staging/DARS_DATA.db
14:45:46 (3.001) /opt/openv/db/staging/DARS_INDEX.db
14:45:46 (3.001) /opt/openv/db/staging/DBM_DATA.db
14:45:46 (3.001) /opt/openv/db/staging/DBM_INDEX.db
14:45:47 (3.001) /opt/openv/db/staging/EMM_DATA.db
14:45:47 (3.001) /opt/openv/db/staging/EMM_INDEX.db
14:45:53 (3.001) /opt/openv/db/staging/NBAZDB.db
14:45:54 (3.001) /opt/openv/db/staging/NBAZDB.log.1
14:45:54 (3.001) /opt/openv/db/staging/NBDB.db
14:45:54 (3.001) /opt/openv/db/staging/NBDB.log.1
14:46:08 (3.001) /opt/openv/db/staging/databases.conf
14:46:08 (3.001) /opt/openv/db/staging/server.conf
14:46:08 (3.001) /opt/openv/db/staging/vxdbms.conf
14:46:08 (3.001) INF - TAR EXITING WITH STATUS = 0
14:46:08 (3.001) INF - TAR RESTORED 13 OF 13 FILES SUCCESSFULLY
14:46:08 (3.001) INF - TAR KEPT 0 EXISTING FILES
14:46:08 (3.001) INF - TAR PARTIALLY RESTORED 0 FILES
14:46:08 (3.001) Status of restore from copy 1 of image created Tue Apr 9 06:09:20 2013 = the requested operation was successfully completed
14:46:09 INF - Server status = 0
14:46:09 (3.xxx) INF - Status = the requested operation was successfully completed.
14:46:14 INF - Database recovery started
14:46:14 INF - Shutting down databases
14:46:14 INF - Moving database files from /usr/openv/db/staging
14:46:14 INF - Source /usr/openv/db/staging/EMM_DATA.db
14:46:14 INF - Destination /usr/openv/db/data/EMM_DATA.db
14:46:14 INF - Source /usr/openv/db/staging/DBM_DATA.db
14:46:14 INF - Destination /usr/openv/db/data/DBM_DATA.db
14:46:14 INF - Source /usr/openv/db/staging/DARS_DATA.db
14:46:14 INF - Destination /usr/openv/db/data/DARS_DATA.db
14:46:14 INF - Source /usr/openv/db/staging/NBDB.db
14:46:14 INF - Destination /usr/openv/db/data/NBDB.db
14:46:14 INF - Source /usr/openv/db/staging/NBAZDB.db
14:46:14 INF - Destination /usr/openv/db/data/NBAZDB.db
14:46:14 INF - Source /usr/openv/db/staging/EMM_INDEX.db
14:46:14 INF - Destination /usr/openv/db/data/EMM_INDEX.db
14:46:14 INF - Source /usr/openv/db/staging/DBM_INDEX.db
14:46:14 INF - Destination /usr/openv/db/data/DBM_INDEX.db
14:46:14 INF - Source /usr/openv/db/staging/DARS_INDEX.db
14:46:14 INF - Destination /usr/openv/db/data/DARS_INDEX.db
14:46:14 INF - Applying transaction log: /usr/openv/db/staging/NBDB.log.1
14:46:15 INF - Applying transaction log: /usr/openv/db/staging/NBDB.log
14:46:15 INF - Applying transaction log: /usr/openv/db/data/NBDB.log
14:46:15 INF - Applying transaction log: /usr/openv/db/staging/NBAZDB.log.1
14:46:16 INF - Applying transaction log: /usr/openv/db/staging/NBAZDB.log
14:46:16 INF - Applying transaction log: /usr/openv/db/data/NBAZDB.log
14:46:16 INF - Creating /usr/openv/db/data/vxdbms.conf
14:46:16 INF - Creating /usr/openv/var/global/server.conf
14:46:16 INF - Creating /usr/openv/var/global/databases.conf
14:46:16 INF - Starting up databases
14:46:19 INF - Database recovery successfully completed
14:46:19 INF - Recovery successfully completed
14:46:19 INF - Catalog recovery has completed.
14:46:19 WRN - NetBackup will not run scheduled backup jobs until NetBackup is restarted.
************************************************************************************************************************************
* We then proceeded to configure/change the new robotic path.
* All test backups as well as some "pre-selected" test restores went fine.
I thought everything went really smooth until the next morning when we receved a restore request from our user. Apparently, when we searched for the files (under backup, archive, and restore), it didn't list the files we were expecting. In fact, in some parts, it didn't show anything. But when we check/search the catalog, we see a bunch of images. In the end, we ended up rolling back to our old server, which is just, disconnecting the new system and reconnecting the old one to our network. True enough, that when we searched for the same files in the old server, it listed them out as expected.
Any inputs or ideas on this? We plan to perform another cutover next week. By the way, the catalog size we'll dealing with is about 180GB.
Cheers,
Dennis
Solved! Go to Solution.
04-10-2013 11:58 PM
04-10-2013 01:26 AM
Installation on Solaris server was in /opt, right?
Was Linux installation done in /opt as well?
Ensure installation is done in /opt, or else link /usr/openv to /opt/openv before you start catalog recovery.
04-10-2013 02:07 AM
Hey Marianne,
Thanks for the feedback. Yep, we've done that. We actually hit this issue during a mock upgrade using VMs. the default NBU installation path between Solaris and Linux are different but we made sure this part was covered. Any more suggestions?
Cheers,
Dennis
04-10-2013 03:09 AM
Just a good look at the restore look to see what exactly was restored:
/usr/openv/netbackup/logs/user_ops/root/logs/Recover#####
Also compare size of image folder after the restore.
The details in the opening post is only showing the restore of the Relational database.
What were your exact catalog recovery command and/or options?
04-10-2013 05:35 AM
The log in the initial post is just a part of the entire log and its just the last segment. The actual log itself is about 44MB :) Mostly hundreds or thousands of line just showing each image file that was untarred or restored. Plus the summary of images xxxxxx of xxxxxx that was successfully restored.
Also, after the restoration was completed the final size of the image folder was about the same size of the backup image/files that was used for the recovery. The funny thing is, its about 8GB smaller compared to the images folder size in the original server. But then again, there was no errors during the FULL CATDB backup that was done prior to the restoration.
Also, the restoration was done thru the GUI. The steps were very similar to http://www.symantec.com/business/support/index?page=content&id=HOWTO33932. We selected "Recover entire NetBackup catalog". Do note that we staged the CATDB backup in NFS (share)
Anyways, we're doing the test below and see if we can replicate the issue.
* Re-installed the OS on the new server. Used the same hostname but different IP. Not sure if having a different IP would matter though.
* Re-installed NBU 7.1.0.4.
* We're now restoring the same CATDB backup we used in our first cutover attempt. Hopefully, I'll get some results tomorrow.
For the meantime, appreciate if there's any more inputs about this matter.
04-10-2013 06:30 AM
04-10-2013 02:09 PM
I'm sure you already have looked at this, but if not, this technote I had bookmarked for when we plan to go down this road later this year. It looks like you mentioned doing everything its states, except for the last step about updating the emm entry for the master.
http://www.symantec.com/business/support/index?page=content&id=TECH77448
Also, I wonder if there was just a problem showing the restore files in the GUI. Would you still be able to test if bplist can list out the requested files for you? If it is a GUI issue, I wonder if the Windows GUI or a Java GUI would work, whichever one you are not using right now.
04-10-2013 08:27 PM
Hi Rusty and Marianne,
The recent test (using Diff IP but same hostname) seemed to work. There was some minor glitches on the java thingy but managed to resolve it. So, we are now running the old and new system in parallel. The 2nd attempt to restore the CATDB was again successful as far as the log is concern.
We tried the bplist and it didn't give a positive result. It only listed very very very minimal number of folders and files. These are the exact files we also see in the GUI. See below:
/usr/openv/netbackup/bin/bplist -C <Clientname> -t 19 -s 11/11/2011 11:11:11 -e 12/12/2013 11:11:11 -R /las_vs0
/las_vs0/a12985/
/las_vs0/a12985/eft000
But running the bpimagelist against the same client shows a lot of stuff as seen below:
/usr/openv/netbackup/bin/admincmd/bpimagelist -client <Clientname> -d 11/11/2011 -e 12/12/2013 11:11:11 -U
Backed Up Expires Files KB C Sched Type Policy
---------------- ---------- -------- -------- - ------------ ------------
03/20/2013 12:33 03/20/2014 3147 490977400 N Full Backup 1time-ARC-ALD04
03/11/2013 17:00 03/11/2014 65253 509325914 N Full Backup 1time-ARC-ALD04
02/27/2013 14:30 02/27/2014 18 453 N Full Backup 1time-ARC-ALD04
02/27/2013 14:29 02/27/2014 398 66106 N Full Backup 1time-ARC-ALD04
02/27/2013 14:24 02/27/2014 249 1093225 N Full Backup 1time-ARC-ALD04
02/27/2013 14:24 02/27/2014 1239 3683678 N Full Backup 1time-ARC-ALD04
02/27/2013 14:24 02/27/2014 683 9436814 N Full Backup 1time-ARC-ALD04
02/27/2013 14:24 02/27/2014 242526 2211511764 N Full Backup 1time-ARC-ALD04
02/27/2013 13:59 02/27/2014 18 453 N Full Backup 1time-ARC-ALD04
02/08/2013 12:47 02/08/2014 73 589 N Full Backup 1time-ARC-ALD04
02/08/2013 12:46 02/08/2014 439 63999 N Full Backup 1time-ARC-ALD04
02/08/2013 12:46 02/08/2014 231 924647 N Full Backup 1time-ARC-ALD04
02/08/2013 12:45 02/08/2014 541 5230952 N Full Backup 1time-ARC-ALD04
02/08/2013 12:40 02/08/2014 2046 16704330 N Full Backup 1time-ARC-ALD04
02/08/2013 12:35 02/08/2014 7145 25807690 N Full Backup 1time-ARC-ALD04
02/08/2013 12:29 02/08/2014 5946 27245956 N Full Backup 1time-ARC-ALD04
02/08/2013 12:13 02/08/2014 4789 31962304 N Full Backup 1time-ARC-ALD04
02/08/2013 12:13 02/08/2014 8131 48344958 N Full Backup 1time-ARC-ALD04
02/08/2013 12:13 02/08/2014 3029 91137622 N Full Backup 1time-ARC-ALD04
02/08/2013 12:13 02/08/2014 256808 2300701330 N Full Backup 1time-ARC-ALD04
12/21/2012 15:47 12/21/2013 4597 102731556 N Full Backup 1time-ARC-ALD04
12/20/2012 10:46 12/20/2013 172946 251191320 N Full Backup 1time-ARC-ALD04
12/20/2012 10:44 12/20/2013 17052 39708904 N Full Backup 1time-ARC-ALD04
12/20/2012 10:44 12/20/2013 40670 963008 N Full Backup 1time-ARC-ALD04
12/20/2012 10:44 12/20/2013 223401 232850434 N Full Backup 1time-ARC-ALD04
12/20/2012 10:44 12/20/2013 7280 1135256 N Full Backup 1time-ARC-ALD04
12/20/2012 09:32 12/20/2013 3884 147508 N Full Backup 1time-ARC-ALD04
12/20/2012 09:31 12/20/2013 2364 207228 N Full Backup 1time-ARC-ALD04
12/20/2012 09:31 12/20/2013 12224 833472 N Full Backup 1time-ARC-ALD04
12/20/2012 09:30 12/20/2013 20806 1536630 N Full Backup 1time-ARC-ALD04
12/20/2012 09:29 12/20/2013 24514 1823734 N Full Backup 1time-ARC-ALD04
12/20/2012 09:29 12/20/2013 43769 1947780 N Full Backup 1time-ARC-ALD04
12/20/2012 09:28 12/20/2013 4919 3228018 N Full Backup 1time-ARC-ALD04
12/20/2012 09:27 12/20/2013 19584 3501972 N Full Backup 1time-ARC-ALD04
12/20/2012 09:26 12/20/2013 13952 7520640 N Full Backup 1time-ARC-ALD04
12/20/2012 09:22 12/20/2013 3031 9659364 N Full Backup 1time-ARC-ALD04
12/20/2012 09:18 12/20/2013 25694 14759554 N Full Backup 1time-ARC-ALD04
12/20/2012 09:17 12/20/2013 14685 24567622 N Full Backup 1time-ARC-ALD04
12/20/2012 09:05 12/20/2013 25007 29089440 N Full Backup 1time-ARC-ALD04
12/20/2012 09:00 12/20/2013 16480 40975006 N Full Backup 1time-ARC-ALD04
12/20/2012 08:39 12/20/2013 52831 41103946 N Full Backup 1time-ARC-ALD04
12/20/2012 08:39 12/20/2013 24954 52626248 N Full Backup 1time-ARC-ALD04
12/20/2012 08:39 12/20/2013 64325 117158256 N Full Backup 1time-ARC-ALD04
12/20/2012 08:39 12/20/2013 60759 294982922 N Full Backup 1time-ARC-ALD04
12/19/2012 19:21 12/19/2013 31126 12728596 N Full Backup 1time-ARC-ALD04
12/19/2012 19:10 12/19/2013 8866 23573170 N Full Backup 1time-ARC-ALD04
12/19/2012 18:59 12/19/2013 32309 23611516 N Full Backup 1time-ARC-ALD04
12/19/2012 18:21 12/19/2013 167499 69787654 N Full Backup 1time-ARC-ALD04
12/19/2012 18:21 12/19/2013 80829 212626168 N Full Backup 1time-ARC-ALD04
12/19/2012 18:21 12/19/2013 211129 322978872 N Full Backup 1time-ARC-ALD04
12/19/2012 18:21 12/19/2013 31038 429556356 N Full Backup 1time-ARC-ALD04
12/19/2012 13:12 12/19/2013 156 1304 N Full Backup 1time-ARC-ALD04
12/19/2012 13:12 12/19/2013 241 30311 N Full Backup 1time-ARC-ALD04
12/19/2012 13:11 12/19/2013 216 20194 N Full Backup 1time-ARC-ALD04
12/19/2012 13:10 12/19/2013 215 29282 N Full Backup 1time-ARC-ALD04
12/19/2012 13:10 12/19/2013 226 86564 N Full Backup 1time-ARC-ALD04
12/19/2012 13:10 12/19/2013 36027 93805624 N Full Backup 1time-ARC-ALD04
12/19/2012 13:10 12/19/2013 245 390501 N Full Backup 1time-ARC-ALD04
12/19/2012 13:09 12/19/2013 211 654240 N Full Backup 1time-ARC-ALD04
12/19/2012 13:09 12/19/2013 239 335332 N Full Backup 1time-ARC-ALD04
12/19/2012 13:08 12/19/2013 317 298863 N Full Backup 1time-ARC-ALD04
12/19/2012 13:08 12/19/2013 205 861408 N Full Backup 1time-ARC-ALD04
12/19/2012 13:05 12/19/2013 2 451 N Full Backup 1time-ARC-ALD04
12/19/2012 13:05 12/19/2013 171014 474871948 N Full Backup 1time-ARC-ALD04
12/19/2012 13:05 12/19/2013 7732 2460436 N Full Backup 1time-ARC-ALD04
12/19/2012 13:05 12/19/2013 129 1045 N Full Backup 1time-ARC-ALD04
11/27/2012 11:56 06/01/2013 80374 498123752 N Full Backup 1time-ARC-ALD04
Also, I found this https://www-secure.symantec.com/connect/forums/cant-restore-bplist-status-227-images-exist-bpimageli... and was hoping it'll help since we have a "similar" situation. The <Clientname> we had actually started with an Uppercase. So, i tried the suggested solution but the issue persists.
Hi Marianne,
Can you please elaborate on the nbemmcmd? I don't remember we ran this command. Thanks.
Cheers.
04-10-2013 10:06 PM
It is point 9 in the TN quoted by Rusty: http://www.symantec.com/docs/TECH77448
(I have simply assumed that you were using the same TN.)
If the operating system has changed (for example from Solaris to AIX) run the command "nbemmcmd -updatehost -machinename <master server name> -machinetype master -operatingsystem <new O/S>". Running this command updates the operating system field in the EMM master server record to reflect the new operating system. A list of valid operating systems can be found in the on-line help for the nbemmcmd -updatehost command.
Have you tried bplist with shorter periods? Maybe not more than a month at a time?
it is possible that "Browse timeout" is experienced due to length of period.
bpimagelist simply reads header info, bplist needs to drill down into '.f' files.
If you have database compression enabled, this prosess will even take longer.
Maybe double-check if image compression is enabled on source server?
If so, it may be a good idea to uncompress before doing the migration.
04-10-2013 11:32 PM
I believe we may have missed that point #9. Anyways, i ran it accordingly but the issue persists :(
With regards to the bplist command, the odd thing is, if I ran the exact command in the old server, it'll display more data. Anyways, I still tried your suggestion of running a shorter date range. See below:
/usr/openv/netbackup/bin/bplist -C <Client-2> -t 19 -s 02/25/2013 11:11:11 -e 03/01/2013 11:11:11 -R /las_vs0
/las_vs0/a14204/
/las_vs0/a14204/restore/
/las_vs0/a14204/restore_symboltable
/usr/openv/netbackup/bin/admincmd/bpimagelist -client <Client-2> -d 02/25/2013 -e 03/01/2013 11:11:11 -U
Backed Up Expires Files KB C Sched Type Policy
---------------- ---------- -------- -------- - ------------ ------------
02/27/2013 14:30 02/27/2014 20 4678 N Full Backup 1time-ARC-ALD03
02/27/2013 14:30 02/27/2014 3 4547 N Full Backup 1time-ARC-ALD03
02/27/2013 14:30 02/27/2014 80 31054 N Full Backup 1time-ARC-ALD03
02/27/2013 14:29 02/27/2014 415 39996 N Full Backup 1time-ARC-ALD03
02/27/2013 14:24 02/27/2014 6230 3846676 N Full Backup 1time-ARC-ALD03
02/27/2013 14:24 02/27/2014 479 10589662 N Full Backup 1time-ARC-ALD03
02/27/2013 14:24 02/27/2014 2060 23637884 N Full Backup 1time-ARC-ALD03
02/27/2013 14:24 02/27/2014 11303 180909900 N Full Backup 1time-ARC-ALD03
02/27/2013 13:59 02/27/2014 20 4678 N Full Backup 1time-ARC-ALD03
Again, the bplist output was very very minimal compared to the result of the running the same command on the original server.
As for DB compression, it's indeed enabled (compress after 14 weeks). It would be nice if we can try doing a backup/recovery of the uncompress state but I'm afraid the current LUN holding the DB in the original server might not have enough space to accommodate the uncompressed version.
We now have opened a ticket with Symantec. I'm hoping they can help on this matter :)
But please don't stop providing your valuable insights and same goes to the other readers of this topic :)
04-10-2013 11:34 PM
While waiting for Symantec to call us. We'll likely perform a new CATDB backup and try to do the restoration again. Maybe we got super unlikely with the CATDB backup we used during the migration :)
04-10-2013 11:58 PM
04-12-2013 07:41 PM
Hi Marianne,
I personally didn't get a chance to do your suggestion but we did show this thread to the tech support. But your conclusion is very much true. The tech guy tried to run the bpimage -uncompress on the new server. It didn't error out but it also didn't do anything :) In the end, we needed to install a linux package to make this work. Odd thing is, we have a different NBU environment running on rhel 5.5 and the same nbu version but the image uncompression is working even without having the package in question. So maybe this is something to do with the difference between rhel 5.7 and 5.5. Not really sure. But the good news is, we're now able to browse the files for restoration properly. As usual, thanks Marianne for the prompt and accurate responses.
Cheers,
Dennis
04-12-2013 08:03 PM
Thanks for the feedback!
Please tell us the name of the package that was installed on the new master?
Someone else with a similar problem may stumble upon this post....
04-14-2013 07:05 PM
Hi Marianne,
The package comes with the RHEL5.7 ISO but is not pre-installed with the OS installation. Our linux admin installed this package manually in the new server when the tech support requested it.
Just had a call with the tech support that they have found a tech note relevant to this case and it indicates that this uncompress needs to be installed to work.
I'll post the technote here when it's available to us.
04-14-2013 07:14 PM
04-14-2013 07:16 PM
This is the technote given Netbackup Compression and decompression