07-09-2013 05:15 AM
Hi,
With the help of symantec technical support, we upgraded our microsoft clustered master server from 7.1.0.1 to 7.5.0.6.
The upgrade process was far from being a breeze even wtih technical support webex set up.
Just a summary of the issue while upgrading.
Now, my main issue here. I was browsing my backup images and realized that i can only see backups done last night. All my previous backups could not be found.
I have the catalog backup of before the upgrade but would like to know if there is anything else we can do without setting up a 7.1 from scratch and recovering the catalog.
I dont suppose there's an option of restore just the image db from my 7.1 catalog backup to the current 7.5.0.6 setup?
Overall, the upgrade has been a nightmare for me. Was on the line with tech support for almost 6 hours straight..
Solved! Go to Solution.
07-11-2013 02:44 AM
SLP_DSSU_MIGRATION_STATE="1"
LIST_FS_IMAGE_HEADERS="0"
Anyway, i have since uninstalled, reinstalled 7.1, restored catalog, and performed the upgrade.
Took a full 8 hrs, luckily this time it managed to import the old backup images correctly.
I think there's a technote with regards to the setup.exe crashing in 2008 R2 and the resolution was to reboot the server, then perform the install again.
So lessons to be learnt.
Thanks for the help guys.
07-09-2013 06:02 AM
Before panic - The GUI is exposed to the names you are using for browising backup. srv1 - srv1.acme.com or SRV1 are not the same servers. Maybe the upgrade changed default name.
Check prvious backup with this command utility:
bpimagelist -d 06/01/2013 -U (or -L or -l)
You might find this T/N usefull :
http://www.symantec.com/docs/TECH91133
http://www.symantec.com/docs/HOWTO43661
Update:
Try this from the master server
bpclimagelist -client {CLINET_NAME_FROM_POLICY} -d 01/01/2013
should output somthing like:
07-09-2013 06:19 AM
I did a bpimagelist and only images from yesterday are listed..
Same with bpclimagelist
07-09-2013 07:14 AM
Are you sure that your backups metadata have been migrated into relational database (NBEMM) ?
Have you an "image cleaning" job after 1st NetBackup 7.5 start ? What is his status ? Logs say that images are migrated into NBEMM ?
You can also look in the <NBU Install dir>\db\images if you have only .lck and .f files..
07-09-2013 07:34 AM
07-09-2013 07:42 AM
Idiot question (but...) : Are you sure that your NetBackup Master service (or IP) is on the node that have the storage owning database files ?
07-09-2013 07:47 AM
The OpsCenter upgrade doesnt surprise me - i have seen the database upgrade part run for several hours, especially if you use analytics and have a lot of data in OpsCenter - they should have left that to run!
Do all of your cluster resources appear in Cluster manager?
The upgrade tends to delete and recreate them all - just wondering if during the hang something was missed - like a drive that held some of your catalogs - and on that thought .... a cluster would use a shared drive for the catalogs - say Y drive ... if the installation hung then maybe it is writing the catalogs to C:\Program files\veritas now and you just have a configuration / registry entry that is incorrect
Take a look to see if last nights backups are in the wrong place - that may be easily fixable!
07-09-2013 07:54 AM
I agree with Mark. That's why I spoke of storage on wrong node... It sounds like a loss of the storage that contains database.
07-09-2013 08:00 AM
07-09-2013 08:06 AM
I dont use analytics.
Yes, my cluster resources appear in the cluster manager. This was also one of the initial issues and the tech support had to run bpclusterutil -c to recreate them.
In my case, the cluster disk is H: and last nights backups do seem to be on the H: drive.
07-09-2013 08:09 AM
Here's a pic of my current cluster manager.
07-09-2013 08:43 AM
Can i take it all the other backups are also on the H drive?
If so that I guess it is the .f files that have not been migrated in correctly - if that is the case then you are going to need further support with this one and may well end up rolling back and doing a catalog restore.
Can I just check a couple of things with you ...
1. What does the output of nbstlutil -active -backupid NetBackup_0000000001 give?
2. Have you failed the cluster over and re-ran the bpimage -cleanup -allclients again so that it has run on both nodes?
3. What does bpgetconfig LIST_FS_IMAGE_HEADERS show for each node (possibly easier to just check the registry as one node will be passive)
07-09-2013 08:52 AM
07-09-2013 09:21 AM
I think that there shouldn't be a - in front of active - sorry
2 /3 OK - see how it goes tomorrow - should do the metadata migration again and both should then be NO after that
I do think you may still be in trouble though!!
07-09-2013 09:31 AM
07-09-2013 10:59 AM
Jon.K.
What is the status for your:
nbemmcmd -listsettings -brief -machinename <master server clustered name> ; look for the status of the:
SLP_DSSU_MIGRATION_STATE and LIST_FS_IMAGE_HEADERS.
has to be similar like this:
SLP_DSSU_MIGRATION_STATE="1"
LIST_FS_IMAGE_HEADERS="0"
If that is different, then your import was failed and need to rerun again.
Check if you can find a file like this: /usr/openv/netbackup/db.corrupt <--- for corrupt images.
Regarding your question, to recover in a stand alone. It could be possible but your master need to have the same name and configuration, to restore from catalog. Similar like a DR exercise.
Hope this help to you
VP
07-11-2013 02:06 AM
I dont think you could do the restore to a standalone - it really should be clustered to do it
I know we are suggesting a lot on here but you really should be on the case of support - just tell them you want the case "escalted" and insist on some urgent action.
If does sound like you may have to uninstall / re-install / recover the catalog (meaning the last 2 or 3 days of backups have been lost (though you could run media report and keep the tapes used over the last few days safe and them import them once you are all done)
I have the feeling that the NBDB is not right - during an upgrade it gets dumped out, a new one created and then pulled back in - my feeling is that when it hung it didn't pull the old data in and when it ran it next migration step it didn't actually have the full details of all of the old backup images.
Take a look under netbackup\temp\install\nbdb_date\
There should be the nbdb log file relating to when the upgrade too place.
The last line should say
create_nbdb: Exiting with rc = 0
but you do a search for any <16> or <32> lines to see if it went wrong somewhere.
Also make sure your netbackupdb\conf\ databases.conf and server.conf look right - the databases.conf should have the correct paths in it.
07-11-2013 02:44 AM
SLP_DSSU_MIGRATION_STATE="1"
LIST_FS_IMAGE_HEADERS="0"
Anyway, i have since uninstalled, reinstalled 7.1, restored catalog, and performed the upgrade.
Took a full 8 hrs, luckily this time it managed to import the old backup images correctly.
I think there's a technote with regards to the setup.exe crashing in 2008 R2 and the resolution was to reboot the server, then perform the install again.
So lessons to be learnt.
Thanks for the help guys.