Reload Solaris 10 operating system on Master/Media server.

[Willie_SA]

Master/Media server            Solaris 10

Netbackup                          7.1

Due to an 'unstable problematic' server we have decided to reload the Solaris operating system on this server.

Without changing anything, what would be the easiest way of doing this?

The system is still alive and Ok, the server/backups just hangs with 'Failed trying to fork a process' condition.

As per the Systems Engineer the solution would be to reload the OS.

[revarooo]

- Run a catalog backup to ensure you can recover.

- Reload the OS

- Re-install NetBackup to the same version it was when you did your catalog backup

- Recover the catalog

See the Disaster Recovery section in the Troubleshooting Guide

http://www.symantec.com/business/support/index?page=content&id=DOC3683

[mph999]

Apart from you can't run a catalog backup due to the issue ...

So, you must use the last catalog backup you have.

You probably have some backups taken since the last catalog backup, this WILL cause a problem after catalog recovery, as there will be backups on the tapes that are (after recovery) not in the NBU catalog.

You MUST run a NBCC check after the catalog recovery AND before you run any jobs else you risk data loss.  Yes, this means the system will be 'down' longer, but if you want to avaoid 'possible' future problems ...

For example :

Your last catalog backup was 2 days ago

Yesterday, you ran a backup that used a new tape that was previously unused - lets say on TAPE1

You then rebuild /recover NBU

Now, NBU doesn't know there is a backup on TAPE1 (the last vatalog backup was taken before this job ran)

NBU will think the tape is empty, and may reuse it, thus over writing the backup on this tape.

Martin

Also, before you rebuild anything, make sure you have the Catalog DR file.

[Marianne]

In addition to above excellent post, suggestion from my side:

If there is an additional/separate disk in the master server, do a catalog backup to tape as well as to disk. Ensure you have dr-files for both tape and disk backup.

Double-check that you have NBU 7.1 software as well as server and client patch (same as what is currently installed) before you start. Save a list of current license keys on the server.

When system is reloaded, ensure partition sizes are correct (e.g. if NBU is currently installed in /opt, ensure /opt is created same size or bigger). Ensure that additional disk is left alone during installation (must NOT be overwritten with new filesystem).

First prize will be to restore from disk. This will save you from having to configure devices (robot, drives, media) before catalog restore can be done.
Mount disk to same mountpoint as before.

[Marianne]

I am way too slow this morning....

Martin is correct - if system goes into a 'hang state' when doing backups, a catalog backup is going to be a 'challenge'.

Is there ANY 'spare' space available on a non-boot disk to copy away ALL NBU catalogs (with NBU shut down)? We can provide a list of what exactly to copy away.

[revarooo]

> The system is still alive and Ok, the server/backups just hangs with 'Failed trying to fork a process' condition.

All server backups hang or just when the system is loaded

Can you get a successful catalog backup when NO other backups are running?

[MartinVi]

Hi, what we do is to copy the whole netbackup directory via zfs remote copy(zfs send) to the failover host running Solaris 10 x86. In case of an error we can either copy back the directory or configure the failover host to use this zfs filesystem.

[mph999]

... I'm still in shock that I beat Marianne to a post ...

[Willie_SA]

Thank you for all the great feedback.

You all make it sound 'not so difficult'.

The system is back up and in production.

I am planning to do this OS replacement controlled - as in Tomorrow or Friday.

So I will be in a position to do a 100% up to date catalog backup after all backups has completed or stopped. So I will have a 100% backup of the catalog at the time the system goes down.

I do not have any disks connected to the server, except for the normal mirrored second disk.

I will see if I can attach some storage to it - to attemt to get a disk copy of the catalog. Otherwise I will have to go with the tape catalog backup.

How much time should I plan to do such a recovery?

The catalog backup is  172 097Kb + 24 354 592 Kb. I just ran a new one now.

[Marianne]

How long does the catalog backup take to run?

Recovery should take more or less the same time.

Put a whole day aside for reloading - 
Catalog backup first. Maybe 2 if you can find additional storage.
Solaris admin needs enough time to install Solaris plus apply a fairly recent patch cluster. This could take a couple of hours...

Then there's the NBU install (+- 20 minutes) followed by patches (another 20 - 30 minutes?).
Then config devices before starting recovery.

[Willie_SA]

Catalog backups takes about 15 minutes. So catalog recovery will not take long.

Regarding patches.

How can I check if any NBU patches are installed on the system?

I know this might be the ideal oppertunity to apply any non installed patches - but I would prefer to bring the system back exactly the way NetBackup was before the recovery.

[mph999]

Good thinking - look in /usr/openv/pack/NB_<version>

[Marianne · Accepted solution]

Not just 'prefer' - it's a requirement. EMM will not recover and startup correctly if not EXACTLY the same version.
NBU must also be installed in exacty the same folder. On Solaris, NBU installs by default in /opt/openv with symbolic link to /usr/openv.

Another way to confirm patch levels (1st one is Server patch and 2nd is Client patch - both needed on NBU server):
# cat /usr/openv/netbackup/version
# cat /usr/openv/netbackup/bin/version

PLEASE go through the catalog recovery section in NBU Troubleshooting Guide.
Steps are summarized in this TN (but does not replace Troubleshooting Guide): http://www.symantec.com/docs/TECH77448

[Willie_SA]

Please look at this - Looks to me there are no maintenance implimented on this sytem.

Just the original NBU 7.1 shipped. I also compared it to a system I have just freshly installed, without any maintenance - and it looks the same.

/ # cat /usr/openv/netbackup/version
HARDWARE SOLARIS_X86
VERSION NetBackup 7.1
RELEASEDATE Thu Feb 03 18:52:19 CST 2011
BUILDNUMBER 20110203
[root @ cprvnmsnetb]
/ # cat /usr/openv/netbackup/bin/version
NetBackup-Solaris_x86_10_64 7.1
[root @ cprvnmsnetb]
/ #
[root @ cprvnmsnetb]
/ # cd /usr/openv/pack
bash: cd: /usr/openv/pack: No such file or directory
[root @ cprvnmsnetb]
/ #
 

[Willie_SA]

Question - Just an idea.

Remember it is UNIX.

Can I not create a directory (/catalog) and present it to NBU in a way that it could be used as a Disk Storage Unit and then make the backup of the Catalog to that? I only need a couple of Meg.

[revarooo]

Yes you can but don't do it on a LOCAL disk if you are wiping the OS!

[revarooo]

Yes you have base 7.1 with no maintenance patches.

[muhanad_daher]

http://www.symantec.com/business/support/index?page=content&id=TECH61849

http://www.symantec.com/business/support/index?page=content&id=TECH57815

[Marianne]

About your /catalog directory...

Revaroo is correct - if it is not a separate disk or partition (that can be preserved during OS reload) the entire root partition and all of it's sub-folders (including /catalog) will be wiped. (Also not too sure how well mount points are preserved during Solaris x86 installation - my experience is on Sparc.)

If you have another Unix/Linux system with sufficient space, a folder/partition can be shared and NFS mounted on the master. You can then config this NFS mount as DSU.

[Yogesh9881]

Dont know will help us or not but..,

Can we run NBCC on existing catalog and clear all inconsistancy and then we can take fresh catalog backup on tape or disk and then re-install OS. and move ahead ?