Highlighted

Netbackup Linux Bare Metal Restore using Open Source "Relax and Recover" (ReaR)

Just wanted to update NetBackup users about a open source tool that I have been using to successfully do bare metal Linux restores using our Netbackup.  We use netbackup 6.5 to backup our Linux and AIX servers. On our AIX servers we use a NIM server and we do network sysback backups on a weekly basis to provide the "Bare Metal" backup functionality.  Unfortunately we had not Bare Metal restore functionality for Netbackup.  I found a Open Source tool called "Relax and Recover" that provided a way to restore the OS using the full backups of the Linux systems on NetBackup.  The tool as an option where it will create a bootable rescue iso file that contains all the Netbackup client executables and configuration to allow you to boot up your Linux server using the iso file then issue a simple command that will mount your OS disks separately and do a bprestore of all needed files to your actual OS disks.  At the end of the restore the system re-create the GRUB information on the boot disk.  The "mkerscue" is process can be ran any time with the Linux system is up and I just store that iso file on a separate server.   The Netbackup code on the "Relax and Recover" was lacking some functionality that I wanted.  Working with the developers they have accepted  enhancements I made to the Netbackup (NBU) code.   I added the functionality of excluding files and/or volume groups in the restore.  I wanted this because I want to mainly use this to restore the OS only.  I also added the functionality to restore from a point in time.  They accepted the changes in SVN trunk 250.   It will probably be in the next release of the product which should come out in one month's time.   The web site for the product is: http://rear.sourceforge.net.   They have a very useful program that has a very active supportive user base.  The developers were very responsive and helpful to me to get the enhancements to the Netbackup interface in place.  The program itself is just a big collection of bash shell scripts.  This made adding the enhancements "doable' for a UNIX administrator such as myself.  It is nice to be able to do a bare metal restore to our Netbackup backed up Linux servers without the need to do some type of clonzilla, "ghost" type backup which requires a system downtime..

17 Replies

one small caveat

Have to submit Planning And Information Notice (PAIN) for enhancements which might justly be called PAIN and the ReaR. smiley

Happy April 1 everybody!

Good joke on that..

Good joke on that.. Smiley Wink

 "Relax and Recover" will do Bare Metal Linux restores using your backed up file on your Netbackup servers.  Just have to get used to the name of this great open source progam being: "ReaR"..res tor  sdfsdf

http://rear.sourceforge.net

Hi djembe65,   you seem to

Hi djembe65,

 

you seem to have good experience with REAR. I am now testing the software for our company. Maybe you could give me a lil advice. I would like to know if it is possible to create an ISO file which would include except of the boot part also system bakcup files. So i could fully recover my OS from such a ISO burned on cd/dvd the same way as it works with USB stick. Simply to recover from cd/dvd instead of bootable USB.

 

Please let me know. In case it could be done give me also a hint how to update the local.conf file.

Thanx very much.

 

Best regards,

 

bassmann22

Yes ReaR works great using

Yes ReaR works great using your full Netbackup backups to do "Bare Metal" Type restores for Linux.   I only restrore the OS portions using this product.  I worked with the developers to modify the NetBackup recover portion to allow excuding filesystems with this restore.  For bare metal recover you usually do not want to also restore larger "data", "app" filesystems.  Those can be restored if necessary after the Linux server is "bare metal" restored.  I also got coded added which allows a point in time recover as well.

 

Here is an example of my /etc/rear/local.conf file:

# Create ReaR rescue media as ISO image
OUTPUT=ISO
ISO_DIR=/sysadmin/reariso
ISO_PREFIX="$HOSTNAME.rear.$(date +%Y-%m-%d.%H%M)"

# optionally define (non-default) backup software, e.g. TSM, NBU, DP, BACULA
BACKUP=NBU

# the following is required on older VMware VMs
MODULES_LOAD=( vmxnet )

# to see boot messages on the serial console (uncomment next line)
# KERNEL_CMDLINE="console=tty0 console=ttyS1"

TIMESYNC=NTP

EXCLUDE_MOUNTPOINTS=( "/var/spool/asterisk/voicemail" )
EXCLUDE_VG=( "gfs_vg" )

PROGS=( "${PROGS[@]}"
ifconfig
ssh-keygen
vi
vim
view
lvchange
lvconvert
lvcreate
lvdisplay
lvextend
lvm
lvmchange
lvmconf
lvmdiskscan
lvmdump
lvmsadc
lvmsar
lvreduce
lvremove
lvrename
lvresize
lvs
lvscan
lvsd
lvm.static
vgs
vgscan
vgchange
vgdisplay
lsof
rmdir
passwd )

 

This is the df output:

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-rootlv
                      9.8G  7.2G  2.6G  74% /
/dev/sda1              99M   79M   17M  83% /boot
tmpfs                 438M     0  438M   0% /dev/shm
/dev/mapper/gfs_vg-gfs_lv
                      764M   40K  764M   1% /var/spool/asterisk/voicemail
 

This is a test asterisk server.   I did not want to restore the gfs filesystem.  I also added many commands I wanted to be able to use when I am booting from the iso file.

 

This product is a lifesaver for us.  Netbackup eliminated thier "bare metal" recovery oprion for Linux.  This program works great in that capacity.
 

 

 

If you are using netbackup as

If you are using netbackup as a backup software, NB-BMR, an intergrated disaster recovery option would be easy and better to protect your clients from DR.

NB-BMR does not require any additional license and comes default with NB standard deployment.

BMR supports DR for almost all main operating systems and latest versions.  BMR also gives flexible features like Dissimilar Disk Restore and Dissimilar System Restore (on windows automated and a tech-note for unix/linux).

Please check the support matrix and BMR admin guide for more details.

Thanks.

Mandar

Tried NB-BMR and dropped any attempt at using it

Thank you for that information but we though looked into the product you are describing.   We received no assistance using this product from Symantec.   Using this product requires a special backup and extra separate servers to implement.  We were VERY FRUSTRATED  implementing this product and gave up on it.

The open source Relax and Recover (ReaR) product is simple and works wonderfully.  It does not require separate servers or special backups to use.   We now have the ability to bare metal restore on our Linux servers using the EXISTING regular NetBackup backups on our system.

Relax and Recover product creates a bootable iso file that is used when a "bare metal" type of restore is needed.  I have configured that backup to only restore the OS and not my larger data volume groups / filesystems.  When booted off of the ISO file all needed Netbackup executables to connect to our Netbackup server are available to initiate a bprestore to the separately mounted OS filesystems.  

The user group is very active.  A received excellent feedback and suggestions and I was able to enhance the NetBackup portion of the product to enable point-in-time restores and to implement the ability to exclude filesystems.

I am using the Relax and Recover to simply restore the OS on the same server.  I am not using for cloning servers etc...  I believe there are ways to do that as well.   I only wanted to have the ability to restore the OS without having to do a re-install of Red Hat from installation media.  This open source product does this simply and well.

Web site is at http://rear.sourceforge.net

Thanks for detailing your

Thanks for detailing your pain points.


Below are my comments. It would be great if you can again put thoughts and give a try using NB-BMR. I am sure this time your experiance would be different. :-)
Looks like you have not received good support on the issues or not received good guidence on correct NB-BMR implementation. There are many benefitial features provided by NB-BMR and as I said as it is tighly intergated with NetBackup, hence it is easy to manage your DR through NB-BMR combination.

1.
>> We received no assistance using this product from Symantec.
Would it be possilble for you to re-open any Symantec support case related to this or can you take Symantec system engineer into loop? We (support, system engineer and Development) can help you guiding and solving your DR issues.
In case required we can also manage giving a BMR demo.


2.
>> Using this product requires a special backup and extra separate servers to implement.
Extra server is really not a requirement.
BMR provides restore either via network boot or media based restore.
Some customers prefer NW boot as they prefer multiple clients restore simultaneously. In this case BMR boot server is required as clients do NW boot over PXE/DHCP.
In case of media boot, you do not require a dedicated BMR Boot server. You can instead use one of the clients to enable BMR Boot server temporarily, create ISO and then disable back. Now this media ISO serves reatore exactly like you are using in ReaR. Even restore is totally automated and you do not require any manual work to use bprestore restoring file systems from NB.

Using BMR you can also flexibly manage dissimilar disk restore using BMR admin GUI. (operations like change original volume sizes, mapping original volume to other disk, change volume group sizes, RESTORE ONLY REQUIRED VOLUMES or ONLY OS etc can be done easily.)

BMR BOOT SERVER IS VERY LIGHT COMPONENT, USED ONLY TO CREATE SYSTEM RECOVERY CRITICAL SOFTWARE AND PROVIDE NW BOOT SUPPOT. You can think about below BMR BS deployments:
A.

In case of NW based restore, if your master server is of same OS family then you can enable BMR bs on the same machine. - YOU DO NOT REQUIRE ANOTHER MACHINE FOR BMR BS.

B.
You can even install it on Virtual machine instance with low resources. YOU DO NOT REQUIRE PHYSICAL HARDWARE.

C.
As mentioned above temporarily enable BMR on a client; have ISO created and disable it back. BMR ISO will now be an option for BMR boot server.


I hope by this time you are thinking to give another try using NB-BMR. :-)

Thanks.
-Mandar

Thanks for the NB-BMR info

Thank you for the additional Infor on NB-NBR.  That is the most info I have recieved on this.  Our Backup administrator attended a week long class  of the product and the Instructor did not reccomend using this product.   That was 2 years ago.   I guess there have been improvements made since then.  I forwarded your information and we will look into trying the product once again.  I am interested in the method of enabling the BMR on the client just to create the Bootable ISO.   Our administrator still has some concerns about any additional backups this may require.

Creating Bootable ISO is a

Creating Bootable ISO is a simple process which can be even be done on the client itself by enabling BMR BS temporarily on it. Regarding additional backup requirements, BMR enabled NBU policy triggers an additional BMR backup which is of few MB (about 2-3 MB) and the information directly goes to NB-BMR database. Also it hardly takes a minute to complete BMR backup.

More details for the administrators are also available on the BMR Admin Guide.

Hi djembe65 >> Our Backup

Hi djembe65

>> Our Backup administrator attended a week long class  of the product and the Instructor did not reccomend using this product.   That was 2 years ago.

Will it be possible for you to provide the conference/session details?

Thanks.

-Mandar

hi djembe65 - Your NBU training experience

'Our Backup administrator attended a week long class  of the product and the Instructor did not reccomend using this product.'

As a NetBackup and BMR instructor for Symantec I would be interested to know where your backup administrator took his/her training. Was it at a Symantec training site or at a partner site? Was this for NetBackup (a five day class) or BMR (a three day class)? It wasn't clear from your post. Thanks.

To be honest the

To be honest the administrator took the class over 3 years ago so I think I would rather move forward and test NB BMR  We have created a good Lab NetBackup so I can now test the NB-BMR.  I continue testing the Open Source relax and Recover product  I used the product recently to Clone a Linux server using NetBackup.  I am going to give the NB-BMR product a good testing now as well.   

Still Prefer Relax and Recover

I attempted to install NB-BMR in our Lab environment.  I had a difficult time setting up a RedHat System Resouce Tree (SRT). It was frustrating that I had to use the 5 install CD's and not just the one DVD OS install disk to set this up.  I was not successful creating the SRT because  the copy failed at disk 5.  At that point I gave up setting up BMR.   Additionally you had to check that you were saving DR info on the policy to use this type of backup which created 2 backup jobs for each backup.   I am sure I could have got this working but it seemend overly complicated.

Using Relax and Recover I only had to Install one rpm file on the client and edit a local.conf file. One minute and the install is finshed.  I had no issues doing Bare Metal Linux Redhat and Centos restores using the regular backups that already exist on the NetBackup server.  Again, much more simple.  I was able to clone servers and do bare metal restores to different disk sizes as well..

If we upgrade from NB 6.5.6 to 7, I may give NB-BMR another look.  for now I am pleased with the simplicity of Relax and Recover for bare metal Linux restores from our Netbackup backups.  The delevolpers are very responsive to fixing issues and implementing patches from the community as well.

Providing a single DVD

Providing a single DVD (instead of multiple CDs) also works fine during SRT creation process. You need not provide multiple CDs.

As you are currently on NBU 6.5.6, in order to understand the failure cause, can you please let us know the RHEL version of the boot server on which the SRT is being created? Also let us know which RHEL version SRT you were trying to create?

I believe BMR on NBU 6.5.6 is qualified till RHEL 5U4.

I use NB-BMR a lot

I admit it can be tricky and there are some hurdles if you are at 6.5, but they are all up front.  Just with any software there is a learning curve. Once you get it all set up, it is very automated and there are lots of nice things you can do with it.  For example, you can create your own scripts to run at specific places during the restore process (External Procedures).  I've used a pre-restore external procedure to split /usr into /usr and /usr/local for example.

If you are doing a point in time restore with ReaR, you also need to do the same kind of Backup that BMR requires for the same reason - collect TIP with Move detection.  The bmr configuration for Linux is almost always less than 1 MB - usually far less.

Unless you are familiar with getting a PXE server, DHCP server, and NFS Server up and running on Linux, just use a CD Based SRT.

Did you ever get ReaR to work

Did you ever get ReaR to work from a Tape based backup? I've got it booted, it adds a restore task to the activity monitor, but then it just sits there saying it needs the tape (even though the tape is in the drive, and the Device Monitor shows the drive with the correct tape in it and in theready state) in restore details. The NBU console never shows any files transfer to the ReaR boot CD, even if I wait an hour...


Any ideas?

Re: Still Prefer Relax and Recover

djembe65, thanks for the information. I know this is an old thread, but this may help other users. I too when thru the same as you but with NetBackup 7.x and 8.0. I tried to set up an SRT for Oracle Linux 6.9 which is a fork of Redhat 6.9 and was never able. The Netbackup software is unbelievable difficult and after months trying to Oracle Linux 6.9 and taking out two tickets, I was told on Dec. 5, 2017 that Veritas is not supporting any Oracle Linux greater than 6.6 and only upto Netbackup 7.5x.

I have currently looking at other backup software such as Acronis that will complete a bmr without issues and I have started looking a "Relax and Recovery" as well.

Just want everyone to know that there is other bmr software that is either open-source like REaR and Acronis that is easier to use and does not cost as much. I also think that NetBackup may not support RHEL as well.  If someone has been able to create an SRT and recovery with Netbackup 8.0 or greater any Linux and you found it easy to do let me know.

 

 

Tags (2)