Updating LUN on master server to use faster disks - can I just copy /usr/openv?

[good2bgold]

We have a Linux master server running 7.6.0.2. It has a separate file system for /usr/openv. Recently, it had a spike in growth and was 99% utilized. File system at that time was 1.0 TB in size. Added 400 GB LUN to the file system to create a 1.4 TB file system and the space issue was alleviated. Unfortunately, the 400GB was on the slowest disks we have in the data center. Now, when we look at large image files that are 2 or 3 months old, the system slows to a crawl while the catalog data that is over 30 days old is uncompressed - iowait time goes to 40 to 50%. Our storage group is going to provide a new 1.4 TB LUN to use instead that has faster disks and a faster communication link, but it will require copying everything from the existing /usr/openv file system to a new file system using the faster disks. Had asked about doing a SAN replication, but the old and new file systems are using the same fabric so the replication option is not available. So, the plan is:

• bring up the new file system as /usr/nbtemp or something like that
• rsync /usr/openv to /usr/nbtemp while NetBackup is running but while activity is relatively low
• Once that initial rsync is complete, shut down NetBackup, remount /usr/openv to a temp location
• remount /usr/nbtemp as /usr/openv
• run a second rsync from the original /usr/openv file system to the new /usr/openv to catch all files that changed
• restart NetBackup using new file system

Should this work?

[Marianne]

You can copy static parts of /usr/openv such as binaries and old images folder, but images being written and the relational db folders can only be copied when activity has stopped and with NBU completely down. There is a section in the NBU HA Guide (about catalog replication ) that explains how only db parts can be copied to new lun and linked back to original folder. This will split installation files and binaries and databases into separate locations which will help with faster db access.

[RiaanBadenhorst · Accepted solution]

Hi,

I'm assuming you already used the volume management software on your server to expand your lun from 1 TB to 1.4 TB. You can use the same volume manage software to mirror your 1.4 TB volume (and file system) to the new 1.4 TB Lun/Disk. 

No need to copy anything and it can be done online.

[Nicolai]

if you are using VxVM/VxFS is volume manger, yes you can do it online as Riaan suggest.

However, if you are using LVM it would use the steps you have outlined in the opening post. 

But before copying everything over, ensure the disk are up to the job. A tool like Bonnie is a good tool candidate

http://sourceforge.net/projects/bonnie/

Example commands using Bonnie

http://www.jamescoyle.net/how-to/913-simple-bonnie-example

[RiaanBadenhorst]

Nicolai,

Does LVM not handle mirroring well? 

Seems straight forward.

       Converts linear logical volume "vg00/lvol1" to a two-way mirror,
       using physical extents /dev/sda:0-15 and /dev/sdb:0-15 for allocation
       of new extents:

       lvconvert -m1 vg00/lvol1 /dev/sda:0-15 /dev/sdb:0-15

       Converts mirror logical volume "vg00/lvmirror1" to linear, freeing
       physical extents from /dev/sda:

       lvconvert -m0 vg00/lvmirror1 /dev/sda

[good2bgold]

Riaan - talked to our Unix admins about your suggestion. They have rethought the way they want to do this and will use a similar plan to what you discussed. Will get our storage team to present the new block device to our VM and will add it to the existing volume group. Then run pvmove to move the data to the new device. When it is finished, run vgreduce to get rid of the old block devices. Any concerns that anyone can think of with this plan?

[RiaanBadenhorst]

Hi

I don't see any issue with this, IMHO its the easiest and safest way to do it. Copying the files implies down time and you're doing it at file level, not block. With a block level operation NBU is not aware that anything is changing. Its still sees /usr/openv as it did previously.

These tools have been available for years so why not use them.

[good2bgold]

Sounds great. Appreciate the help!

[Nicolai]

Our UNIX admin is not too happy with LVM mirroring.