Multiplexing & Performance

[l33tpfy]

For the last couple of days I've been looking at different options to improve performance.

Right now i'm trying to evaluate the media multiplexing setting in NB6.0MP6. 8 LTO3 drives in a StorageTekL700e. I've seen quite a bit of mixed opinions of Multiplexing. Most of my policy schedules have 4 currently set with the exception of my FileServer which has 5.

Anyone care to share their experiences with this setting or recommendations? It definitely seems like I might be able to increase the setting by at least a couple and gain some overall throughput to the drive. I really don't feel like the drives are being taxed at all. Ideas for benchmarking the overall drive throughput?

Thanks,

Daniel Godbout

[Dion]

The issues around multiplexing backups to tape are linked to the restore time.  When you multiplex to a single tape drive the tape drive consists of sequencial blocks of data belonging to different servers.  If you multiplex individual servers as well these sequencial blocks will consist of mixed block of data from multiple servers and multiple streams of each server.

The issues that most people have with this is the restore times which increase dramatically and will, most likely, cause you tape drives to suffer from 'shoe shining" which is when you tape drive has to keep forwarding and rewinding the tape to find each beginning block of the next section of the backup instead of streaming the data off.  This can become a major issue because when data goes missing it is generally required in haste!

The other factor that you should take a look at is the network bandwidth congestion of each of you media servers.  If this is the bottleneck, it might not even help to increase the number of streams being pushed through the NICs of the media servers.  In this case you might need to increase your network capacity by either aggregating more NICs into a team or adding more media servers to handle the load.

There are also NetBackup network tweaks that you can look at to help as well.

Cheers

Dion

[CY]

Hi Daniel,

I used to do media multiplexing (from 4 to 6) as well as mutlistream backup jobs.  Nowadays since I have VTL I don't use media multiplexing any more.  My backups go to disks (VTL) first and get duplicated to LTO tapes later with excellent throughput (> 50 MB/sec on LTO-2 drive).

Yes media multiplexing certainly can help with the overall backup performance - it streamline the backups to tape with less drive/media wornout (shoe-shining).  But there is a cost of slower restore, especially when you do restore of DB from multiplexed tapes.

if you don't have VTL, but have enough disk space to create DSU/DSSU, I would recommand to backup to disk first and destage the backups to tapes.  It should give you pretty good backup performance pust saving you some headache of media problems.

[l33tpfy]

Ahh i see. I may already be seeing this issue on my Fileserver so you've given me an excellent point of consideration. Multiplexing may need to be turned down to balance in this case.

Well considering my network design I don't see there being a bottle neck there, but I am continuing to monitor it.

The media servers are configured with dual Gig NIC teams . Some clients are configured to run through a dedicated backup VLAN. Both the client and Media servers don't seem to have a problem there. I think also if that was a problem I would see a variance in throughput at different times and scenarios.

Ok, well let me throw a new aspect to this and possibility rid myself of the shoeshine effect... I've been testing a NetApp Filer (NAS) to be used as a DSSU. When using a DSSU I am precompiling the backup images and later sending them to Tape. If I use this method I shouldn't incur additional fragementation and my restore time should remain excellent (Especially if I have a couple days ready on disk). What do you think??? Shouldn't this allow for multiplexing without the downside?

[l33tpfy]

Thanks CY. Apparantly you and I replied at about the same time, but thinking similarly. I was writing up a reply to Dion and missed the notification.

Anyhow. I do have DSSU as I mentioned and trying to get that up and running the way I want. Excited to hear that this worked for you. Overall I have believed I just have way too many small files that doesn't permit my tape drives to really get put to use. I think 8 should be more than enough, though I do have a lot of clients and data to backup.

Hopefully once I get this up and running I can start to see hear tape drives humm.

BTW, to really see if this is working I need a good way to benchmark the drive thoughput. Any ideas?

[CY]

When you can run backups to disks (whether VTL, SAN storage array, or NAS filers), there is no need to do media multiplexing any more.    The reason is that these storage frames or appliance all have RAID configuration (in the backend) for data/disk protection and so your backup is written to mutliple disks all the time.

[l33tpfy]

Well i'm not sure I could configure it to Multiplex to disk even if I wanted to. I know I can tweak with the Frag, which I set to 10GB. I do plan on allowing multiplexing from disk to tape though.

Thanks for the assistance! This gives me a sense of reassurance...

[CY]

I guess this is how your (new) backup traffic will go:

NBU clients -> LAN or backup VLAN -> NBU Media server -> backup VLAN -> NetApp (DSSU)

and destage to tape:

NetApp (DSSU) -> backup VLAN -> NBU Media server -> SAN -> tape library

Check every link and see how fast you can get.  For example, to benchmark the backup performance to the DSSU, try create a large file on the NBU Media server and copy it (over the backup VLAN) to NetApp's share.  It should give you some idea.

I am concerned if your backup VLAN would be next bottleneck... but you know your environment best so I would stop here.

[l33tpfy]

Good call. Yeah the network does become inundated with traffic with DSSU configured.

IMO, network bandwidth is much easier to monitor, upgrade, and control than tape performance. Right now I have the max jobs pretty limited until I can increase the available NICteam/etherchannel bandwidth.