cancel
Showing results for 
Search instead for 
Did you mean: 

Long backup time.

Evan
Level 4

Hey all,

 

I've noticed lately that my backups are taking a long time. We have Netbackup 7 with dedupe and I am running a policy to backup approx 1.3 TB of data onto LTO3 tape. I started the process at 7:13am and it didn't finish until about 3:30am the next morning. So, that was more than 20 hours! That's not normal right? The other thing that I think could be slowing it down is that I ran it with compression on so it wouldn't take up so many tapes, but obviously I lose performance by doing so. But wow, that is a long time! I'm pretty new to backup speed testing so, let me know if I'm wrong, and that's a normal time for a job like this. I'll try to provide more information about my setup below. let me know if you have any ideas/questions!


Thanks!


Evan

 

info:

We have NBU 7, backing up images on a FC RAID array.

We have 2 LTO 3 tape drives

policy said it wrote 19219 KB/Sec

took approx 20 hours

wrote approx 1.3 TB

 

1 ACCEPTED SOLUTION

Accepted Solutions

Marianne
Level 6
Partner    VIP    Accredited Certified

I agree with Quindor - to 'backup' the backups from disk to tape, you need to 'Duplicate' the disk images.

It can be done automatically using Storage Life Cycle Policies.

The data on disk is Dedupe data? You need to understand that duplicating this data to tape will need to 'rehydrate' the backup images, i.e. all data needs to be written to tape and not just the unique segments that was written by the backup image to be duplicated. Performance therefore can be expected to be on the slow side...

See chapter 14 of Administrator's Guide for Windows I  http://www.symantec.com/docs/TECH127079 for detailed info on SLP's.

PLEASE also read the Performance Tuning Guide - it covers SIZE_DATA_BUFFERS_DISK and other 'BUFFER' files that are used for tape...

View solution in original post

9 REPLIES 9

Marianne
Level 6
Partner    VIP    Accredited Certified

Backing up across the network or is this a media server backing up itself directly to tape?

Backing up all data in a single stream? If so, can you divide backup selection into multiple streams?

You need to test each component of the data-path. We know that LTO3 tape is capable of writing more that 60Mb/sec (I have seen more than 100Mb/sec) and transfer rate on a Gigabit network can also do close to 100Mb/sec.

Most important is to test read-speed from disk. See this TN on how to do this: http://www.symantec.com/docs/TECH17541

If data can be pushed fast enough to the tape drives, you can have a look at SIZE_ and NUMBER_DATA_BUFFERS as per Perf Tuning Guide: http://www.symantec.com/docs/TECH62317

Ed_Wilts
Level 6

First, don't use compression - this only helps if the bottleneck is in the network between your client and your media server.  The LTO-3 drives will compress the data.

There is some possibility that 20MB/sec is all you're going to get.  If you have clients with lots and lots of little files, you may not be able to get the data to NetBackup fast enough.  

Marianne gave you a good place to start - there's a lot of detail in the Perf Tuning Guide.

If you do think you're suffering from the "lots of little files problem", so if you can get an evaluation key for the snapshot client and do a FlashBackup test.

Is this 1 client and a single stream?  Does it go faster if you break it up into 2 simultaneous streams?

Kevin_Good
Level 5
Certified

If so, that initial dedupe backup will take slightly longer than a "traditional" backup, and the subsequent backup durations will be reduced (depending upon your change rates)

 

If you were backing it up directly to tape, then the client side dedupe does not give you any benefit.

Quindor
Level 4
Partner

There are so many ways to set this up, it isn't even funny anymore. Also a lot of tuning can probably be done.

Not using dedup : First thing I would do is to check what your fragmentation level of the files on your FC raid are.

Using dedup : If you are indeed using Dedup as a staging mechnism, this might not have been the best choice. While dedup is VERY fast in doing backups (doesn't explain your long backup times) it is rather slow at reading it back (by design).

How are you using this disk at all. You state your policy is backing up images from the FC RAID (configuration of this?).

 

If you wish real help, you will need to describe your situation in more detail. From what I can tell, you have more then enough hardware to backup the 1.3TB. The two tape drives at nominal speed would do it around 1 and a half hour! That isn't completely realistic, but with good tuning and configuration, only using the tape drives 2 to 3 hours is.

So with disks added to the mix (not always faster, but can be for restore purposes) you shoud easily be able to make your backups A LOT faster (if the clients are able to deliver better speeds then they are now).

Evan
Level 4

Thanks for the quality responses! I'll be going through the tips today and post up more info.

Evan

Evan
Level 4

So far, I've found a few things out.

 

I'm saving numerous policies to a our FC RAID drive(H: in this example) which is connected to my media server which is also attached to the tape drive. Then the policy in question backs up that drive (H:). Through reading and some experimentation i found that multiplexing wont help me, in fact I saw my kb/sec basically split in half. 

I checked with the best practices guide and was looking into the data buffers. I noticed that I don't have a install_path\NetBackup\db\config\SIZE_DATA_BUFFERS_DISK file. is that normal? Should I create one or something?

I don't think that this is a case of the small files. I have other backups of dedupe images that might have the problem, but not in this case. I think there are about 200 files in total in this folder.

Looking into bpbkar testing on a client right now...

 

thanks for your help with this all!


Evan

Quindor
Level 4
Partner

Ay, I think you might have to reconsider your whole situation!

Backupping with multiple streams to a single disk volume, although supported is a bad practice in reality. This causes a great deal of fragmentation and will slow down the fastest array to a crawl because of the random read nature (and write) after a while. Special pre-cautions can be taken using settings for the stripe size, format cluster size, buffer number and sizes, etc. But in the end, it will eat your performance away because of how NTFS works and how disks work.

But a bigger problem even, is that the situation you have built is *not* supported and not really useful. You do not backup to disk and then backup that folder again to tape. You will get into trouble if you actually have to restore something because Netbackup will have no idea what is going on.

Sorry to be so hard on you. It's a common mistake, if you’re not really into it, not your fault.

Before testing clients and bpbkar, etc. (which is good practice) you'll need to fix your server side first.

From what I understand, you are trying to use the disks to accelerate your backups. Contrary to what you may have been told, this is not always possible and disk can actually slow down your backups considerably, just like is happening with you now. Only a highly tuned situation can get you a faster backup and completion overall.

Best is to figure out what your fast clients are and which are your slow clients. You can do this by backing them up to disk or tape using a single stream in total. Then you get good KB/sec values in your admin console and judging from that, we can go on.

Slow clients will need to go through disk not to wreck the tape drive, fast clients can go directly to tape with some multiplexing on top of it to get everything going at a good pace. In your information you do not mention if you need duplicate tapes or disk restore ability, so I will assume you do not. If for instance disk restore ability is required, you can add those clients to the “going through disk” selection. But a period to keep on disk cannot be set. This requires a different setup.

Set up your tape drive correctly (please read performance guide very carefully, un-tuned or tuned can give you double if not TRIPLE the performance). And don’t set your multiplex astronomically high, start with 8 and check your BPTM logs to see if you need more or not. Also, setup monitoring of your network cards (NetMeter is a good program to visualize traffic) and performance monitor looking at your tape drives and disk I/O, queue length, etc. to find your bottleneck. Right now, during the backup you will probably see a very high queue length, and during the backup to take probably a queue length of 1. ;)

The disk part takes some work too. Talk with your storage guy on how disks, your array, bandwith, stripe size and clusters work. In the end you’ll probably end up with a few DSSU’s (Disk Staging Storage units) which you can use for your smaller clients. Then, an automatic schedule will take care of duplicating these to tape and freeing space when needed on the disk again.

 

Ok, I’m leaving it at this for now. If you have any more questions, feel free to ask. I will not go into any specific values or such, because well, then you’d have to hire our firm. But I am certainly willing to help and get you on your way!

Marianne
Level 6
Partner    VIP    Accredited Certified

I agree with Quindor - to 'backup' the backups from disk to tape, you need to 'Duplicate' the disk images.

It can be done automatically using Storage Life Cycle Policies.

The data on disk is Dedupe data? You need to understand that duplicating this data to tape will need to 'rehydrate' the backup images, i.e. all data needs to be written to tape and not just the unique segments that was written by the backup image to be duplicated. Performance therefore can be expected to be on the slow side...

See chapter 14 of Administrator's Guide for Windows I  http://www.symantec.com/docs/TECH127079 for detailed info on SLP's.

PLEASE also read the Performance Tuning Guide - it covers SIZE_DATA_BUFFERS_DISK and other 'BUFFER' files that are used for tape...

Evan
Level 4

Quindor and Marianne, Thanks for your input. I inherited this setup from others before me, so I was just continuing a methodology that was clearly outdated. I've setup SLP's which make much more sense. Thanks for the input regarding that!

 

Marianne, I read the performance and tuning guide, however I don't have any SIZE_DATA_BUFFER files in my install_path\NetBackup\db\config\ folder. Where are they so I can change their values?. I read about modifying the buffers, but didn't see where/how you do that. Sorry if thats another dumb question

EDIT- looking back I found it on page 128

"How to change the size of shared data buffers

You can change the size of shared data buffers by creating the following file(s) on
the media server. In the files, enter an integer size in bytes for the shared data
buffer. The integer should be a multiple of 1024 (a multiple of 32 kilobytes is
recommended)."
 
Evan