Backup speed is too slow

[qiblatain]

Netbackup server is connected to the File server via LAN. Same Netbackup server is connected to the SAN Switch via Fiber. The File server is also connected to the SAN Switch via Fiber. SAN Switch is connected to one Robot Library having shared tape drive. Necessary zoning required on the SAN Switch has been done by the SAN Engineer to take the traffic on the SAN. We are able to take the backup of the File Server through SAN using its own Media Server. We have done all the necessary tuning & buffering in order to increase the speed, we used the buffer sizes as under

DATA_BUFFER_SIZE =  262144
NUMBER_DATA_BUFFER = 128
NET_BUFFER_SZ = 2097152

 It has increased the speed but just 2 to 3 mb/sec more than previous speed.
The tape drive has a maximum speed of 80 mb/sec & SAN switch has maximum speed of 4 Gb/sec. At starting the backup speed starts at 45 mb/sec, then suddenly starts to decrease and for a long time it sustains at 17 mb/sec, again suddenly decreases and at last the average speed is 10 mb/sec, the total data size of the file server which is backed up is 19 GB.
 
We are using Windows Netbackup server version 6.5.3. This Netbackup server is installed on the 2003 R2. I have made the File Server (FS1), as the second (additional) media server specific for SAN having version 6.5.2. The File Server (FS1) has Windows 2008. This File Server is clustered with another File Server having name FS2.

The more important information is that we have archived the 19 GB data of the file server FS1, Keeping in mind that this is the file server having a lot of small files. It nearly became 16 GB after archiving. After the completion of the backup, it gave the average speed of 73 mb/sec , Which is quite enough and even more than our expectation.

But the problem is that we can not do this archiving on the daily basis & it takes a lot of time for archiving as well. We tried another testing as well that we used “Flash backup for windows” as a policy type for the file server FS1 instead of MS Windows, but the backup failed because “Flash Backup for Windows” policy type in a clustered environment is not supported, As previously mentioned that our file server FS1 is clustered with another File Server FS2.

Can any of you please guide me in the current scenario to give me the answers of the following questions?

1. Is it possible to get more speed than 10 mb/sec, because we have File Server FS1 with a lot of directories & small files OR it is not possible, we can not get more that 10 mb/sec.
2. Is there any necessary performance tuning which we are missing in order to get more speed.
3. Is there any other way by which we can get better results regarding the speed.

 
Your quick response will be highly helpful for me.
 
 
 

[Ed_Wilts]

FlashBackup may be the answer for you even though it's unsupported.  It's been working fine for us for at least 5 years with many releases of NetBackup.  It absolutely works.  I have never tested making any of these clients media servers and I do not know if this will break things.

In general, my statement is that NetBackup tolerates Windows clusters - what I would call full support for a clustered environment isn't there as of 6.x (I haven't checked out 7.0 yet).

What we do is:

1. Create a virtual server, say it's named VS1
2. Put your data volume (let's use D:) in the cluster group on VS1
3. Present a separate volume dedicated to FlashBackup (our's are typically 30GB but it's dependant on your change rate - see the manual on high to size this).  Let's call this this F:
4. Add this FlashBackup volume to the same cluster group - this is critical!
5. Set the shadow copy disk for D: to be F: and set the size to be unlimited
6. Create a new policy for VS1 and tell it to back up \\.\D:\ using the FlashBackup for Windows policy type.

When you've done this setup, the volume will be backed up under the virtual server name and the backups will properly float from file server to file server within the cluster. 

Disadvantages:  You can't limit the number of jobs on your physical systems since NetBackup sees the virtual servers as separate clients.  You'll have 3 clients - FS1, FS2, and VS1.  You'll still need to back up C: & the system state on the physical servers and then treat each individual virtual server separately.

[sdo]

Performance issues are notoriously difficult to nail.  But, if you stick with it - you can find your bottleneck.  Good luck.

This doc, although old, will help you decipher the "waits" between (i.e. both "from" and "to") the data producers and data consumers:
http://eval.veritas.com/downloads/van/4019.pdf

Also, try performing what is known as the "bpbkar to null" test, if this is still slow on aerage then the issue isn't the tape drive:
http://seer.entsupport.symantec.com/docs/242918.htm

Also see the MAX_FILES_PER_ADD setting for media servers here:
http://seer.entsupport.symantec.com/docs/273884.htm


Finally, I may be mis-leading you with this next point - as I have only used it so far as tuning on/for VMware VCB backups which contain a small number of large files, the oppossite to your scenario.  I only mention it here, as I don't know enough to know whether this may help or not:

On our VCB server we use the “bpbkar32” disk read tuning file for Windows clients.  This is a touch value file:
<install-path>\NetBackup\SectorsPerBuffer
...and we have it set to contain the value:
65536
...i.e. specify the quantity of 512 byte sectors to use:
...so, a value of:     4096 =   2 MB
...therefore a value of:  65536 = 32 MB
...i.e. tells bpbkar32 to read the VMDK files in chunks of 32MB, before breaking the data into multiple client "Buffer_Size" fragments for transport to a media server's “NET_BUFFER_SZ”.
Like I said, take this last point with a pinch of salt.  It's quite likely not relevant in your situation, but may be worth exploring/testing.

[Ed_Wilts]

From the 6.5.6 release notes at ftp://exftpp.symantec.com/pub/support/products/NetBackup_Enterprise_Server/341279.pdf

(ET2003460) Support for FlashBackup in a Microsoft Cluster (MSCS)
environment

Beginning in NetBackup 6.5 GA, the use of FlashBackup in a Microsoft Cluster
(MSCS) environment is supported, with the following limitation: Raw partition
restores can only be performed when the disk being restored is placed in
extended maintenance mode or removed from the MSCS resource group.

Note: Earlier versions of MSCS (such as those versions that were shipped with
Windows versions before Windows 2003 SP1) do not allow extended
maintenance mode functionality. If the cluster does not support placing disks
in extended maintenance mode, it is still possible to perform raw restores to
an alternate, non-shared disk.

[CRZ]

DOCUMENTATION: Statement of support for NetBackup's FlashBackup in a Microsoft Cluster Server (MSCS) environment.
 http://support.veritas.com/docs/347010

[J_H_Is_gone]

To others on the forum - I thought that to use the SAN to backup you had to set up the Fiber Transport. (of which you have to have the correct HBA card on your server)

If you just setup a SAN media server and zoned the tape drives to the server - you were not really using the SAN for backup.  The data still goes to the Media server then to the tape drive, that it does not go from the disk directly to the tape drive.

Am I confused on this?

[qiblatain]

Dear All,

Thanks a lot for your replies.

My main question is about archiving the data for the backup of the file server. we archived 19 GB data, and it became 16 GB, and was show as a single file. The average speed for this one file having data of 16 GB hava a speed of 73 MB/Sec, But similarly without archiving this data with 19 GB size, having nearly 48000 files, hava an average speed of 10 MB/Sec (even though this speed is on SAN for File Server).

Don't you guys  think, which i observe but not sure, that this slow SAN speed problem is due to the file structure and a lot of no. of files for the File Server?

Waiting for your response.

[qiblatain]

Dear All,

What you guys recommend " Is File Server is optimum for the SAN backup OR its not the better choice for the SAN backup because of the structure of the File Server (having so many files & for writing of every file header is modified according to the writing algorithm).

Waiting for your response.

THANKS.