backup of many many small files

[Prates]

Good morning!

I would love the tips of you. I have the following scenario.

I have a disk with the total size of 4.2TB in which I have a folder with the total size of 3.70TB. The problem is that I have 29 million small files.

I'm using Flash Backup put this low rate.

Current Rate - 14597 KB / sec
Elapsed Time - 13:52:14
Kilobytes: 725553920
Backup Selection: \ \ \ Y:. \ Example \

What can I change to improve this rate?

Thank you very much.

[Marianne]

Current Rate - 14597 KB / sec

You need to find out where the bottleneck is before trying any performance tuning from NBU point of view.

Is the client also configured as media server (i.e. local backup directly to SAN)?
Or backing up over the network?
Which OS on client?

You can use the bpbkar32 test in this TN to test local backup of the raw device to NUL device to determine disk read capability, but this can take a long time if problem is with disk/LUN layout. 
Alternatively - use Perfmon on Windows client to monitor disk speed.

Next step will be to test network transfer rate...

Some reading matter:

How to benchmark the performance of the bpbkar32 process on a Windows client
http://www.symantec.com/docs/TECH1754

Overview of NetBackup performance testing. 
http://www.symantec.com/docs/TECH147296

Symantec NetBackup™ 7.0 - 7.1 Backup Planning and Performance Tuning Guide 
http://www.symantec.com/docs/DOC4483

About tuning Windows for NetBackup 
http://www.symantec.com/docs/HOWTO56210

[Prates]

Hi Marianne,

The backup done via network and 10gb.
S.O the Client is Windows 2008
is the disc mount point

Was checked and there is no bottleneck. The problem began to occur due to the increase of small files.

NEW_STREAM would be feasible to use this folder for each disc using FlashBackup?

Therefore we tested the FlashBackup disc altogether. Without separating it.

[Prates]

Hi Marianne,

The backup done via network and 10gb.
S.O the Client is Windows 2008
is the disc mount point

Was checked and there is no bottleneck. The problem began to occur due to the increase of small files.

NEW_STREAM would be feasible to use this folder for each disc using FlashBackup?

Therefore we tested the FlashBackup disc altogether. Without separating it.

[Prates]

Hi Marianne, The backup done via network and 10gb. S.O the Client is Windows 2008 is the disc mount point Was checked and there is no bottleneck. The problem began to occur due to the increase of small files. NEW_STREAM would be feasible to use this folder for each disc using FlashBackup? Therefore we tested the FlashBackup disc altogether. Without separating it.

[Marianne]

If you have 2 volumes listed in the Backup Selection, simply select 'Allow Multiple Data Streams' in Policy attributes.
No need for NEW_STREAM.

Ensure Max Jobs per Client is set to 2 or more in Master Server Global Properties.

[Prates]

Marianne,

I have this structure. I thought about creating a way for NEW_STREAM below


Y: \ APP-DISK7-PROD4 \ Systems \ SGIMG \ BACKUP
Y: \ APP-DISK7-PROD4 \ Systems \ SGIMG \ LF
Y: \ APP-DISK7-PROD4 \ Systems \ SGIMG \ LFD


Y: \ APP-DISK7-PROD4 \ Systems \ SGIMG \ Laserfiche \ DEFAULT \ 00
Y: \ APP-DISK7-PROD4 \ Systems \ SGIMG \ Laserfiche \ DEFAULT \ 01
Y: \ APP-DISK7-PROD4 \ Systems \ SGIMG \ Laserfiche \ DEFAULT \ e00
Y: \ APP-DISK7-PROD4 \ Systems \ SGIMG \ Laserfiche \ DEFAULT \ TEMP
Y: \ APP-DISK7-PROD4 \ Systems \ SGIMG \ Laserfiche \ DEFAULT \ VOLUME.ID

Y: \ APP-DISK7-PROD4 \ Systems \ SGIMG \ Laserfiche \ DOCS \ 00
Y: \ APP-DISK7-PROD4 \ Systems \ SGIMG \ Laserfiche \ DOCS \ e00
Y: \ APP-DISK7-PROD4 \ Systems \ SGIMG \ Laserfiche \ DOCS \ TEMP
Y: \ APP-DISK7-PROD4 \ Systems \ SGIMG \ Laserfiche \ DOCS.OLD
Y: \ APP-DISK7-PROD4 \ Systems \ SGIMG \ Laserfiche \ SEARCH

Befits perform this action?

[Marianne · Accepted solution]

So, yo want to stop using FlashBackup and revert back to MS-Windows backup?

If so, I would recommend breaking up Y-drive into 4 streams.
Your last post seems to show 3 streams, right?

Try to break down streams in equal-sized jobs.
Getting this right normall requires careful monitoring and chopping and changing.

Please ONLY make changes when Full backup is due.