Fileserver with million files , backup is taking more than 80 hours

[Catalyst]

I have a fileserver cluster windows 2008 with million files , we are using netbackup verions 7.6 with HP StoreOnce Catalyst version 2.1 , Full Backup is taking around 80 Hours + , and incrmental is taking around 14 hours + , i am searching for a solution to minimize the backup windows size for the full back less than 48 hours and the incrmental less than 8 hours , please can anyone help me in that , Accelerator is not supported in storeonce so i need another soluiton, please help ASAP

[ontherocks]

Is Multiplexing/Multistreaming enabled in your netackup ?

There is a single backup policy for fileserver ? or multiple ?

-----------------------------------------

[ontherocks]

Other than Accelerator you can go for Synthetic Backups or FlashBackups for backing up huge fileserver.

[ontherocks]

Check below link :-

https://www-secure.symantec.com/connect/forums/flash-backup-backup-file-server

[Marianne]

I have suggested over here to break down a large volume into multiple streams.

We have done this at many customers and managed to cut down 2 to 3-day backups to 8 hours.

Please only change the policy before a full backup is due, otherwise Incr backups will run as full.

[Catalyst]

ontherocks: Multiplexing is disabled because it is not supported by StoreOnce , Does flash backup support reteriving single file as you do that daily and would you please give me more informaation about Synthetic Backups  , It is done in two backup policies one for each server of the cluster with mulptlie stream up to 16

Marianne: i tried to do that but customer has like 2 folders who has the problem realy realy huge number of files so i  ran one in each backup policy , the file server total backup type is 80 + hours ,first fileserver node take around 50 or more and the other one take the 80 so i am taking the longest duration.

we don't have network or IO bottleneck , but we have the followng the drives are compressed , so it is doing some other overhead or penllity with the number of files, what i understand is Symantec Netbackup scaning each file ( decomprissing and checking ) , so my question also do NTFS change log provider / change journel can help

thanks so much for help

[Marianne · Accepted solution]

Are you using HP StoreOnce as VTL or as OST or CIFS/NFS?

If VTL, it is correct that multiplexing is not supported, but multistreaming is. 
So, multiple streams from the fileserver can be sent to one virtual drive each.
If you break up the Backup Selection into 4 streams as per my example in the link above, 4 streams from the client will be generated and sent to 4 virtual tape drives.
This means that VTL Storage Unit will have MPX set to 1, number of concurrent drives 4 (or more).
Max Jobs per Client on the master set to 4 (or more)
Allow Multiple Data Streams in policy enabled.

This HP StoreOnce Best Practice doc confirms that multiplexing is not supported but that multiple data streams (multistreaming) is recommended for performance: 
http://www.conres.com/stuff/contentmgr/files/0/a1d681554de12de089adc5229589f778/download/hp_storeonc...

Yes, FlashBackup will perform raw backup of volumes with the ability to restore individual files.
See this blog for more info: https://www-secure.symantec.com/connect/blogs/netbackup-snapshot-client-terminology-explained

[Catalyst]

Hello Marianne , I am using OST , the File Server is divided to 14 - 16 Drives divded on two servers , i am using the muliple streams max 16 per policy , but the real problem is that most of the drives finish after 2 -3 hours except the two that has the folders with million of files , i will read more about the flash backup , do you know if NTFS change log provider / change journel can help me or not and if decompressing the drives on anohter LUNS will give better performance.

[Marianne]

NetBackup Administrator's Guide, Volume I     says the following:

■ If the NTFS volume contains more than 1,000,000 files and folders and the
number of changed objects between incremental backups is small (less than
100,000), the volume is a good candidate for enabling NetBackup change journal
support.

Please read the entire section under this topic in Admin Guide: 

How to determine if change journal support is useful in your NetBackup environment
and
Guidelines for enabling NetBackup change journal support

[DG-2005]

+1 to what Marianne states, change journal is most likely your solution as well as breaking up streams of the larger folders in the backup selection.

[Marianne]

....customer has like 2 folders ...... so i  ran one in each backup policy.... 

 Please note this section in the manual before you enable change journal:

■ A NetBackup client using change journal support must belong to only one policy.
To use one policy avoids the confusion that multiple backup settings causes.
Multiple backup settings can cause conflicted update sequence number (USN)
information in the permanent record.

So, rather use one policy with multiple data streams.