Restore of many small files on ntfs, improve speed ?

Buffer sizes makes no difference to retores - the data is read at using the same buffer size it was written with - it's impossible to change it.

I suspect this will be a OS limitation, and writing to NFS on the top of that won't be helping due to the 'NFS overheads'.  I'm not  a windows person, but ceertainly have seen similar with Unix, with the limitation being how fast the OS could create the inodes.

The TN you refer to related to restore 'single' files from backups with a large number of files and large fragment size.  This is less relevant these days as tape drives have fast block positioning.  Also, you seems to be restoring many small files, not a single file out of many.

Best option would be to redirect the restore direct to the server that has the NFS share, if possible - this at least removes the NFS overhead.

The above are just my first thoughts, noyt enough info to make further comments :

Eg.  What tests were run, were the test restores of the same / similar data to an NFS location, or restore of a large file to none NFS ...

The number of data buffers can be cahanged for a restore, I've never seen an improvemnt doing this, and the restore would have to be restarted to pick up the change - it depends where the bottleneck is as to if this would have any difference.  We need to see really the '... was delayed xx times' lines from activity monitor to get some idea, but unfortunately these are only written at the end of the job.

Hello,

try to simply copy (outside NetBackup) these files between two NTFS's, you will probbaly get a similar speed. So I guess the speed limitation is also outside NBU.

Consider FlashBackup as your backup/restore method, at least for the whole parititons loss scenario.

Or VMware based backup when it is a VM (again, for the whole sever/partition loss. When restoring only subset of many small files from these backups, restore will be slow again)

Regards

Michal

Was more thinking along the lines of something that could be done to improve writing many files in the NTFS system. Can find a lot about improve reading files, but nothing about writing

Tests was done with sqlio and copy of 1 TB file both locally and through the network.

Restoring to original or new filesystem?

Heavy fragmentation maybe a problem...

One thing that Martin and I disagree about  - IMHO STU fragment size is still applicable to restores.

Additional to fragmentation you should ask your Windows-admins to tune the ntfs using fsutil.

Another bottleneck I have seen in the past are the cheap "high performance" zero-chache raid contollers. This makes no sense in Windows server with ntfs.

Where does the higher speed in tests come from ? Copying small files ?

ciao

Martin

User "Performance Monitor" and/or "perfmon.msc" to view your disk "write queue length" for the logical and physical volume counters.  Anything odd seen?

What is the NTFS volume's cluster-block-size-factor (use fsutil to find out)?

Have you ever run a script to find out what your quartiles are for file sizes?  Maybe a cluster factor of 64KB would give you better performance (but at a wasted space cost/overhead - hence understanding by analysis the distribution of "file sizes" versus "file count" - i.e. how many are in which size ranges?).

FYI - in NTFS, any files (plus ACL length) that are smaller than a few KB (I can't remember whether it is 1KB or 4KB) are actually stored in the MFT itself (master file table) - i.e. whilst they appear to be in a folder in Windows Explorer (and "dir" commmand) they are actually in the MFT, which is supposed to speed things up.

When one initializes an NTFS volume, when using CLI one has the option of specifiying "how large to pre-allocate" the MFT.    (Here's a tip, unless you have some really old physical spindles then don't worry about the location of the MFT.)

So, maybe your MFT has become ridiculously fragmented?

Could you elaborate on tune with fsutil ? most of what I have found is concerning improving read speeds from a file server.

8.3 names is disabled and ntfsmemoryusage is set to 2, changing the ntfsmemoryusage from 0 to 2 did not seem have any effect, guess it is mostly relevant when reading files.

Not if fast block positioning is used ....  - which it should bewith 'modern' tape drives.

However, I would agree that if the backup is multiplexed this may not be the case ...

:0)

Michael, are you restoring to a SAN LUN?