cancel
Showing results for 
Search instead for 
Did you mean: 

Restore of many small files on ntfs, improve speed ?

Michael_G_Ander
Level 6
Certified

Is restoring some file servers with a fair amount of files, and is getting a poor performance 30-60 MB/sec compared to the hardware and tests 200-330 MB/sec.

Have tried to play around with the buffer sizes, but the biggest factor seems to be the number files restored, so suspect we are waiting on something in ntfs as the client is a Windows 2012R2 server with 10Gbit connection.

Found the technote below in an old thread, but cannot seem to find the current equivalent

DOCUMENTATION: Why are file system restores are running very slow with large number of small files?
http://seer.support.veritas.com/docs/279061.htm

 

The standard questions: Have you checked: 1) What has changed. 2) The manual 3) If there are any tech notes or VOX posts regarding the issue
1 ACCEPTED SOLUTION

Accepted Solutions

Michael_G_Ander
Level 6
Certified

Have talked with one of our windows guys, it seems that is a limitation of the way ntfs works when writing files so there is nothing do.

Untill either Microsoft rewrites the code or Veritas comes up with an accelerator for restore.(Hint, Hint Veritas)

The standard questions: Have you checked: 1) What has changed. 2) The manual 3) If there are any tech notes or VOX posts regarding the issue

View solution in original post

20 REPLIES 20

mph999
Level 6
Employee Accredited

Buffer sizes makes no difference to retores - the data is read at using the same buffer size it was written with - it's impossible to change it.

I suspect this will be a OS limitation, and writing to NFS on the top of that won't be helping due to the 'NFS overheads'.  I'm not  a windows person, but ceertainly have seen similar with Unix, with the limitation being how fast the OS could create the inodes.

The TN you refer to related to restore 'single' files from backups with a large number of files and large fragment size.  This is less relevant these days as tape drives have fast block positioning.  Also, you seems to be restoring many small files, not a single file out of many.

Best option would be to redirect the restore direct to the server that has the NFS share, if possible - this at least removes the NFS overhead.

The above are just my first thoughts, noyt enough info to make further comments :

 

Eg.  What tests were run, were the test restores of the same / similar data to an NFS location, or restore of a large file to none NFS ...

The number of data buffers can be cahanged for a restore, I've never seen an improvemnt doing this, and the restore would have to be restarted to pick up the change - it depends where the bottleneck is as to if this would have any difference.  We need to see really the '... was delayed xx times' lines from activity monitor to get some idea, but unfortunately these are only written at the end of the job.

 

Michal_Mikulik1
Moderator
Moderator
Partner    VIP    Accredited Certified

Hello,

try to simply copy (outside NetBackup) these files between two NTFS's, you will probbaly get a similar speed. So I guess the speed limitation is also outside NBU.

Consider FlashBackup as your backup/restore method, at least for the whole parititons loss scenario.

Or VMware based backup when it is a VM (again, for the whole sever/partition loss. When restoring only subset of many small files from these backups, restore will be slow again)

Regards

Michal

Michael_G_Ander
Level 6
Certified

Was more thinking along the lines of something that could be done to improve writing many files in the NTFS system. Can find a lot about improve reading files, but nothing about writing

Tests was done with sqlio and copy of 1 TB file both locally and through the network.

The standard questions: Have you checked: 1) What has changed. 2) The manual 3) If there are any tech notes or VOX posts regarding the issue

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Restoring to original or new filesystem?

Heavy fragmentation maybe a problem...

One thing that Martin and I disagree about  - IMHO STU fragment size is still applicable to restores.

tunix2k
Level 5
Partner Accredited

Additional to fragmentation you should ask your Windows-admins to tune the ntfs using fsutil.

Another bottleneck I have seen in the past are the cheap "high performance" zero-chache raid contollers. This makes no sense in Windows server with ntfs.

Where does the higher speed in tests come from ? Copying small files ?

 

ciao

Martin

sdo
Moderator
Moderator
Partner    VIP    Certified

User "Performance Monitor" and/or "perfmon.msc" to view your disk "write queue length" for the logical and physical volume counters.  Anything odd seen?

What is the NTFS volume's cluster-block-size-factor (use fsutil to find out)?

Have you ever run a script to find out what your quartiles are for file sizes?  Maybe a cluster factor of 64KB would give you better performance (but at a wasted space cost/overhead - hence understanding by analysis the distribution of "file sizes" versus "file count" - i.e. how many are in which size ranges?).

sdo
Moderator
Moderator
Partner    VIP    Certified

FYI - in NTFS, any files (plus ACL length) that are smaller than a few KB (I can't remember whether it is 1KB or 4KB) are actually stored in the MFT itself (master file table) - i.e. whilst they appear to be in a folder in Windows Explorer (and "dir" commmand) they are actually in the MFT, which is supposed to speed things up.

When one initializes an NTFS volume, when using CLI one has the option of specifiying "how large to pre-allocate" the MFT.    (Here's a tip, unless you have some really old physical spindles then don't worry about the location of the MFT.)

So, maybe your MFT has become ridiculously fragmented?

Michael_G_Ander
Level 6
Certified

Could you elaborate on tune with fsutil ? most of what I have found is concerning improving read speeds from a file server.

8.3 names is disabled and ntfsmemoryusage is set to 2, changing the ntfsmemoryusage from 0 to 2 did not seem have any effect, guess it is mostly relevant when reading files.

The standard questions: Have you checked: 1) What has changed. 2) The manual 3) If there are any tech notes or VOX posts regarding the issue

mph999
Level 6
Employee Accredited

Not if fast block positioning is used ....  - which it should bewith 'modern' tape drives.

However, I would agree that if the backup is multiplexed this may not be the case ...

:0)

sdo
Moderator
Moderator
Partner    VIP    Certified

Michael, are you restoring to a SAN LUN?

sdo
Moderator
Moderator
Partner    VIP    Certified

See "tunix2k" point... maybe your battery BBU has died in your RAID card?   If any of these are true, then your disk write speed in pretty much what it is...

1) RAID HBA BBU discharged or mal-functioning

2) missing, never ordered, stolen, gone for a Burton?

3) RAID BBU cache set to 100% read

...

If your disks are SATA 7200, then what did you really expect?

N.B:

SAS 15k = max 160 IOps

SAS 10k = max 120 IOps

SAS 7.2k = max 70 IOps

SATA 7.2k = max 60 IOps

...sometimes the numbers above are a little higher, but use these numbers above in your calculations is probably a good idea - to err slightly under, rather than err slightly over.

...

Basically, Windows has awful random write IO, NTFS is a "scatter" file system.  Every write to RAID will very likely involve... read stripe, replace data, XOR parity, write stripe... because the writes have to be committed in an atomically correct order.

Add a RAID HBA BBU and lots of RAID HBA cache, set it to say 50/50 or 72/25 read/write, or why not tweak it to 25/75 read/write whilst the restore is running... and you really should see an improvement.

.

Perfmon.msc will reveal your write IO profile - look at write reponse time, average write IOps, average write size/length, write throughput, write queue depth - check both logical and physical "disks" counters in perfmon.

sdo
Moderator
Moderator
Partner    VIP    Certified

Poeple who implement Windows servers without a BBU + cache on their RAID HBAs should be shot!

A Windows server without a RAID BBU + cache is basically a laptop with bigger fans (when it comes to en-mass restore of millions of files).

Michael_G_Ander
Level 6
Certified

Is restoring to a SAN lun, and we cannot see anything problematic on the SAN expect that it only receives about 100 IOPS, and the disks in raid groups under the LUN should be able to handle about 900 IOPS. The machine I restore to, is the only machine connected to these raid groups.

Restoring to a internal disk gives the same picture.

The write queue length is almost non-existant in case of the SAN lun.

Tape/dedup disk is not the issue as the media server parent bptm is waiting on the child bptm, which means the client.

A strange thing is that windows reports a high reponse time on the SAN lun, but the storage system report a low reponse time at same time when moving data.

Copying a set of restored files from the media server to client, gives more or less the same resulst as a restore directly to the client.

 

 

The standard questions: Have you checked: 1) What has changed. 2) The manual 3) If there are any tech notes or VOX posts regarding the issue

tunix2k
Level 5
Partner Accredited

Copying a set of restored files from the media server to client, gives more or less the same resulst as a restore directly to the client.

Looks like you cant find a netbackup related solution.

On SAN Storages you can have the same behavior like with local RAID Controllers. Is write back enabled ? Is there are a BBU on RAID controllers? (Same problems like SDO stated are possible) Is the client connected to both storagecontrollers ? (at least to the preffered controller)

sdo
Moderator
Moderator
Partner    VIP    Certified

Looks like a SAN LUN issue.

Correct multi-pathing policy?

As tunix2k says... about preferred controller... some storage arrays/shelves say they are "active/active"... but if you don't access via the preferred path... then it results in what some vendors refer to as a "LUN trespass"... NetApp have a different name for it... anyway, it all results in horrible slowness because the "controllers" have to talk to each other and pass IO between themselves instead of talking natively to the actual storage themselves.

No write-back cache on the storage array?  Or depleted write-back-cache or swamped or overloaded?   All result in IO having to actually be written rather than cached for later stripe de-staging.

You say you have lots of spindles at RAID parity-group, but what type are they?  If they're SATA 7.2k then what you see is what you get.  Remember the whole stripe has to be written.  If you don't have a write back cache then when a write occurs then all disks in the parity group all have to write at the same time.  Not good.

Maybe someone didn't put your new LUN in the correct 'tier' or 'pool' - and so your LUN is going through any write back cache at all?

Michael_G_Ander
Level 6
Certified

All this with cache and disks I would expect to see under the sqlio and copying 1 TB file too, where the performance was very good. Actual measuring shows a write-pending around 4% under restore, which should not be problematic.

Running to SAS-nearline, but 36 disks should be able cope with more than 10O IOPS even with the write penalty from raid6. Vendor says 900 IOPS, my conversative disk only calculation gives 480 IOPS.

There is no tiering involved, it is an "old-fashion" LUN.

Thinking there is something either in the file system or the path from the server to storage, we just have not identified yet.

Unfortunaely I don't have access to all the elements in the path.

 

 

The standard questions: Have you checked: 1) What has changed. 2) The manual 3) If there are any tech notes or VOX posts regarding the issue

sdo
Moderator
Moderator
Partner    VIP    Certified

36 disks?  Sounds like a NexSAN array.  Correct?

Michael_G_Ander
Level 6
Certified

Have talked with one of our windows guys, it seems that is a limitation of the way ntfs works when writing files so there is nothing do.

Untill either Microsoft rewrites the code or Veritas comes up with an accelerator for restore.(Hint, Hint Veritas)

The standard questions: Have you checked: 1) What has changed. 2) The manual 3) If there are any tech notes or VOX posts regarding the issue

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
In all fairness - this is exactly what sdo said on 29 Jan...