Unable to read a large no of files on windows

[Sid1987]

Hi Guys,

 We have large no of files on data drive directory of an windows VM and backup doesn't get successful and takes ages. We have suggested vmware snapshots but we don't have licenses. What could be other possible solutions without any cost. SAN client or Flashbackup would also incure cost. It has around 20 million files and around 14k sub folders.

We have opened a case with veritas to perform sas and dev null tests, will see what we can find from there.

I need directions for other possible solutions or temporary remediation

Thanks

Sid

[sdo]

In my experience when they get that big, then you are approaching the limits of the file-system.  It's not a NetBackup limitation, or a NetBackup tunable.  It's an operating system and/or file system limitation.  Those using the operating system and file system have exceeded it's capability.

1) stop adding data, and create another file server - any continued additions to the 10M files will only make things worse

2) move some files off, perhaps back down to 8M files when backups used to work

3) the customer has exceeded usage levels so they now have to pay for alternative backup methods

.

Finally some goofy guesses:

- disable or reduce to zero NetBackup logging on the client

- try enabling Accelerator (but maybe you can't)

- check that the client isn't running the "tracker" - i.e. someone may have mistakenly set that to start or manually started it.

[quebek]

Hey

not sure if possible but maybe you can split this one huge backup into smaller pieces... all depends how data is organised and where it do reside (physical HDDs, RAID, SAN etc)... just an idea...

[sdo]

@quebek is right... here's an extension of that idea...

Imagine/assuming all 10M files are on E: and that E: is 2TB LUN.  And assuming all SMB/CIFS/share clients access share names on E:.  So, why not create three new LUNs, each 500 GB, and mount at F: G: H:.   Now use maybe TreeSize Pro to find the split of folders, and identify four sets of 500GB.  At a quiet time, no users, no backups, offline the share(s), or just down the NIC hosting the shares... now then, copy/move three of the 500GB folder trees away from E: and on to F: G: H:.   Then delete the original (since copied) three folder paths.  Now mount the F: G: H: at the original folder path.

Now you can split your backup job across four distinct paths, four distinct LUNs.

All of the above assumes that the performance bottleneck is limited to SCSI queue depth to FC SAN storage array SCSI queue depth translation for one drive (E:) on one LUN within the guest virtual OS.  All of this assumes that the disk read IO performance throughput issue is not on your SAN - which you haven't proven.

Have you checked your disk IO read latency?  Maybe it's all a case of the disk being really slow, and so all of the above would be pointless anyway.

[Tousif]

Hi

If you have disk storage then go for accelerator and multi streams. Split backup selection in multiple policy.

Thanks

[Mike_Gavrilov]

---

 What could be other possible solutions without any cost.

---

You can use   Raw-patition backup to get rid of all filesystem overheads   https://www.veritas.com/support/en_US/article.v42176572_nbu_core_admin_web

But you'll have to restore whole partition is case you need to restore something or restore everything onto another disk and then copy necessary data into original location.

[Sid1987]

Its a VM with IBM San storage which in turn are datastores

[Sid1987]

Possibly there could be Disk I/O latency, we created a clone VM and we are trying to robocopy E drive to another drive and is taking ages to even copy 80GB. So what could be possible steps for improvement other than SAN to ESX split?

[quebek]

Look after archiving solution... Per me most of these files there do have their atime/mtime more then a year... Do these really have to reside on 'live filesystem'?? I have the same pain here ;( but still trying to have something like Enterprise Vault in place - or at least the manul User Archive being made for these 'obsolete' files...

[Sid1987]

Application vendor is not certain if the older files will be required or not, still there is a clean up going on from Application side which will segregate to different drive. Let's see.

[Sid1987]

Latest depressing update is Windows team is able to backup the drive in fact the entire server through their native tool(wbadmin) in 8 hours and have restored it as well. Now the question infront us How is windows is able to backup that too remotely and netbackup is not able to.

Highly frustrating when Netbackup is compared with other tools.

[GeForce123]

Could be an I/O limitatin on your datastores or your storage units?

We currently have a W12R2 VM with about 240million files and is around 25TB that backs up with accelerator in a VMware policy. We don't do a local agent backup because it would only be1 stream because its only 1 data disk and we can't dig down and break into multiple streams) that finishes a full in 32 hours and a daily in about 12 hours.

We currently get about 200KB/s - 450KB/s throughtput for this job.

Originally this backup would take days to finish, like 80+ hours if it even finished and I noticed that buffers were huge, so I changed the storage unit from an older model array to a newer model array and throughput skyrocketed.