cancel
Showing results for 
Search instead for 
Did you mean: 

Backing up 180TB, 15825579 files @ 185KB/s :(

o1crimson1o
Level 4

We normally see good performance (what I expect from nfs, 10GB connection and 6x LTO7 drives).

 

But currently I am at 168 hours, 108,989,304 kilobytes backed up.

 

I've checked network, media server, robot, and nas storage. All are opereating normally and optimally. When I try rsyncing files off I'm seeing speed I would normally expect.

But in terms of netbackup I have no idea where to even start looking to see the issue. I mean backing up at like 185KB/s seems completely ridiculous.

 

Thoughts?

7 REPLIES 7

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

There was another post recently where backup over NFS was very slow.

https://vox.veritas.com/t5/NetBackup/ZFS-BACKUP-SLOW-THROUGHPUT/td-p/854727

You will see that I suggested to use tar as a test to simulate the NBU backup process (without actually using NBU).

The OP solved the problem by changing the mount options on the master/media server. 

Testing with tar, I get the speed I would expect. Even rsync across the nfs share.

 

This is def something related to Netbackup :\

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

If you feel that NBU is at fault, you should log a call with Veritas Support.

mph999
Level 6
Employee Accredited

How fast is it if you just copy some files from the NFS share to the local file system o the media server ?

Additionally, make a small backup from the NFS share, just enough to watch it go slowly for say 10 - 15 mins - then when the backup complates, look in the Details Status of the job for the lines that look like:

bptm waited xx times, delayed yy times

sdo
Moderator
Moderator
Partner    VIP    Certified
Please can we have a detailed description of: 1) the networking and routing that your NFS and rsync traffic traverses 2) the networking and routing that your backup traffic traverses . hint: don't miss any interfaces / cables along the way of both paths.

Genericus
Moderator
Moderator
   VIP   

I have had similar issues with backups where the total was not too outrageous, but the details were significant.

180TB is not an issue, 15,825,579 files is not an issue - BUT

If 160TB is one file, and the remaining 20TB was in 15,825,578 files that were all in one directory - you would experience this type performance.

I know my isilon support had some tools to scan directory structures looking for anomalies - it could be a single directory with tonnes of files in it...

 

NetBackup 8.3.0.1 on Solaris 11, writing to Data Domain 9800 6.2.1.0
duplicating via SLP to LTO5 in SL8500 via ACSLS

Alexis_Jeldrez
Level 6
Partner    VIP    Accredited Certified

@o1crimson1o

You did the tar with the same backup selection as you choose in NetBackup?

Makes no sense that you get different results when using Linux's tar and NetBackup's tar.