ZFS BACKUP SLOW THROUGHPUT

Question

Hi.I have a Netbackup 7.7.3 environment with a Solaris 11 SPARC Master Server which backups an Oracle ZFS storage, besides another applications.The ZFS shares are mapped to the Master Server by NFS, so this shares are seen as FS. The information of these ZFS shares goes to LTO5 tapes. I use 2 LTO5 tape drives for this backup operation.The communication from the ZFS to the Master Server goes through Infiniband.Two months ago, the throughput of the backups have been too slow. Before that, I registered throughtputs of 180 Mb/sec. That was the average throughput. Now, the throughput is 50 Mb/sec max.I haven't done any changes on the backup configuration.Could you help me with some tips to improve the throughput?

cesarjimgon · Accepted Answer

Thanks Marianne.I checked the NFS configuration in the Master Server&nbsp;and compared it to another Master Server with the same ZFS backup solution. I found there were missing options:root@uxvtbck1 # more /vfstab/vfstab: No such file or directoryroot@uxvtbck1 # more vfstab#device&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; device&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; mount&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; FS&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; fsck&nbsp;&nbsp;&nbsp; mount&nbsp;&nbsp; mount#to mount&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; to fsck&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; point&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; type&nbsp;&nbsp;&nbsp; pass&nbsp;&nbsp;&nbsp; at boot options#/devices&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /devices&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; devfs&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; no&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -/proc&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /proc&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; proc&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; no&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -ctfs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /system/contract ctfs&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; no&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -objfs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /system/object&nbsp; objfs&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; no&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -sharefs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /etc/dfs/sharetab&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; sharefs -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; no&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -fd&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /dev/fd&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; fd&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; no&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -swap&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /tmp&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; tmpfs&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; yes&nbsp;&nbsp;&nbsp;&nbsp; -/dev/zvol/dsk/rpool/swap&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; swap&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; no&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -192.168.10.50:/export/respaldo1 -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /respaldo1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; nfs&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; yes&nbsp;&nbsp;&nbsp;&nbsp; rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio192.168.10.51:/export/respaldo2 -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /respaldo2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; nfs&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; yes&nbsp;&nbsp;&nbsp;&nbsp; rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio192.168.10.50:/export/respaldo3 -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /respaldo3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; nfs&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; yes&nbsp;&nbsp;&nbsp;&nbsp; rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio192.168.10.51:/export/respaldo4 -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /respaldo4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; nfs&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; yes&nbsp;&nbsp;&nbsp;&nbsp; rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio192.168.10.50:/export/respaldo5 -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /respaldo5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; nfs&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; yes&nbsp;&nbsp;&nbsp;&nbsp; rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio192.168.10.51:/export/respaldo6 -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /respaldo6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; nfs&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; yes&nbsp;&nbsp;&nbsp;&nbsp; rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio192.168.10.50:/export/respaldo7 -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /respaldo7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; nfs&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; yes&nbsp;&nbsp;&nbsp;&nbsp; rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio192.168.10.51:/export/respaldo8 -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /respaldo8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; nfs&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; yes&nbsp;&nbsp;&nbsp;&nbsp; rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectioThe missing options were proto=tcp,vers=3,forcedirectioThese options were added to the /etc/vfstab file of the Master Server. Before that, the ZFS&nbsp;mount points were unmounted. After the configuration these mount points were mounted again.I'm running a full backup right now. The throughput is&nbsp;138 Mb/sec. The backups run faster.

marianne · Answer

What is the size of the siebel directory? Maybe check the size of all subdirectories (du -sk /zfsbackup6/*) and test with a bit more data?

marianne · Answer

CesarJimGon
Have you tried to test read speed outside of NBU?
Something like :
time tar cBf&nbsp; /dev/null &lt;/mountpoint&gt;Or instead of mountpoint, a subdirectory that is about 5 - 10GB.Get the size before you start in order to work out read speed.&nbsp;

cesarjimgon · Answer

Hi Marianne.I'm testing your suggestion, but I don't think I'm doing it right.Checking size of test directorydu -as /usr/openv/netbackup35739031&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /usr/openv/netbackupRead testtime tar cBf /usr/openv/netbackuptar: Missing filenamesreal&nbsp;&nbsp;&nbsp; 0m0.020suser&nbsp;&nbsp;&nbsp; 0m0.003ssys&nbsp;&nbsp;&nbsp;&nbsp; 0m0.008s

marianne · Answer

You missed /dev/null. You need to specify /dev/null as backup device, meaning you are simply reading the files and not really writing to anything. Please choose a folder on the NFS mount.

cesarjimgon · Answer

&nbsp;Hi MarianneI send the outputcd /zfsbackup6zfsbackup6# ls -ltotal 5540drwxr-xr-x 7 nobody nobody 13368 Dec 6 2014 BKP_TRCdrwxr-xr-x 14 1001 1001 14 Feb 16 2016 bkupdrwxr-xr-x 3 1000 1001 4 May 27 2015 listeners-rw------- 1 nobody nobody 53687091200 Nov 19 2014 loaddrwxr-xr-x 3 1001 1001 6 Aug 31 2015 Patchesdrwxr-xr-x 2 1001 1001 2 Sep 29 2015 siebeldrwxr-xr-x 4 1000 1001 4 May 29 2015 Traces/zfsbackup6# time tar cBf /dev/null/zfsbackup6/siebeltar: Missing filenamesreal 0m0.011suser 0m0.003ssys 0m0.008sdf -kFilesystem 1024-blocks Used Available Capacity Mounted onrpool/ROOT/solaris 286949376 55595990 29243717 66% //devices 0 0 0 0% /devices/dev 0 0 0 0% /devctfs 0 0 0 0% /system/contractproc 0 0 0 0% /procmnttab 0 0 0 0% /etc/mnttabswap 32643296 2832 32640464 1% /system/volatileobjfs 0 0 0 0% /system/objectsharefs 0 0 0 0% /etc/dfs/sharetabfd 0 0 0 0% /dev/fdrpool/ROOT/solaris/var286949376 592731 29243717 2% /varswap 32824512 184048 32640464 1% /tmprpool/VARSHARE 286949376 2263739 29243717 8% /var/sharerpool/export 286949376 3496 29243717 1% /exportrpool/export/home 286949376 32 29243717 1% /export/homerpool/export/home/admin286949376 389 29243717 1% /export/home/adminrpool/repo 286949376 11558600 29243717 29% /reporpool 286949376 73 29243717 1% /rpoolrpool/VARSHARE/zones 286949376 31 29243717 1% /system/zonesrpool/VARSHARE/pkg 286949376 32 29243717 1% /var/share/pkgrpool/VARSHARE/pkg/repositories286949376 31 29243717 1% /var/share/pkg/repositories192.168.10.99:/export/zfsbackup68933536765 1934208810 6999327955 22% /zfsbackup6

Forum Discussion

ZFS BACKUP SLOW THROUGHPUT

12 Replies

Related Content

Backup-to-disk-to-tape slow

Slow Network Throughput

Oracle backup channels throughput

Backup Exec 15 Performance Slow

BE Slow Throughput to HP Disk Library...

Recent Discussions

MS-SharePoint policy restore error (2804) .

How to restore a backup

How to configure RBAC

command: bperror

10 years old netbackup appliance database service down, ssl certification out date