Point 1:
Additional details to "Netbackup Environment:" section in my first post:
NBU upgrade and Zfs
- Only on the Master Server.
- Master Server is not used as a media server.
- There are no Storage Units on the Master Server.
- ZFS is used for '/usr/openv' volume. This in any reason cannot be cause for the issue I am experiencing.
Media Servers: (Still at 6.5.5, using VxFS)
- We have two media servers (identical configuration) but lets only consider one as the other is at a DR site and wouldn't take much load.
- Media Server in the Question: Media2
- NBU Version : 6.5.5
- Staging Unit Filesystem : VxFS
So considering that I am still using the VxFS on the Staging unit (AdvanceDisk), the issue can only be caused by NBU.
Point 2:
NBU behaviour at HIGH WATER MARK:
- Existing job will continue running: TRUE in my case.
- New Jobs will not be run: Partially True in my case with a slight issue.
- At Highwater mark, image cleanup kicks in and tries to remove the staged or expired images.
- If Image cleanup clears the images and drops STU usage below Highwater mark, then it will start executing new jobs.
- If Image cleanup cannot clear images and STU usage stays above Highwater mark, then it will start failing the jobs.
Assumption: Even if the Staging unit is on ZFS, the issue I am seeing still have to be a NBU related. Because..
ZFS filesystem Issue: “Inability to commit ZFS files system capacity statistics atomically”.
(ref: http://www.symantec.com/business/support/index?page=content&id=TECH66993)
Means if we use ZFS, there will be a LITTLE EXTRA DELAY before NBU DETECTS the Highwater mark (Again depends on the frequency at which NBU makes its checks) and kicks the Image cleanup. And the Image Cleanup job follows the later sequence. But the jobs should not fail straight away.
How frequently does Netbackup monitor the STU capacity?? How quick do you need??
Below is the explanation for “Disk storage unit capacity check frequency”. Found in ‘General Server’ options in master server host properties.
______________________________________________________________
This property determines how often NetBackup checks disk storage units for available capacity. If checks occur too frequently, then system resources are wasted. If checks do not occur often enough, too much time elapses and backup jobs are delayed.
The default is 300 seconds (5 minutes).
Note: This property applies to the disk storage units of 6.0 media servers only. Subsequent releases use internal methods to monitor disk space more frequently.
______________________________________________________________
In Regards to Migrating to ZFS:
I was not trying to compare VxVM to ZFS but I compared “Disk Pooling capability” of of AdvanceDisk to ZFS and VxVM.
I only considered ZFS based upon my requirements.
Requirements in the order of importance:
- Logical Volume management (pooling multiple luns). As there is a 2 TB lun size limitation on our Storage Array. (ZFS and VxVM)
- Data integrity (zfs)
- Performance (zfs with large block, highly sequential workloads)
- Cost (zfs)
- Ease in Management (zfs)
- Scalability (VxVM has some advantages but zfs fits fine in my case)
Cons with Netbackup: (in my situation)
VxVM:None
ZFS: ZFS filesystem Issue: “Inability to commit ZFS files system capacity statistics atomically”.
(ref: http://www.symantec.com/business/support/index?page=content&id=TECH66993)
I am only effected in terms of the time DELAY between a) Actual file system size hitting ‘High water mark’ and b) Netbackup detecting that File system has hit the High water mark.
What will be the actual impact?
STU size : 22 TB
High water mark : 90% (i.e, 19.8 TB)
Room before job starting to fail : 10% (i.e, 2.2 TB)
Average Throughput : 350 MB/sec (i,e 1.2 TB per hour).
According to the above figures, I easily have a minimum of 2 hour period before the disk fills up and backup starts to fail (only if the Netbackup cannot detect that file system has reached the High Water Mark).
So considering the 2 hour limit, I don’t think I should be worrying about “delayed capacity statistics” for zfs.
I guess this issue only effects when the Storage Units are smaller and the difference between highwater mark and the disk full condition is less. Which is not the case in my situation.
