Job failed with error 69. log: 33 minute timeout waiting on VSS snapshot

[Yolkin]

Hi,

We have several clients, each of them has several folders on logical volumes to be backed up. Every folder is assigned to a different stream so that many streams run at the same time. For the client I am looking at now - 13 streams. Every day 2-3 streams fail with error 69. I was told by Symantec support that it happens because VSS is busy with writing snapshot for another folder of the same drive. When NetBackup is requesting for a snapshot to backup - it fails as no snapshot has been done yet. Here is part of the log:

---------------------------

10:33:07.314 AM: [3668.10092] <2> tar_backup_tfi::setupFileDirectives: TAR - backup filename = Shadow Copy Components:\User Data\Distributed File System Replication\DfsrReplicatedFolders\ROBU - E
10:33:33.038 AM: [3668.10092] <2> ov_log::V_GlobalLog: INF - Using VSS to generate a new SnapshotSetID
10:33:33.053 AM: [3668.10092] <2> ov_log::V_GlobalLog: INF - VssSnapshotVolume- a previous VSS snap is in progress. Entering retry/timeout waiting for snap to complete. Timeout period: 0x7d0
.
. < 33 minute timeout waiting on VSS snapshot>
.
11:07:03.992 AM: [3668.10092] <2> ov_log::V_GlobalLog: INF - StartSnapshotSet failed with error: [0x80042316][VSS_E_SNAPSHOT_SET_IN_PROGRESS]
11:07:03.992 AM: [3668.10092] <2> ov_log::V_GlobalLog: INF -  EXIT VssSnapshotVolume::Initialize() return=[0x80042316 VSS_E_SNAPSHOT_SET_IN_PROGRESS]!
11:07:04.164 AM: [3668.10092] <2> tar_base::V_vTarMsgW: INF - EXIT STATUS 69: invalid filelist specification

---------------------------

My question is whether it is possible to extend this 33 min timeout of waiting for the snapshot.

Temporary solution I test atm is to limit number of multiple streams to 3 at the same time, but it does not always work without error 69. To get rid of it with this implementation I will have to reorder streams carefully to avoid placing folders from the same drive one after another. That might work, but I am not sure that backup will go within 24 hours. In addition, this is a very fragile set up and might break any time data raises or folders added or whatever.

Thank you

[sdo · Accepted solution]

Some sites with very large servers will allocate/present additional storage to servers sepcifically for the sole purpose of acting as VSS delta change snapshot space.  i.e. the additional storage/LUNs/volumes can only, and will only, ever be used by VSS - so that VSS is able to respond faster because it has dedicated volumes, and does not need to wait for a brief moment to quiesce (i.e. flush/hold/snap/release).  Remember, whilst VSS is active, it is 'intercepting all writes, but the file system driver needs to interject itself - and so can't if too many snapshots are taking place.  So, like I said some sites have a 1-to-1 relationship between volumes with data and additional volumes for VSS space.

With VSS you can configure specific volumes to act as snapshot space for other specific volumes.

It would seem that your issue is a Windows server configuration issue, and not really a NetBackup issue.  NetBackup leverages VSS, and so an inappropriate VSS configuration will impair the effectiveness of NetBackup Client, and indeed any backup product.

You've already done what you can from the NetBackup side of things - so, you may need to push back to your Windows admins and/or management team and seek approval for additional storage to act purely as VSS space.

[Marianne]

I agree - the amount of simultaneous streams seems to be an issue for VSS.

Other than trying to fix VSS, you can cut down on the amount of streams.

Start with 4 streams.
If all is fine, try 6.
Then 8... and so on.

When it breaks, you know what the Windows server's limit is.
Or else try to backup without snapshots...

[sdo]

Exactly what version of NetBackup Client is running on the client?

If it's NetBackup Client v7.0.1 or earlier, then maybe you are experiencing this:

https://support.symantec.com/en_US/article.ALERT1049.html

[sdo]

How many shadows already actually exist?  In total, and per volume?

Some maximums are described are:

https://msdn.microsoft.com/en-us/library/office/aa579091(v=exchg.140).aspx

[Yolkin]

Thank you very much for fast response and useful suggestion.

sdo: I was adviced to create separate logical volumes for each folder to make VSS write snapshots separately, but Windows team said its not possible due to huge amount of data and many dependencies. I am wondering now if it is possible to point different folder on the the logical volume to make shadow copies on separate logical volumes (we would create more for that).

NBU version is 7.5.0.7

By asking of number of copies - do you mean how many of them exist for one image, one client or during one backup? If the last one - there are 13 streams with 1 copy for each.

Marianne: thank you, I was told that you are one of the most active and professional technicians within questions forums. I limited streams per client to 2, but that did not make any difference unfortunately. Same errors 69 and some 196 error in addition in case of full backups.

I am still wondering if it is possible to increase the time which NBU is waiting for snapshot to be taken (in my case from logs - 33 minutes). Would you have any idea?

Thank you

BR

Artjom

[Varunthilak_B]

If you have multiple clients in single policy, I would suggest you to split the policy for single client and then run/test.

Make sure, all the folder paths are exists on client. Are you using All_Local_Drive or New_Stream method?

Also set the VSS cache size to NO LIMIT on client and then re-try backup.

Why don't you use Flash-Windows for particular drive ?