Virtual machine backup fails with error 130

[jaydoubleyou]

The backup was working fine until two weeks ago when one of the snapshots on the vSphere failed with the error:
"An error occurred while saving the snapshot: msg.snaps-hot.error-QUIESCINGERROR". I researched a solution for this error and found a potential fix on the VMWare site, however this did not resolve the issue. All subsequent snapshots on the vSphere have completed fine and I can also create a snapshot in vSphere with no problems, however all backups on Netbackup have failed, initially with error 6, lately with error 130.

The backup fails with the following errors in the Netbackup console logs:
06/01/2016 21:23:44 - Critical bpbrm (pid=227410) from client [clientname]: FTL - terminated by signal 6 
06/01/2016 21:23:44 - Warning bpbrm (pid=227410) from client [clientname]: WRN - ALL_LOCAL_DRIVES is not frozen 

06/01/2016 21:23:45 - Critical bpbrm (pid=227537) from client [clientname]: FTL - cannot open /usr/openv/netbackup/online_util/fi_cntl/bpfis.fim.[clientname]_1464812615.1.0 
06/01/2016 21:23:45 - Info bpfis (pid=227547) done. status: 4207: Could not fetch snapshot metadata or state files 

I have tried removing and readding the client from the policy, and have enabled logging on the client - there is a line in bpcd saying "process_requests: Server access denied" but I'm not sure where or how to enable access.

Master/media server is Linux (SUSE Linux Enterprise Server 11), client is Windows (Server 2012) 

[jaydoubleyou · Accepted solution]

This is now sorted, I ended up logging it to Veritas who suggested trying NBD transport mode rather than SAN transport mode as a workaround. This was fine but to get it working with SAN transport mode I enabled verbose logging and sent off the vxms, bpfis and bpbkar logs. The latter included the line "Cannot create directory /tmp/vmware-root/..." and the explanation I was given was: "sometimes bpbkar hangs or crashes and it doesn't clear up a directory it uses to cache metadata. Then when the next backup comes in, it tries to write to that directory (specific for that VM) and it is not allowed because another process has not yet cleared it up". So, I made sure there were no backups running, cleared the contents of /tmp/vmware-root on the backup host and rerun the backup with SAN mode enabled, which has worked ever since (approx 2 weeks now).

[jaydoubleyou]

(Duplicate comment)

[jaydoubleyou]

(Duplicate comment)

[jaydoubleyou]

(Duplicate comment)