Issue with flashbackup

[prats]

Hi,

Netbackup 6.5.6

We have recently started using flashbackup for one of our clients.The flashbackup for most of the servers is working fine, but there are some EV servers for which the flashbackup is not working.

The backups are failing with error 11: system call failed, VFM error = 2059.The servers are W2k8 servers and the snapshot method is set to auto.We are trying to backup the data on mount points. The backup fails only for some mount points.

Thanks in advance for all the help.

Regards

Prats.

[pikachu]

You need to check from the client:

• bpfis
• bpbkar
• vxms
• maybe bpbrm from the media server

Also how big is the volume?

What is the fragmentation on the drive?

What is the excat OS of the client?

[Yogesh9881]

what is backup selection in policy ?

Because Flashbackup will not supported for clustered volumes.

[prats]

Its not a cluster.

Selection List has :

\\.\M:\EV\EV_Vaults where EV_vaults is the mount point.

What to look for in the logs?

[AAlmroth]

I've noticed similar conditions where the change rate was too high on the volume so that VSS snapshot could not keep up. You will need to check the events on the client for any VSS errors.

Other cases were when the file system was not NTFS or corrupted NTFS.

Also, as you mention EV; are you sure the mount points are local attached disk, and not some NAS attached storage? On quite many occasions I've seen the vault stores being stored on NAS filers. I know that mounted CIFS shares is not recommended or evensupported, and EV should use the full CIFS path instead...

/A

[Marianne]

What to look for in the logs?

Look for <16>'s or <32>'s, then scroll up a couple of lines and read the <2>'s with the same process id (in square brackets [.....] ) for indication of what caused the problem.

[Mark_Solutions]

What does it say in the event logs on the client?

There should be a VSS error in there that may give a clue to what is needed (disk space, memory, VSS Hot Fix?)

[Mark_Solutions]

I had remembered something so just checked and the following appears in the 6.5.6 Release Notes:

About VSS and hot to manage snapshot volume size
In the NetBackup 7.0GA, 6.5.4 and 6.5.5 releases, NetBackup attempted to
manage the size of the VSS snapshot volume. To do that, NetBackup changed
any unbounded VSS associations to a fixed size. That size was based on the
size of the disk drive. The objective was to prevent the unbounded VSS
associations that can fill a hard drive and affect the system operation. In
releases after 7.0GA and 6.5.5 that is no longer be done and that system
management setting is beyond the scope of NetBackup to alter.
A customer who has upgraded from one of these levels might have failed
backups because NetBackup has changed this setting. For that to occur all of
the following must be true:
■ Customer has upgraded from 6.5.4, 6.5.5 or 7.0GA.
■ An unlimited VSS association on a client
■ NetBackup backed up that client (and therefore modified the setting)
■ The client OS is Windows 2008 or 2008 R2 Server
■ The backups are either multistreamed, or the backups create the VSS
snapshots that overlap in time.
In this case the association is set too small and future snapshots might be
deleted before they are backed up. To correct this issue, set the association to
a larger, more appropriate value. Refer to Microsoft documentation for details
on how to set this system management setting.

[prats]

no VSS errors found in the event logs.The mount points are on SAN.File system is NTFS.

Found the following in the bpbkar log on the client

 [16004.4460] <2> tar_base::V_vTarMsgW: INF - VxMS error - severity 3.

: [16004.4460] <2> tar_base::V_vTarMsgW: INF - VxMS Error message 1 = xm_start_index_traversal: vfm_open_id <ntfs>\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy27 failed error 2059

: [16004.4460] <2> tar_base::V_vTarMsgW: INF - FIML startup time = 0

: [16004.4460] <2> tar_base::V_vTarMsgW: ERR - FIML_start on <ntfs>\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy27 failed VFM error = 2059

: [16004.4460] <4> tar_backup_tfi::flash_start_state: Error starting flash, flash Error = 11, returning TASK_PRIORITY

: [16004.4460] <4> tar_backup::backup_done_state: INF - number of file directives not found: 0

 [16004.4460] <4> tar_backup::backup_done_state: INF - number of file directives found: 1

: [16004.4512] <4> tar_base::keepaliveThread: INF - keepalive thread terminating (reason: WAIT_OBJECT_0)

: [16004.4460] <4> tar_base::stopKeepaliveThread: INF - keepalive thread has exited. (reason: WAIT_OBJECT_0)

: [16004.4460] <2> tar_base::V_vTarMsgW: INF - EXIT STATUS 11: system call failed

: [16004.4460] <4> tar_backup::backup_done_state: INF - Not waiting for server status

: [16004.4460] <4> dos_backup::tfs_reset: INF - Snapshot deletion start

: [16004.4460] <4> OVStopCmd: INF - EXIT - status = 0

: [16004.4460] <2> tar_base::V_Close: closing...

: [16004.4460] <4> dos_backup::tfs_reset: INF - Snapshot deletion start

: [16004.4460] <32> TransporterRemote::write[2](): FTL - SocketWriteException: send() call failed, could not write data to the socket, possible broken connection.

: [16004.4460] <32> Packer::close(): FTL - OperationFailedException: close operation failed.

: [16004.4460] <2> ov_log::V_GlobalLog: INF - BEDS_Term(): enter - InitFlags:0x00000101

: [16004.4460] <2> ov_log::V_GlobalLog: INF - BEDS_Term(): ubs specifics: 0x001d0000

: [16004.4460] <16> dtcp_read: TCP - failure: recv socket (324) (TCP 10053: Software caused connection abort)

: [16004.4460] <16> dtcp_read: TCP - failure: recv socket (324) (TCP 10053: Software caused connection abort)

: [16004.4460] <16> dtcp_read: TCP - failure: recv socket (324) (TCP 10053: Software caused connection abort)

: [16004.4460] <16> dtcp_read: TCP - failure: recv socket (324) (TCP 10053: Software caused connection abort)

: [16004.4460] <4> OVShutdown: INF - Finished process

: [16004.4460] <4> WinMain: INF - Exiting C:\Program Files\Veritas\NetBackup\bin\bpbkar32.exe

[pikachu]

It looks like the issue is with VxMS. As I said... get the vxms logs.

Also a simple chkds+defrag might help this situation before you jump into the big vxms log.

[prats]

Hi,

Thanks for the reply.How to generate the vxms logs?

Could you also explain which "software" is it referring to in the above lines?

[pikachu]

http://www.symantec.com/docs/TECH72862

set the logging to 7580

[Michael_G_Ander]

Would try to set the snapshot method to VSS, to check if it is caused by Netbackup choosing the wrong method for some reason.