shadow copy components backup failing in a SQL 2008 server

[DPO]

Whie trying to take the full filesystem (OS level )backup of a Windows SQL server using NetBackup 7.1.0.4, it takes quite time to start and runs for a while and after that we are getting "Network timeout" error. But  DB backup  of this server using NetBackup is/was working fine. After some research, we suspected that there could be an issue with SQL Server VSS writer and so enabled TCP/IP under "Protocols for MSSQL Server" which is under SQL server network configuration of SQL server configuration manager. Since then backup is working fine. It is apparent that after enabling TCP/IP, SQL server VSS writer has allowed taking backup of its files at OS level. The backup was successfull previously prior to these changes as well. We just faced backup failure for a week .

But the question here is, how the backup was successful previously before these changes for some days?

[Marianne]

Please tell us more about the "Network timeout" error?

Is this a Connect Timeout error or Read Timeout error?

Please show us all text in Job Details for failed backup.

I cannot comment on the changes that you have made - I will only try and troubleshoot current failure.
I cannot see that SQL TCP/IP settings will affect OS level backups.
 

[DPO]

status code 41 is the error. We have increased the client read timeout prior to make changes in TCP/IP. Disabliing SQL TCP/IP causing the backups to fail now. If I stop SQL server vss writer service, backup is fine but we don't want to stop this service.

[Marianne]

Please show us all text in Job Details for failed backup.

Also check output of the following on SQL client:

vssadmin list writers

vssadmin list providers

[DPO]

Below is the part of o/p from bpbkar log. After seeing this error, we enabled TCP/IP settings and backup went fine.

tar_backup::backup_done_state: INF - number of file directives not found: 0
7:41:38.222 AM: [1332.6540] <4> tar_backup::backup_done_state: INF -     number of file
 directives found: 10
7:41:38.222 AM: [1332.6076] <4> tar_base::keepaliveThread: INF - keepalive thread terminating (reason: WAIT_OBJECT_0)
7:41:38.222 AM: [1332.6540]
 <4> tar_base::stopKeepaliveThread: INF - keepalive thread has exited. (reason: WAIT_OBJECT_0)
8:21:47.732 AM: [1332.6540] <2> tar_base::V_vTarMsgW: INF -
EXIT STATUS 0: the requested operation was successfully completed
8:21:47.732 AM: [1332.6540] <16> dtcp_write: TCP - failure: send socket (612) (TCP 10054:
Connection reset by peer)
8:21:47.732 AM: [1332.6540] <16> dtcp_write: TCP - failure: attempted to send 72 bytes
8:21:47.732 AM: [1332.6540] <16> dtcp_read: TCP -
 failure: attempted to send 72 bytes
8:21:47.732 AM: [1332.6540] <16> dtcp_read: TCP - failure: recv socket (672) (TCP 10054: Connection reset by peer)
8:21:47.732 AM:
[1332.6540] <16> dtcp_write: TCP - failure: send socket (612) (TCP 10054: Connection reset by peer)
8:21:47.732 AM: [1332.6540] <16> dtcp_write: TCP - failure:
attempted to send 39 bytes
8:21:47.732 AM: [1332.6540] <4> dos_backup::tfs_reset: INF - Snapshot deletion start
8:21:47.732 AM: [1332.6540] <4> OVStopCmd: INF -
EXIT - status = 0
8:21:47.732 AM: [1332.6540] <2> tar_base::V_Close: closing...
8:21:47.732 AM: [1332.6540] <4> dos_backup::tfs_reset: INF - Snapshot deletion
start
8:21:47.735 AM: [1332.6540] <2> ov_log::V_GlobalLog: INF - BEDS_Term(): enter - InitFlags:0x00000001
8:21:47.743 AM: [1332.6540] <4> OVShutdown: INF -
Finished process
8:21:47.745 AM: [1332.6540] <4> WinMain: INF - Exiting C:\Program Files\Veritas\NetBackup\bin\bpbkar32.exe
8:21:49.742 AM: [1332.6540] <4>
 ov_log::OVClose: INF - Closing log file: C:\Program Files\Veritas\NetBackup\logs\BPBKAR\091012.LOG
 

Earlier we were not able to see SQL writer. But after enabling TCP/IP we can see the writer. Below is the o/p of command before TCP/IP changes.

The client is a windows 2008 R2 standard server and NetBackup 7.1 is installed on it.
C:\>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Waiting for responses.
These may be delayed if a shadow copy is being prepared.

Writer name: 'Task Scheduler Writer'
Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
State: [1] Stable
Last error: No error

Writer name: 'VSS Metadata Store Writer'
Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
State: [1] Stable
Last error: No error

Writer name: 'Performance Counters Writer'
Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
State: [1] Stable
Last error: No error

Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {36862aef-abdc-4847-9a42-7e430c25601d}
State: [1] Stable
Last error: No error

Writer name: 'ASR Writer'
Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
Writer Instance Id: {9c30222b-e017-4882-9265-9912a66c4f04}
State: [1] Stable
Last error: No error

Writer name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {710fe3f0-ef6c-4801-904d-411aa87609ff}
State: [5] Waiting for completion
Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
Writer Instance Id: {304a5380-1a7c-4d42-989d-063fad3c92b6}
State: [1] Stable
Last error: No error

Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {091b3b05-6c1c-41e7-a4a0-09844d7bdf95}
State: [1] Stable
Last error: No error

Writer name: 'IIS Config Writer'
Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
Writer Instance Id: {b16e365f-5d3d-4894-89bc-7a44fa890f63}
State: [1] Stable
Last error: No error

Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {49d1c398-a084-4807-af40-fbfab5736e74}
State: [1] Stable
Last error: No error

C:\>vssadmin list providers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
Provider type: System
Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
Version: 1.0.0.7

[Mouse]

Do you have any idea why the WMI driver is "Waiting for completion" ? It should be in "Stable" state when no VSS backups being performed.

Maybe NBU waits for VSS to create a snapshot and fails after VSS fails to deliver a snapshot? Are there any relevant messages in the System log?

[badri]

see if you can take backup through windows backup tool. Was any patch implemented recently.

I would recomend to have server rebooted once.