netbackup 156 error

[nbustarter380]

Hi All,

We have a windows client that gets 156 errors, the Netbackup client is  on 7.6.0.1

I have attached the bpcd, bpbkar and bpfis logs.

VSS  open file backups are activated

C:\>vssadmin list providers

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2005 Microsoft Corp.

Provider name: 'Microsoft Software Shadow Copy provider 1.0'

   Provider type: System

   Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}

   Version: 1.0.0.7

C:\>vssadmin list writers

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2005 Microsoft Corp.

Writer name: 'Task Scheduler Writer'

   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}

   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}

   State: [1] Stable

   Last error: No error

Writer name: 'VSS Metadata Store Writer'

   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}

   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}

   State: [1] Stable

   Last error: No error

Writer name: 'Performance Counters Writer'

   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}

   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}

   State: [1] Stable

   Last error: No error

Writer name: 'System Writer'

   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}

   Writer Instance Id: {bd2e4514-0efa-48fc-9bb3-1ca77e956641}

   State: [1] Stable

   Last error: No error

Writer name: 'ASR Writer'

   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}

   Writer Instance Id: {50f27659-1973-4535-89c5-1ba55af3ba0a}

   State: [1] Stable

   Last error: No error

Writer name: 'IIS Config Writer'

   Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}

   Writer Instance Id: {674b1230-d858-4179-9e98-5ff475bca218}

   State: [1] Stable

   Last error: No error

Writer name: 'Shadow Copy Optimization Writer'

   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}

   Writer Instance Id: {7d424843-6740-458b-9b2d-c7e721038c4b}

   State: [1] Stable

   Last error: No error

Writer name: 'Registry Writer'

   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}

   Writer Instance Id: {5fd4ce8d-8e6b-40f4-b401-4dd83028bebb}

   State: [1] Stable

   Last error: No error

Writer name: 'WMI Writer'

   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}

   Writer Instance Id: {db42c221-ebd2-4ee5-a3f4-b6ee95295b03}

   State: [1] Stable

   Last error: No error

Writer name: 'COM+ REGDB Writer'

   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}

   Writer Instance Id: {0dd71c68-187c-4420-aabb-286897273c17}

   State: [1] Stable

   Last error: No error

Writer name: 'BITS Writer'

   Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}

   Writer Instance Id: {398d0460-e3ab-4e20-9dc3-0f32373ffce9}

   State: [1] Stable

   Last error: No error

Writer name: 'IIS Metabase Writer'

   Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}

   Writer Instance Id: {f1c76bc1-37b2-497c-825f-06306d865ea5}

   State: [1] Stable

   Last error: No error

[root@nbusoc4# bptestbpcd -client pohansay14.mycompany.com -verbose -debug

13:08:19.992 [1446] <2> bptestbpcd: VERBOSE = 0

13:08:19.992 [1446] <2> ConnectionCache::connectAndCache: Acquiring new connection for host nbusec1, query type 223

13:08:19.993 [1446] <2> vnet_pbxConnect: pbxConnectEx Succeeded

13:08:19.994 [1446] <2> logconnections: BPDBM CONNECT FROM 472.41.91.40.45706 TO 472.41.91.40.1556 fd = 3

13:08:19.994 [1446] <8> vnet_get_user_credential_path: [vnet_vxss.c:1471] status 35 0x23

13:08:19.994 [1446] <8> vnet_get_credential_path: [vnet_vxss.c:1098] vnet_get_user_credential_path failed 35 0x23

13:08:19.994 [1446] <8> vnet_get_oc: [vnet_vxss_helper.c:3167] Invalid Argument: vxss_coninfo

13:08:19.996 [1446] <2> db_CLIENTsend: reset client protocol version from 0 to 8

13:08:20.039 [1446] <2> db_end: Need to collect reply

13:08:20.045 [1446] <2> db_freeEXDB_INFO: ?

13:08:20.058 [1446] <2> vnet_pbxConnect: pbxConnectEx Succeeded

13:08:20.058 [1446] <2> logconnections: BPCD CONNECT FROM 472.41.91.40.28085 TO 472.41.80.90.1556 fd = 3

13:08:20.061 [1446] <2> vnet_pbxConnect: pbxConnectEx Succeeded

13:08:20.116 [1446] <8> do_pbx_service: [vnet_connect.c:2152] via PBX VNETD CONNECT FROM 472.41.91.40.62486 TO 472.41.80.90.1556 fd = 4

13:08:20.116 [1446] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:443] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa

13:08:20.330 [1446] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:460] ipc_string 54142

13:08:20.668 [1446] <8> vnet_get_user_credential_path: [vnet_vxss.c:1471] status 35 0x23

13:08:20.668 [1446] <8> vnet_get_credential_path: [vnet_vxss.c:1098] vnet_get_user_credential_path failed 35 0x23

13:08:20.695 [1446] <2> bpcr_get_version_rqst: bpcd version: 07600000

1 1 1

472.41.91.40:28085 -> 472.41.80.90:1556

472.41.91.40:62486 -> 472.41.80.90:1556

13:08:20.696 [1446] <2> bpcr_get_peername_rqst: Server peername length = 22

13:08:20.697 [1446] <2> bpcr_get_hostname_rqst: Server hostname length = 25

13:08:20.698 [1446] <2> bpcr_get_clientname_rqst: Server clientname length = 25

13:08:20.698 [1446] <2> bpcr_get_version_rqst: bpcd version: 07600000

13:08:20.698 [1446] <2> bpcr_get_platform_rqst: Server platform length = 7

13:08:20.714 [1446] <2> bpcr_get_version_rqst: bpcd version: 07600000

13:08:20.756 [1446] <2> bpcr_patch_version_rqst: theRest == > <

13:08:20.756 [1446] <2> bpcr_get_version_rqst: bpcd version: 07600000

13:08:20.815 [1446] <2> bpcr_patch_version_rqst: theRest == > <

13:08:20.815 [1446] <2> bpcr_get_version_rqst: bpcd version: 07600000

PEER_NAME = nbusoc4.mycompany.com

HOST_NAME = pohansay14.mycompany.com

CLIENT_NAME = pohansay14.mycompany.com

VERSION = 0x07600000

PLATFORM = win_x64

PATCH_VERSION = 7.6.0.1

SERVER_PATCH_VERSION = 7.6.0.1

MASTER_SERVER = nbusoc4.mycompany.com

EMM_SERVER = nbusoc4.mycompany.com

NB_MACHINE_TYPE = CLIENT

13:08:20.849 [1446] <2> vnet_pbxConnect: pbxConnectEx Succeeded

13:08:20.885 [1446] <8> do_pbx_service: [vnet_connect.c:2152] via PBX VNETD CONNECT FROM 472.41.91.40.57376 TO 472.41.80.90.1556 fd = 5

13:08:20.885 [1446] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:443] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa

13:08:21.082 [1446] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:460] ipc_string 54145

472.41.91.40:57376 -> 472.41.80.90:1556

<2>bptestbpcd: EXIT status = 0

13:08:21.158 [1446] <2> bptestbpcd: EXIT status = 0

more information

windows 2008 64 bit  netbackup client 7.6.0.1

Master server  netbackup 7.6.0.1

Thanks for any help in advance!

[Marianne]

Please tell us more about the 2 bptestbpcd commands?
Same command with one failing and the other successful? 

bptestbpcd -client -pohansay14.mycompany.com 
(In this command output there is also reference to client name cohansey20 ?! )
13:07:44.553 [1367] <2> bptestbpcd: EXIT status = 48 
client hostname could not be found

bptestbpcd -client pohansay14.mycompany.com 
13:08:21.158 [1446] <2> bptestbpcd: EXIT status = 0

Were the 2 commands done from different servers?
Maybe master and a media server?

Seems 2nd command was done from 'nbusec1' and we see in this output that the master name is 'nbusoc4', right?
But the source IP addresses for both commands are the same?
472.41.91.40
472.41.91.40
 

About the log files - they are quite big and difficult to read, since .doc files need to be downloaded first and opened with MS Word  - .txt files are easier to read since they can be opened online.

I have had a look at bpbkar file and see incremental backup attempt with way too many entries such as these:

detected renamed/new directory: ....... forcing full backup

When last did a full backup run for this policy? 
Please do not attempt Incr backups unless you have a successful Full backup.

What is in Backup Selection for this policy?
If ALL_LOCAL_DRIVES, please select 'Allow Multiple Data Streams' in policy attributes so that SCC and drive letters will be backed up in separate streams.
Separate streams will make it easier to pinpoint where exactly the snapshot problem is.

You can in the meantime test snapshots of each drive individually using this TN:  http://www.symantec.com/docs/TECH47808

[Mark_Solutions]

What i do see in the logs is this:

00:05:59.135 [18812.16712] <16> dtcp_read: TCP - failure: recv socket (580) (TCP 10053: Software caused connection abort)

Interesting though that the new files are the NBU installation:

detected renamed/new file:<E:\Program Files\Veritas\NetBackup\bin\AgentUtil.exe>, forcing backup

Has the client been re-installed - perhaps something has gone wrong during that?

[rk1074]

Whats is the size of the free psace on the shadow storage devices. min 10 % is required for the successfull snapshot operaion.

[nbustarter380]

rk1074 Thanks,

I believe this is what you are referring to?  1st time I have issued that command.

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Windows\system32>vssadmin list shadowstorage
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Shadow Copy Storage association
   For volume: (E:)\\?\Volume{b45cf681-ed43-11e1-931d-0024e86ef979}\
   Shadow Copy Storage volume: (E:)\\?\Volume{b45cf681-ed43-11e1-931d-0024e86ef9
79}\
   Used Shadow Copy Storage space: 0 B (0%)
   Allocated Shadow Copy Storage space: 0 B (0%)
   Maximum Shadow Copy Storage space: 320 MB (0%)

Shadow Copy Storage association
   For volume: (\\?\Volume{b45cf685-ed43-11e1-931d-0024e86ef979}\)\\?\Volume{b45
cf685-ed43-11e1-931d-0024e86ef979}\
   Shadow Copy Storage volume: (E:)\\?\Volume{b45cf681-ed43-11e1-931d-0024e86ef9
79}\
   Used Shadow Copy Storage space: 47.688 MB (0%)
   Allocated Shadow Copy Storage space: 2.976 GB (0%)
   Maximum Shadow Copy Storage space: UNBOUNDED (100%)

Shadow Copy Storage association
   For volume: (C:)\\?\Volume{56812b8e-ed36-11e1-b68c-806e6f6e6963}\
   Shadow Copy Storage volume: (C:)\\?\Volume{56812b8e-ed36-11e1-b68c-806e6f6e69
63}\
   Used Shadow Copy Storage space: 0 B (0%)
   Allocated Shadow Copy Storage space: 0 B (0%)
   Maximum Shadow Copy Storage space: 320 MB (0%)

C:\Windows\system32>

Regards

[nbustarter380]

Hi Marianne and Mark Thanks for the Reponses

Marianne,

Sorry, for the confusion  Yes, nbusoc4 is the master (edited the first thread)

The last full backup was 4 days ago it was a rerun because the full backup  received the 156 errors  the incremental are getting the 156 errors also.

Backup Policy is  ALL_LOCAL_DRIVES, and we have 'Allow Multiple Data Streams' in policy attributes selected.

I will see about testing snapshots not sure I should do that or let the Windows System Admin try it. Sometimes they can be picky about something like that. It shouldn’t  cause an issue though.

Mark

Maybe that's from when it was upgraded to the 7.6.0.1 client?  Maybe I should try a uninstall/reinstall on the client and also request a server reboot as well I guess it could not hurt.

These 156 errors are a pain sometimes it’s as simple as a client upgrade and other times it’s not.

Thanks again to both of you, I will post an update.

[nbustarter380]

Marianne, Mark and Rk1074!

update- There were patches put on the server and it was rebooted. I did a uninstall/reinstall on the Netbackup client and a full is scheduled.

I will update you on the results, Thanks!

Regards

[Mark_Solutions · Accepted solution]

Just set to manual should be fine - it will start when called and stop once its work has finished

If that is not happening for you it would be worth checking for any windows updates / service packs or hotfixes

There are a few VSS hotfixes for Windows 2003 - search for:

VSS rollup package 2003

[nbustarter380]

Hi

I tried a few things and with this and noticed that when I set the Volume Shadow Copy service to started the backup doesn’t get the 156 error.

I think I read that the Volume Shadow Copy Service should be set to Manual  Start type.  I believe this is true because it’s the default setting and most all of the other backup are working find with that setting.

Should the Volume Shadow Copy Service be set to started?

[nbustarter380]

Hi Mark,

Thanks..I know there were patches put on a week ago and the server was rebooted.  Is some way to test and check to make sure the service starts when it's called?

I will search for the VSS rollup package 2003

Regards