One Single Client refuses to complete backup

[w_boggs]

All of my backups are working, with the exception of one single client.  It is a virtual Solarwinds server.  Windows Server 2008R2.

Below is a summary of the errors.

Checked the status of this policy only.

Differential and Full Backups Fail

Reads from the clients , writes thousands of files and then disconnects....no actual files actually get written to the storage device

Detailed Status:

Info bpbkar32  change journal not enabled for E: Drive  (not a big deal – don’t use change journal for any client)

error bpbm(pid=7332) from client ERR: Unable to backup System State or Copy -  Please check the state of VSS and associated writers.inf  Exit Status 69

Error bptm (pid=7712) socket operation failed –10054 (at child c.1296)

Error bptm (pid=7712) unable to perform read from client socket,

When I check the properties of this client, it connects and gives full details.  It writes thousands of files during backup, but somehow never completes the job and DOES NOT partial files to the storage unit in it's proper folder.

Any help would be greatly appreciated!

Bill

[Will_Restore]

error bpbm(pid=7332) from client ERR: Unable to backup System State or Copy -  Please check the state of VSS and associated writers.inf  Exit Status 69

on the client, run as Administrator

vssadmin list writers

and review the output for clues

[Dyneshia]

Also check the application event viewer for any ESE or VSS errors.

Since this is virtural, check the bpfi on the storage server.

[Maurice_Byrd]

How are the backup selections listed?  If you have more than one path listed, try enabling multiple data streams.  If you have All Local Drives, try listing the actual drives/paths that you want to backup instead.

This would eliminate the System State backup. If you require System State then you'll have to figure out why the VSS writers are failing.

[ontherocks]

Check this TN

http://www.symantec.com/business/support/index?page=content&id=TECH164829

[w_boggs]

Interesting comments.  The VSS comments perplex me.  I was specifically told to turn off (disable) VSS on every single client when the NetBackup Rep came here to install the software.  It has been and is disabled on all 27 Client systems and ALL work perfectly every day! Except this ONE!

[CRZ]

Have you double checked that it is disabled? 

(It wouldn't hurt to check again!)

[Maurice_Byrd]

That is interesting... did he say why you needed to disable VSS?  Are you backing up a bunch of old systems or something?

At the OS level, what's the difference between the failing client and the successful clients?

[w_boggs]

@Maurice,

   There is no difference between OS...most are Server 2008R2.  Even the few 2003 Server systems back up successfully every night

@Chris

   vssadmin list writers comes back with unexpected error.... since VSS isn't enabled

The only errors I have in the Event Viewer Application Log  is VSS errors -  Volume Shadow Copy Service error:  Unexpected error calling routine CoCreateInstance.  The service can not be started, either because it is disabled or because it has no enabled devices associated with it.

So, my question is this:    

      27 Clients total  -  27 systems with VSS disabled -  Master and 2 Media Servers VSS disabled-

      26 Successful backups full and differential - 1 Failure

Don't think its a VSS issue pertaining to that specific client.  

[CRZ]

Well, it was worth a shot to ask.

Tell us some more about your environment.

Is this client in a policy with clients that work, or is it in its own policy?

Can you share the policy with us?  (If it is in its own policy, could you also share any other working policy just so we can determine if there are any differences?)

Would you be able to create a new test policy for just this client, just for SCC, and see if you get the same error - or a different one?

[w_boggs]

@CRZ

This client is just like the others.  It is a virtual environment and runs solarwinds which is active 24/7, thought that might be part of the problem. Have all of my backups scheduled in the off-hours, so there should be little activity going on with that system.  The read and network connectivity is good between the Master, 2 Media Servers and this client.  It reads several thousand files on most nights, but just disconnects for no reason and I don't get a full differential or full backup.

It is a Server 2008R2 system, just like my Server and 2 media Servers.  There is nothing different about this client.

It was in a policy with 3 other clients, so I moved it out and created its own policy. ----  Same results

Don't see a single thing different in this configuration than any other.

Thanks,

Bill

[watsons]

Have you increased the "client read timeout" for this particular client? An error 69 could be a socket timeout issue.

What if you just backup one or two folders with less files (< 100+) and see if it works?

May also want to increase the backup host's TCP keepalive time : http://www.symantec.com/docs/HOWTO56221

[babu_beesetty]

First can you please compare the OS version of failed client with other clients? (SP1 or SP2)

Could you please even check with your windows team, if there are any Windows updates pending on the client server, if there are any can they schedule for updating patches.

[Maurice_Byrd]

I agree with Watsons. It sounds like a timeout issue. You may have to enable logging on the client to see if the backup is getting stuck on a particular file. 

You might also try changing the backup selection to target the specific local drives: C:\, E:\, etc.  And enable multiple data streams. If it is getting stuck on a particular file, this should help you narrow that down.

[babu_beesetty]

Hello Maurice,

To my understanding from post1, backups are failing when trying to take backup of  Shadow Copy Componenets. is it still requried to check the backups of Local drives..  please correct me if am wrong.

ERR: Unable to backup System State or Copy

[Mark_Solutions · Accepted solution]

You cannot backup the System State / Shadow Copy Components without using VSS

If you policy uses ALL_LOCAL_DRIVES then that will include the System State / Shadow Copy Components and these are always backed up using VSS - there is no other way

If you have disabled VSS then you can only backup the hard drives - so change you selections to C:\, DL\ etc.

Hope this helps

[w_boggs]

@Mark_Solutions

Checked status of client, it does have multiple drives, so I changed the selection to C;\Drive only....ran a differential backup and it completed successfully.....with the exception of some Endpoint Protection Errors.

Thanks so much!   Any suggestion on how to get rid of warning messages bpbm(pid=5128)  WRN: -can't open file c:\program data\symantec\symantec endpoint protection\12.1.2015.2015.105\Data.....

Is there some place to go to set exceptions, or a specific way to deal with Stmantec Endpoint warnings.

Thanks,

Bill

[w_boggs]

@All,

Would really like a defoinitive answer too on VSS... Technician told me to disable on all clients when we did initial install of NetBackup.

What's the right answer..  Enabled or Disabled ?   Client,  Master and Media?

Bill

[Mark_Solutions]

Maybe he meant VSP - disabled (or at least not selected to be used) in Master Servers Host Properties

If you want to backup a servers Syatem State / Shadow Copy Components you need it to be enabled - it can be set to manual startup but should not be disabled.

As for the endpoint error it will also be down to VSS not being used as you cannot backup any open files

Either enable VSS to allow open files to be backed up or exclude the files / directories that cause issues

I would enable VSS - that way you can back everything up correctly