cancel
Showing results for 
Search instead for 
Did you mean: 

windows servers failing with 156

Albatross_
Level 5

Hi

we have few windows clients with 2008 R2 with SQL.

when we scheduled backup couple of clients are failing with 156 error.

The servers are having many paritions mounted on it. everytime when backup runs it failes with error code 156. when checked the logs they say 

I have checked with windows team, they claimed everything seems to fine

This is failing for L volume and K volume for few servers.

below are the logs. 

Any help will be highly appreciated

 

01/13/2016 00:42:56 - Info nbjm (pid=8353) starting backup job (jobid=4629902) for client sql01.mycompany.in, policy PROD_WIN, schedule Daily_Inc
01/13/2016 00:42:56 - estimated 590058730 kbytes needed
01/13/2016 00:42:56 - Info nbjm (pid=8353) started backup (backupid=sql01.mycompany.in_1452663776) job for client sql01.mycompany.in, policy PROD_WIN, schedule Daily_Inc on storage unit DD_01_DSU
01/13/2016 00:42:57 - started process bpbrm (pid=12919)
01/13/2016 00:42:58 - Info bpbrm (pid=12919) sql01.mycompany.in is the host to backup data from
01/13/2016 00:42:58 - Info bpbrm (pid=12919) reading file list for client
01/13/2016 00:42:59 - connecting
01/13/2016 00:43:00 - Info bpbrm (pid=12919) starting bpbkar on client
01/13/2016 00:43:00 - connected; connect time: 0:00:00
01/13/2016 00:43:02 - Info bpbkar (pid=16148) Backup started
01/13/2016 00:43:02 - Info bpbrm (pid=12919) bptm pid: 12920
01/13/2016 00:43:02 - Info bptm (pid=12920) start
01/13/2016 00:43:04 - Info bptm (pid=12920) using 1048576 data buffer size
01/13/2016 00:43:04 - Info bptm (pid=12920) setting receive network buffer to 262144 bytes
01/13/2016 00:43:04 - Info bptm (pid=12920) using 16 data buffers
01/13/2016 00:43:04 - Info bpbkar (pid=16148) change journal NOT enabled for <L:\>
01/13/2016 00:43:05 - Info bptm (pid=12920) start backup
01/13/2016 00:43:06 - Info bptm (pid=12920) backup child process is pid 12921
01/13/2016 00:43:06 - begin writing
01/13/2016 01:04:23 - Error bpbrm (pid=12919) from client sql01.mycompany.in: ERR - failure reading file: L:\PROD\APROD_20160107063021.trn (WIN32 2: The system cannot find the file specified. )
01/13/2016 01:04:25 - Error bpbrm (pid=12919) from client sql01.mycompany.in: ERR - Snapshot Error while reading file: GLOBALROOT\Device\HarddiskVolumeShadowCopy337\TxnLogs\PROD\APROD_20160107063021.trn
01/13/2016 01:04:27 - Critical bpbrm (pid=12919) from client sql01.mycompany.in: FTL - Backup operation aborted!
01/13/2016 01:04:28 - Error bptm (pid=12921) system call failed - Connection reset by peer (at child.c.1307)
01/13/2016 01:04:29 - Error bptm (pid=12921) unable to perform read from client socket, connection may have been broken
01/13/2016 01:04:29 - Error bptm (pid=12920) media manager terminated by parent process
01/13/2016 01:04:29 - Error bpbrm (pid=12919) could not send server status message
01/13/2016 01:04:30 - Info bpbkar (pid=16148) done. status: 156: snapshot error encountered
01/13/2016 01:04:30 - end writing; write time: 0:21:24
01/13/2016 05:23:56 - job 4629902 was restarted as job 4630178

 

Thanks

 

1 ACCEPTED SOLUTION

Accepted Solutions

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

You don't need to wait for server admins.

Simply add the following to Exclude List using Host Properties -> Clients on the master server:
*.mdf
*.ldf
*.trn

Inform server owners and your manager of your actions.
Explain that db files cannot be backed up in a file-level backup while database is online.

Do this as well:

On master server, open Host Properties -> Master -> Client Attributes.
Add Client name.
Select Client name. 
In Windows Open File Backup tab, select 'Disable snapshot and continue'.

View solution in original post

13 REPLIES 13

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
You need to exclude all database-related files from filesystem backups. SQL db's should be backed up using MS-SQL policy. If K and L drives are used for SQL db's and logs, exclude entire drives. Add K:\* and L:\* to Exclude List.

areznik
Level 5

Since you are trying to backup a server with SQL DBs using MS-Windows policy & VSS snapshots (good luck to you if you need to recover from this backup!), VSS probably running out of ShadowStorage space due to high I/O on the server.

run this command on the client:

vssadmin list shadowstorage

This shows you how much storage vss is currently using for snapshots (should be 0 if you're not in the middle of a backup) and the max its allowed to use. I believe the default is 300MB, which is low for a high I/O server. If its low, this is the command to change it:

vssadmin resize ShadowStorage /For=K: /On=K: /MaxSize=5GB

/For is the drive/volume you're changing it for, and /On is where the snapshot will be written to (recommend you keep snapshots on the same drive you're snapping).

 

 

 

Albatross_
Level 5

Hi Marrianne,

I have logged off from my work hours. I will be back tomorrow and check with the policy and other settings. I believe it should be MSSQL policy configured.

But anyways I will check and get back to you..

Thanks for quick reponse.

 

Thanks

 

nbutech
Level 6
Accredited Certified

This is certainly Ms-Windows policy type and not SQL policy as I don't see other processes in Job details which are started during SQL backups

You can follow earlier suggestions as NBU will see issues with db files as it will not be able to quiesce the db also access current transaction logs

Albatross_
Level 5

Hi All,

Yes nbutech, This client is included in MS-Windows type. but its a sql server and not included in MS-SQL Policy. I dont know why its been configured in such a way.

Yesterday It again scheduled Inc backup, and it successfully completed the backup on the volume L it was failing before. No errors found for this client.

Can anyone throw some light on this.

 

Thanks

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Successful backup this time means that VSS snapshot of SQL db's succeeded, but this is in NO WAY a guarantee that a restore of SQL db components will work.

Well, the restore will probably be successful but db's won't start....

(See areznik's post over heregood luck to you if you need to recover from this backup!)

VSS snaphot backups at OS level is not supported for online SQL db's.

You can only guarantee successful backup and restore via OS-level backup if databases are shutdown and all SQL services stopped.

If SQL cannot be stopped for the duration of the backup, you have 2 choices:

  1. a) Exclude SQL db folders/objects from file-level backups
    b) Create SQL policy for SQL db backups
  2. a) Exclude SQL db folders/objects from file-level backups
    b) Use SQL to backup/dump to disk, follwed by NBU file-level backup of the SQL dumps.

 

Albatross_
Level 5

Thanks Marianne.

I will check with the other peers. what made them to configure in such a way.

I do agree with you. Let me check with them and get back to you.

Thanks a lot.

 

Cheers

 

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
Any update here?

Albatross_
Level 5

Hi Marriane,

Yes I have requested windows team to check if the volumes contain any db related files. They have not come back.

Still awaiting.

Cheers

 

sdo
Moderator
Moderator
Partner    VIP    Certified

Hey Albatross... some questions:

1) Are you using "ALL_LOCAL_DRIVES" in the "Backup selections" of the backup policy?

2) Are you using "Allow multiple data streams" on the backup policy?

3) What does this command, on the master, report?     bpclient -L -client sql01.mycompany.in

4) What does this command, on the client, report?    vssadmin list shadowstorage

.

The are many different ways to configure backups, probably several "good" methods, probably only a few "better"methods, and usually only one or two "best" methods.

Now the question is:

5) Do you want/need to just cease the current error and move on to something else?  Or do you want to move through "good, better, best" ?

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
So, are backups still failing? Meaning you have no backups for these servers and they remain unprotected?

Albatross_
Level 5

Hi Sdo,

1. Yes we are using ALL_LOCAL_DRIVES .

2. Yes multiple datastreams enabled

3 bpclient: no entity was found (227) 

error code 227 : no entity was found
 

4. I need to check with windows team, I dont have access to the windows boxes.

5. Yes as of now I wanted to cease the error 156, ( I cannot take any more buggings from boss wink ) . 

But definetly I am looking forward to move to best configuration.

 

Cheers

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

You don't need to wait for server admins.

Simply add the following to Exclude List using Host Properties -> Clients on the master server:
*.mdf
*.ldf
*.trn

Inform server owners and your manager of your actions.
Explain that db files cannot be backed up in a file-level backup while database is online.

Do this as well:

On master server, open Host Properties -> Master -> Client Attributes.
Add Client name.
Select Client name. 
In Windows Open File Backup tab, select 'Disable snapshot and continue'.