Backup failing with error 13 for windows 2008 R2

[Luckky_________]

Hi All,

I am facing backup failure issue with multiple windows 2008 R2 servers which is failing with error 13.

Master server: HPUX, NBU version: 7.5.0.6
Media server: HPUX, NBU version: 7.5.0.6
Client server: windows 2008, NBU version 7.5.0.4
Storage destination: DDBoost

Issue details: If backups run without multi stream backups failing for servers with error 13, iam feeling strange that how it is possible, if backups run in multi streaming its completing successfully but when we run jobs without multi stream, jobs getting fail with error 13.

If we run job in multistream bpbkar log s writing , without multistream bpbkar is not writing single byte of data and also its stuck in connecting state.

Connectivity is working fine i have verified from master and media servers.

ping 
Telnet 
btestbpcd
bpclntcmd from client also fine.

Below is the job logs, kindly refer and suggest me if anything that can be done.

one more strange thing is if i run backups with prod IP jobs running fine in multistream and without multistream :)

04/22/2015 14:47:51 - estimated 0 kbytes needed
04/22/2015 14:47:51 - Info nbjm (pid=14586) resumed backup (backupid=xxxxxx) job for client A, policy yyyy, schedule Daily_Incremental on storage unit zzdd-stu-disk
04/22/2015 14:47:53 - started process bpbrm (pid=27195)
04/22/2015 14:47:54 - connecting
04/22/2015 14:47:58 - Info bpbrm (pid=27195) starting bpbkar on client
04/22/2015 14:47:58 - connected; connect time: 0:00:00
04/22/2015 14:52:58 - Error bpbrm (pid=27195) socket read failed: errno = 232 - Connection reset by peer
04/22/2015 14:52:59 - Info bpbkar (pid=0) done. status: 13: file read failed
04/22/2015 14:52:59 - end writing
file read failed  (13)

[sdo]

When multi-streaming is used, are you running single stream (i.e. max_jobs 1 for the policy and/or max_jobs 1 for the client)?  Or do all streams all attempt to start at the same time?

What does this show:

# bpclient -L -client my.client.name

[watsons]

What is the "backup selection"? ALL_LOCAL_DRIVES?

How many clients are in that policy? 

[Luckky_________]

Hi Sdo,

backups runnign without multistream(meand multistream in policy attributes is unchecked), Policy have 160 clients .. once policy triggered all josb will come to activity monitor based on resource availability jbs are coming to active state.

Below is the command output:

max jobs per client value is : 130

#-> bpclient -L -client <Client name>
Client Name: <client name>
 Current Host:
        Hostname: <client name>
 Dynamic Address:       no
 Free Browse:   Allow
 List Restore:  Not Specified
 Max Jobs This Client:  Not Specified
 WOFB Enabled:  no
 WOFB FIM:      VSP
 WOFB Usage:    Individual Drive Snapshot
 WOFB Error Control:    Abort on Error
 Client Direct: Deduplication on the media server or
                Move data via media server
 Client Direct Restore: Move data via media server
 OST Proxy:     Off
 OST Proxy Server:      Unspecified
Connect options:        2 2 3
 Offline:       No

Hi watsons,

Yes backup selection is ALL_LOCAL_DRIVES and total 160 clients in policy

[sdo]

Master server global max jobs of 130 is quite a strange number.

What does this show:

> bpconfig -L

.

For the client that has a problem, why not try:

> bpclient -client name -update -max_jobs 1 -WOFB_enabled 1 -WOFB_FIM 1 -WOFB_usage 0 -WOFB_error 1

...and then re-try the multi-streamed backup?

[watsons]

So is this clientA a Windows 2008 or Windows 2008 R2? 

I agree with sdo's suggestion to enable WOFB and use VSS for snapshot (instead of VSP which has been discontinued). 

Since you mentioned when it used prod IP the backup would work regardless of whether you use multi-streaming, this setting may not be the root-cause of why it failed. It could be the backup IP - can you confirm if the name resolution of your IP is working fine? 

[sdo]

Yes, what I noticed is that WOFB is not even enabled, so it won't be attempting to use either VSP or VSS - and...

...AFAIK, the Symantec NetBackup VSP kernel driver ceased to be installed with one of the v7.x versions (I can't remember which version it ceased to be bundled with), and so, on fresh NetBackup Client installs, since that version (which I can't recall right now), VSP is no longer even present on some/most Windows clients these days.

[Luckky_________]

Hi sdo,

Here is the output for bpconfig -L command, I have disabled VSP and ran job, backups completed

#-> bpconfig -L

Mail Admin:             *NULL*
Job Retry Delay:        10 minutes
Max Jobs/Client:        130
Backup Tries:           2 in 12 hours
Keep Logs:              14 days
Max drives/master:      0
Compress DB Files:      (not enabled)
Media Mnt Timeout:      45 minutes
Shared Timeout:         0 minutes(unlimited)
Display Reports:        0 hours ago
Keep TIR Info:          1 days
Prep Interval:          4 hours (default)
Max Backup Copies:      3
DB Clean Interval:      12 hours
DB Clean Wait Time:     60 minutes
Policy Update Interval: 10 minutes

I have executed below command as well, its completed successfully, can you please eloborate what this comand will do and what is the use of it, i would like to know this becouse i have to make sure there is no impact for other client jobs.

> bpclient -client name -update -max_jobs 1 -WOFB_enabled 1 -WOFB_FIM 1 -WOFB_usage 0 -WOFB_error 1

[sdo]

Do you not have any NetBackup manuals/guides close to hand?

If not, see page 81 of the NetBackup 7.5 Commands Reference Guide here for details re the CLI arguments for 'bpclient':

https://support.symantec.com/en_US/article.DOC5182.html

Basically it enables WOFB (Windows Open File Backup), i.e. tells the backup client to use a snapshot handler, and tells NetBackup to use VSS as the snapshot handler, and tells NetBackup Client to call VSS in such a way as to use one snapshot per drive, and to attempt to continue the backup if the attempt to secure a VSS snapshot based backup fails - and sets the maximum number of concurrent backup job streams to 1.

.

No, it won't affect any other clients - it is specific to the one named client.

.

Did the non-multi-streamed backup work afterwards?

[Luckky_________]

Hi Sdo,

It worked fine, Multistream and without multi stream jobs are runnign fine.

Thanks for the solution.

Apologize for delay in responce as i am in out off office couple of days......