Backup Job for VM servers ends with 'system call failed(11)'

[Nerdling]

Hi Guys,

When running a backup of my Virtual servers the job appears to fail.  Job appears to write data out to tape, however it fails with 'system call failed (11).

OS on both VM and Master server is Win2K8 r2.  I think the version of ESXI is 4.0

This is the output from the job log:

19/01/2012 1:34:39 PM - requesting resource MasterSrv-hcart-robot-tld-0
19/01/2012 1:34:39 PM - requesting resource MasterSrv.NBU_CLIENT.MAXJOBS.MyVMHOST
19/01/2012 1:34:39 PM - requesting resource MasterSrv.NBU_POLICY.MAXJOBS.FSGE-VMware-W2K8
19/01/2012 1:34:39 PM - granted resource MasterSrv.NBU_CLIENT.MAXJOBS.MyVMHOST
19/01/2012 1:34:39 PM - granted resource MasterSrv.NBU_POLICY.MAXJOBS.FSGE-VMware-W2K8
19/01/2012 1:34:39 PM - granted resource A00034
19/01/2012 1:34:39 PM - granted resource IBM.ULTRIUM-HH4.001
19/01/2012 1:34:39 PM - granted resource MasterSrv-hcart-robot-tld-0
19/01/2012 1:34:39 PM - estimated 0 Kbytes needed
19/01/2012 1:34:39 PM - started process bpbrm (5868)
19/01/2012 1:34:43 PM - connecting
19/01/2012 1:34:43 PM - connected; connect time: 00:00:00
19/01/2012 1:34:43 PM - mounting A00034
19/01/2012 1:35:23 PM - mounted; mount time: 00:00:40
19/01/2012 1:35:23 PM - positioning A00034 to file 156
19/01/2012 1:37:32 PM - positioned A00034; position time: 00:02:09
19/01/2012 1:37:32 PM - begin writing
19/01/2012 2:17:41 PM - Error bpbrm(pid=5512) from client MyVMHOST: ERR - Read of VM file read 0 bytes, should have read 46592 bytes
19/01/2012 2:26:15 PM - Error bptm(pid=5848) read of TIR file failed: the file list is incomplete  
19/01/2012 2:34:41 PM - Error bptm(pid=5848) read of TIR file failed: the file list is incomplete  
19/01/2012 2:34:46 PM - Error bptm(pid=5848) socket operation failed - 10038 (at bptm.c.29307)     
19/01/2012 2:34:59 PM - end writing; write time: 00:57:27
system call failed(11)

[Marianne]

Seems nothing is written....
"Error bpbrm(pid=5512) from client MyVMHOST: ERR - Read of VM file read 0 bytes, should have read 46592 bytes"

We need a bit more info please:
NBU version?
Backup method - client software installed in VM's or via VMware vStorage API?

Is this a new installation that has never worked before or a setup that used to work previously?

Please post policy config. Run this from cmd on master (remember to open cmd with 'Run As Administrator'):
....\netbackup\bin\admincmd > bppllist <policy-name> -L      (big l, please)

Logs needed (as a start):
bpbrm and bptm on the master/media server (seems your master is also media server).

more logs are needed, but will depend on backup method used.
 

[Nerdling]

Hi Marion,

NBU version 7.0 on Win2K8r2

There is no client software installed on the VM, how would I check to see if this is a VMware vStorage API?

The NBU install is current, the VM's are new and have not been backed up before.

Here is the output from bppllist:

Policy Name:       FSGE-VMware-W2K8
Options:           0x0
template:          FALSE
c_unused1:         ?
Names:             (none)
Policy Type:       FlashBackup-Windows (29)
Active:            yes
Effective date:    11/16/2010 11:52:12
File Restore Raw:  yes
Mult. Data Stream: no
Perform Snapshot Backup:   yes
Snapshot Method:           VMware
Snapshot Method Arguments: nameuse=0,Virtual_machine_backup=2,trantype=1,disable_quiesce=0,file_system_optimization=1,snapact=2,vmmono=0
Perform Offhost Backup:    yes
Backup Copy:               0
Use Data Mover:            no
Data Mover Type:           2
Use Alternate Client:      no
Alternate Client Name:     masterserver
Use Virtual Machine:      1
Hyper-V Server Name:     (none)
Enable Instant Recovery:   no
Policy Priority:   0
Max Jobs/Policy:   10
Disaster Recovery: 0
Collect BMR Info:  no
Keyword:           (none specified)
Data Classification:       -
Residence is Storage Lifecycle Policy:    no
Client Encrypt:    no
Checkpoint:        no
Residence:         masterserver-hcart-robot-tld-0
Volume Pool:       NetBackup
Server Group:      *ANY*
Granular Restore Info:  no
Exchange Source attributes:              no
Exchange 2010 Preferred Server: (none defined)
Generation:      37
Ignore Client Direct:  no
Client/HW/OS/Pri:  GW3AD02 vmx-07 windows7Server64Guest 0 0 0 0 ?
Client/HW/OS/Pri:  gw3db01.fsge.net vmx-07 windows7Server64Guest 0 0 0 0 ?
Include:           ALL_LOCAL_DRIVES
Schedule:          Monthly-Full
  Type:            FULL (0)
  Calendar sched: Enabled
    Friday, Week 5
  Maximum MPX:     5
  Synthetic:       0
  PFI Recovery:    0
  Retention Level: 9 (infinity)
  u-wind/o/d:      0 0
  Incr Type:       DELTA (0)
  Alt Read Host:   (none defined)
  Max Frag Size:   0 MB
  Number Copies:   1
  Fail on Error:   0
  Residence:       (specific storage unit not required)
  Volume Pool:     FSGE-Monthly
  Server Group:    (same as specified for policy)
  Residence is Storage Lifecycle Policy:     0
  Daily Windows:
   Day         Open       Close       W-Open     W-Close
   Sunday      000:00:00  000:00:00
   Monday      000:00:00  000:00:00
   Tuesday     000:00:00  000:00:00
   Wednesday   000:00:00  000:00:00
   Thursday    000:00:00  000:00:00
   Friday      018:00:00  065:10:00   138:00:00  185:10:00 017:10:00
   Saturday    000:00:00  000:00:00
Schedule:          Weekly-Full
  Type:            FULL (0)
  Calendar sched: Enabled
    Friday, Week 1
    Friday, Week 2
    Friday, Week 3
    Friday, Week 4
  Maximum MPX:     5
  Synthetic:       0
  PFI Recovery:    0
  Retention Level: 3 (1 month)
  u-wind/o/d:      0 0
  Incr Type:       DELTA (0)
  Alt Read Host:   (none defined)
  Max Frag Size:   0 MB
  Number Copies:   1
  Fail on Error:   0
  Residence:       (specific storage unit not required)
  Volume Pool:     FSGE-Weekly
  Server Group:    (same as specified for policy)
  Residence is Storage Lifecycle Policy:     0
  Daily Windows:
   Day         Open       Close       W-Open     W-Close
   Sunday      000:00:00  000:00:00
   Monday      000:00:00  000:00:00
   Tuesday     000:00:00  000:00:00
   Wednesday   000:00:00  000:00:00
   Thursday    000:00:00  000:00:00
   Friday      018:10:00  066:00:00   138:10:00  186:00:00 018:00:00
   Saturday    000:00:00  000:00:00
Schedule:          Daily-Incremental
  Type:            INCR (1)
  Frequency:       7 day(s) (604800 seconds)
  Maximum MPX:     5
  Synthetic:       0
  PFI Recovery:    0
  Retention Level: 1 (2 weeks)
  u-wind/o/d:      0 0
  Incr Type:       DELTA (0)
  Alt Read Host:   (none defined)
  Max Frag Size:   0 MB
  Number Copies:   1
  Fail on Error:   0
  Residence:       (specific storage unit not required)
  Volume Pool:     FSGE-Differential
  Server Group:    (same as specified for policy)
  Residence is Storage Lifecycle Policy:     0
  Daily Windows:
   Day         Open       Close       W-Open     W-Close
   Sunday      000:00:00  000:00:00
   Monday      018:00:00  030:00:00   042:00:00  054:00:00
   Tuesday     018:00:00  030:00:00   066:00:00  078:00:00
   Wednesday   018:00:00  030:00:00   090:00:00  102:00:00
   Thursday    018:00:00  030:00:00   114:00:00  126:00:00
   Friday      000:00:00  000:00:00
   Saturday    000:00:00  000:00:00
 

[Marianne · Accepted solution]

The policy config tells us that this is VMware vStorage backup and that the master server is also the backup host.

Since this is a new configuration - have you gone through the initial configuration steps as per chapter 3 in NBU for VMware Guide  http://www.symantec.com/docs/TECH127072 ?
e.g.
■ Adding the VMware backup host to NetBackup configuration
■ Adding NetBackup credentials for VMware

Have the datastores been zoned to the backup host?

Logs needed on backup host:
\Program Files\Common Files\VERITAS\VxMS\Logs
install_path\NetBackup\logs\bpfis

See Troubleshooting section of manual - link above.

[VirtualED]

Assuming this is an NBD base backup as the error is an indication that there is a timeout in VDDK and the VC/ESX host.  VDDK made a request for the data and after 3 attempts of not receiving anything, it failed.

I would advise upgrading your NIC drivers on the VMWare Backup Host and the Virtual Center.

If you have a SAN option, I would recommend switch to SAN base transport types.

http://www.symantec.com/docs/TECH167193

[Nerdling]

Thanks Guys,

sorry for the delay in replying.  The problem was in how the NBU Master server was talking to the VM Management sevrer (vCentre).

The credentials were configured correctly.

I added the master sevrer and vCentre sevrer and respective client ipaddress and host name to each host file on each server.

I installed/reinstalled VMware tools on each VM.

Everything seems to be working at the moment... til next time ;)