Netbackup Vmware backup error 11 system called failed

[blanco_adalbert]

Hello All, 

I would like to ask for a suggestion. I am facing with a backup error on vmware. I am backing up 400 VMs but when i try to backup some of them i got the error 11 system call failed.  I have read lots of posts but none of them help me to solve this problem.

I am backing up over SAN.

I have review the netbackup  logs and i have identify this.

BPBKAR on the backup host ( Master/Media server on linux).

16:24:18.663 [31487] <4> bpbkar: INF - Snapshot mapping type: ON
16:24:18.664 [31487] <4> bpbkar: INF - Snapshot BLIB type: OFF
16:24:39.730 [31487] <4> bpbkar: INF - Found 3 virtual volume(s)
16:37:51.298 [31487] <8> bpbkar: FTL - cleanup() failed, status 11
16:37:51.324 [31487] <16> bpbkar: ERR - bpbkar FATAL exit status = 11: system call failed
16:37:51.324 [31487] <4> bpbkar: INF - EXIT STATUS 11: system call failed
16:37:51.329 [31487] <4> bpbkar: INF - setenv FINISHED=0
[netbackup@S01NETAGNPN01 bpbkar]$

BMRBM on the backup host ( Master/Media server on linux).

16:37:51.298 [31481] <32> bpbrm main: from client S01ANTVIRPN01: FTL - cleanup() failed, status 11
16:37:51.298 [31481] <2> set_job_details: Tfile (50851): LOG 1431380271 32 bpbrm 31481 from client S01ANTVIRPN01: FTL - cleanup() failed, status 11

16:37:51.298 [31481] <2> send_job_file: job ID 50851, ftype = 3 msg len = 91, msg = LOG 1431380271 32 bpbrm 31481 from client S01ANTVIRPN01: FTL - cleanup() failed, status 11

16:37:51.299 [31481] <4> db_error_add_to_file: from client S01ANTVIRPN01: FTL - cleanup() failed, status 11
16:37:51.324 [31481] <2> bpbrm main: client S01ANTVIRPN01 EXIT STATUS = 11: system call failed

VxMS on the backup host ( Master/Media server on linux).

Begin VxMS Session...
Logmask set to 0x04000000
Current Environment Variables
VFM_ROOT = /usr/openv
VFM_PRIVATE_ROOT = /usr/openv
VFM_MAP_API_LIB = NULL
VFM_MAP_DIR = NULL
VFM_UTIL_LIB = NULL
   05/11/2015 15:59:57 : getIndexFromPath:RvpNtfsFileSystem.cpp:2401 <ERROR> : Bytes transferred = 0
   05/11/2015 15:59:57 : readRegistry:RvpNtfsFileSystem.cpp:1918 <ERROR> : Index for registry not found
   05/11/2015 15:59:57 : resolveVolumesFromNtfs:RvpFileSystemMgr.cpp:2660 <ERROR> : Reading mounted devices from regirstry failed
   05/11/2015 16:13:51 : writeData:RvpFile.cpp:310 <ERROR> : Returning: 28
   05/11/2015 16:13:51 : readFileDataIntoTempFile:RvpNtfsFileSystem.cpp:2025 <ERROR> : Execption: VERIFY_LINUX
   05/11/2015 16:13:51 : openId:RvpFileSystemMgr.cpp:3551 <ERROR> : Invalid NTFS object
   05/11/2015 16:13:51 : openId:RvpFileSystemMgr.cpp:3619 <ERROR> : Returning: 3
   05/11/2015 16:13:51 : rvpMapOpenId:RvpCoordinator.cpp:692 <ERROR> : Returning: 3
   05/11/2015 16:13:51 : rvp_map_open_id:rvpntfs.cpp:570 <ERROR> : Returning: 3
   05/11/2015 16:13:51 : openId:RvpObject.cpp:195 <ERROR> : Returning: 3
   05/11/2015 16:13:51 : openVolume:VixGuest.cpp:3168 <ERROR> : Returning: 3
   05/11/2015 16:13:51 : openVolume:VixManager.cpp:511 <ERROR> : Returning: 3
   05/11/2015 16:13:51 : openId:VixManager.cpp:361 <ERROR> : Returning: 3
   05/11/2015 16:13:51 : vixMapOpenId:VixCoordinator.cpp:1027 <ERROR> : Returning: 3
   05/11/2015 16:13:51 : vix_map_open_id:libvix.cpp:1314 <ERROR> : Returning: 3

[Pangal]

Hi,

         1) Are you  able to create snapshot at vcenter end

          2) Also check this forum  https://www-secure.symantec.com/connect/forums/vm-backup-alwayes-finished-backup-failed-back-request...

[blanco_adalbert]

Hello, 

Yes, I can create the snapshot.  Also I have lots of VMs over SAN and this is the one VM that has the error.

[blanco_adalbert]

I have a question, Does the virtual machine need to have installed the netbackup client inside to take VM backup over SAN?

BR.

[Marianne]

No, the VM does not need to have NBU client installed. Which OS and filesystem on the VM? Have you checked compatibility?

[Michael_G_Ander]

Wondering the failing VMs had something common ? Like being Windows VMs for example

Looks to me as there is some NTFS calls done which is failing, don't know if you need some specific libraries on Linux backup host for running backup of Windows VMs

[blanco_adalbert]

The Client Operating System is Windows 2008 R2 with 2 volume drives and 4GB RAM memory.

The virtual machine also the datastore on the esx has sufficient disk space indise.  There are 6 VM failing with this error and the total VMs that i am backing up is 400.

BR.

[blanco_adalbert]

I have found on the logs that the vm has found 3 volumes. I have reviewd and it has 2 drives C:\ and E:\  but it also have the system reserverd partition for the windows operating system boot.,

I am openig a support  case and one of the technotes sugested indicate that there is an unmapped volume drive: 

BUG REPORT: A VMware Consolidated Backup (VCB) can hang or fail reporting status code 11 (system call failed) if the Virtual Machine consists of an unmappable volume.

http://support.symantec.com/en_US/article.TECH66291.html

BMBRM Log

11:19:36.610 [21165] <2> write_file_names: buffering file name 'BACKUP /usr/openv/netbackup/online_util/fi_cntl/bpfis.fim.S01ANTVIRPN01_clon_1431446704.1.0.NBU_DATA.xml USING /dev/mapper/vg0-lv01 OPTIONS:FITYPE=MIRROR,MNTPOINT=/,FSTYPE=ext3' for output
11:19:36.610 [21165] <2> write_file_names: successfully wrote buffer to COMM_SOCK
11:19:36.610 [21165] <2> bpbrm main: wrote CONTINUE on COMM_SOCK
11:19:36.610 [21165] <2> bpbrm main: Value of PFI = 0
11:19:38.720 [21165] <2> bpbrm main: from client S01ANTVIRPN01_clon: INF - Snapshot mapping type: ON
11:19:38.720 [21165] <2> bpbrm main: from client S01ANTVIRPN01_clon: INF - Snapshot BLIB type: OFF
11:19:50.791 [21165] <2> bpbrm main: from client S01ANTVIRPN01_clon: INF - Found 3 virtual volume(s)
11:21:16.226 [21165] <2> Bpbrm_siginfo_print: 0: delay 0 signo SIGUSR1:10 code 0 pid 21173
11:21:16.226 [21165] <2> bpbrm mm_sig: received ready signal from media manager
11:21:38.682 [21165] <32> bpbrm main: from client S01ANTVIRPN01_clon: FTL - cleanup() failed, status 11
11:21:38.682 [21165] <2> set_job_details: Tfile (51080): LOG 1431447698 32 bpbrm 21165 from client S01ANTVIRPN01_clon: FTL - cleanup() failed, status 11

11:21:38.682 [21165] <2> send_job_file: job ID 51080, ftype = 3 msg len = 96, msg = LOG 1431447698 32 bpbrm 21165 from client S01ANTVIRPN01_clon: FTL - cleanup() failed, status 11

11:21:38.683 [21165] <4> db_error_add_to_file: from client S01ANTVIRPN01_clon: FTL - cleanup() failed, status 11
11:21:38.687 [21165] <2> bpbrm main: client S01ANTVIRPN01_clon EXIT STATUS = 11: system call failed
11:21:38.687 [21165] <2> bpbrm kill_child_process: start
11:21:38.687 [21165] <2> bpbrm wait_for_child: start
11:22:03.333 [21165] <2> bpbrm wait_for_child: child exit_status = 82 signal_status = 0
11:22:03.333 [21165] <2> bpbrm Exit: sending signal=1 to client=S01ANTVIRPN01_clon, client_pid=21171
11:22:03.333 [21165] <2> bpcr_send_signal: Ignoring connect_opts = 0x01030202
11:22:03.333 [21165] <2> read_client: ?
11:22:03.333 [21165] <2> read_client: opendir() failed: S01NETAGNPN01.actinver.com.mx: No such file or directory (2)
11:22:03.333 [21165] <2> vnet_same_host_and_update: [vnet_addrinfo.c:2905] matched as locals NAME1=localhost NAME2=S01NETAGNPN01.actinver.com.mx
11:22:03.333 [21165] <2> vnet_same_host_and_update: [vnet_addrinfo.c:2905] matched as locals NAME1=localhost NAME2=S01NETAGNPN01.actinver.com.mx
11:22:03.333 [21165] <2> local_bpcr_connect: bpcr.c.333: connect_opts = 0x01020001 connect_opts2 = 0x01020001
11:22:03.333 [21165] <2> local_bpcr_connect: bpcr.c.340: connect_opts = 0x01020001

[Marianne]

Wow! That TN is dated  January 23, 2009 and was for NBU 6.5.3. Supposed to be fixed in NBU 6.5.4.

Have you pointed this out to the Support engineer?

Does it mean that the issue was re-introduced in NBU 7.6?