Need help diagnosing san transport status 6 on VMWare backup host

[banjo67xxx]

I have sucessfully configured the vCentre and backup host to work with NBD, however we only have a 100Mb/s LAN connection between the backup host and the ESX servers which makes it impractical to backup 5TB.

For now we havent got a license key to make the backup host a media server, but we do have 1Gb/s LAN to the media server (which is also NBU master) so we could backup 5TB in 17hrs if we could get san transport working. Once I get the license, I should be able to use 2x4Gb/s SAN to the VTL tape drives so get this down to 3-4hrs.

The NBU Media server/master is version 7.1 on linux. The VMWare backup host is NBU client 7.1 on Win2008R2. The VMWare backup host has LUN mappings to all the VMWare datastores in its own SAN host group, and I have setup Win2008R2 with automount disabled and SAN Policy=OnlineAll as per http://www.symantec.com/connect/forums/error-code-6-vstorage-backup-san-mode

However I'm still getting status 6 when I try to start a manual backup.

I've tried using vxlogview -a -t 00:02:00 to see what happened in the last 2mins, but I can't see the wood for the trees. Which log files do I need to look at to get the right amount of detail in the right place?

04/01/14 15:53:21.560 [Info] V-117-94 FSM JobTable[FBU_WINDOWS:EndNotifyScript] m_JobStep=11, Jobid=4715, returns=0
04/01/14 15:53:21.560 [Info] V-117-107 *********** Jobid 4715 status summary for FBU_WINDOWS: *********
04/01/14 15:53:21.560 [Info] V-117-114 Step 1 StartNotifyScript status 0 jobstatus -
04/01/14 15:53:21.560 [Info] V-117-114 Step 2 StepByCondition status 0 jobstatus -
04/01/14 15:53:21.560 [Info] V-117-114 Step 3 StreamDiscovery status - jobstatus -
04/01/14 15:53:21.560 [Info] V-117-114 Step 4 BMRSave status - jobstatus -
04/01/14 15:53:21.560 [Info] V-117-114 Step 5 ReadFileList status 0 jobstatus 0
04/01/14 15:53:21.560 [Info] V-117-114 Step 6 CreateSnapshot status 0 jobstatus 0
04/01/14 15:53:21.560 [Info] V-117-114 Step 7 PEMPreprocessed status 6 jobstatus 6
04/01/14 15:53:21.560 [Info] V-117-114 Step 8 ValidateImage status - jobstatus 6
04/01/14 15:53:21.560 [Info] V-117-114 Step 9 DeleteSnapshot status - jobstatus 6
04/01/14 15:53:21.560 [Info] V-117-114 Step 10 DeleteSnapshotOnExit status 0 jobstatus 6
04/01/14 15:53:21.560 [Info] V-117-114 Step 11 EndNotifyScript status 0 jobstatus 6
04/01/14 15:53:21.560 [Info] V-117-114 Step -12 StopOnError status 0 jobstatus 6
04/01/14 15:53:21.560 [Info] V-117-111 ********** job status is 6 **********
04/01/14 15:53:21.560 [Info] V-117-52 GenBackupJob Terminating status=6
 

[banjo67xxx]

I've found the license key on the NBU master, and now configured the VMWare backup host as a Media server.

I've also verified that SSO is working, so as soon as the transfer type is fixed I should get the full 4Gb/s per channel to the FC VTL.

Still can't work out why the VMWare traffic is refusing to work with transfer type san, and will only work with transfer type nbd ????? 

[Marianne]

Please post all text in Details tab of failed job as well as bpfis and VxMS logs on backup host.
Copy logs to txt files (e.g. bpfis.txt) and upload as File attachments.

[banjo67xxx]

attachments added

[banjo67xxx]

Thanks for the name of the log file. I'm new to NBU troubleshooting so I dont know which files to look in.

I found a bpfis file in /usr/openv/netbackup/db/snapshot/VM-BACKUP-HOSTNAME/bpfis_CLIENT_nnn and a search for the jobid finds a couple of files in /usr/openv/netbackup/db/jobs/*log*/ (attached)

I cant find anything other than dll files in the only VxMS folder that Windows search could find on the media server.

BTW, the error has changed from status 6 to status 13, but it still works with NDB and fails with SAN.

Key point in one of the logs was ...

LOG 1389183601 16 bpbrm 2156 from client ftl-apr01.sbetprod.ads: ERR - Error opening the snapshot disks using given transport mode: Status 23.

[Marianne]

Please read through the Troubleshooting section (Chapter 11) of NBU for VMware Admin Guide   http://www.symantec.com/docs/DOC3663

It tells you which log directories to create where in this section:

NetBackup logging for VMware

The next section is: Enabling VxMS logging 

Please post logs as File attachments as soon as you have them.

[banjo67xxx]

Hi,

I added the hostnames of the vCenter and every ESX server to the hosts file on both the NBU Master and NBU Media server/VMWare Backup host as per https://www-secure.symantec.com/connect/forums/error-opening-snapshot-disks-using-given-transport-mo...

I even added the Master and Media server to the hosts file on the vCenter server

I setup logging as per the penultimate comment on https://www-secure.symantec.com/connect/forums/error-opening-snapshot-disks-using-given-transport-mo...

I further checked for any more relevant logging from Chapter 11 of the NBU VMware Admin Guide

Here are the bpfis, bpbkar and VxMS logs from the media server vmware bacjup host ...

[RamNagalla]

        01/08/2014 15:49:01.0881 : g_vdInterfaceLogger:.\libvix.cpp:1683 <INFO> : DISKLIB-LINK  : "san://snapshot-1757[FUS-B-DS02] prod-vl-mw-web01/prod-vl-mw-web01.vmdk@MAL-VC01?kbooth-a/XXX" : failed to open (You do not have access rights to this file).  

        01/08/2014 15:49:01.0881 : g_vdInterfaceLogger:.\libvix.cpp:1683 <INFO> : DISKLIB-CHAIN : "san://snapshot-1757[FUS-B-DS02] prod-vl-mw-web01/prod-vl-mw-web01.vmdk@MAL-VC01?kbooth-a/XXX" : failed to open (You do not have access rights to this file).

        01/08/2014 15:49:01.0881 : g_vdInterfaceLogger:.\libvix.cpp:1683 <INFO> : DISKLIB-LIB   : Failed to open 'san://snapshot-1757[FUS-B-DS02] prod-vl-mw-web01/prod-vl-mw-web01.vmdk@MAL-VC01?kbooth-a/XXX' with flags 0x1e You do not have access rights to this file (3390).

see highleted above...

looks like ID that is provided in netbackup for Vcenter does not have enough rights for the user kbooth-a

[banjo67xxx]

This part of the log looks interesting ...

3:29:02.036 PM: [2224.6988] <2> tar_base::V_vTarMsgW: INF - virtual snapshot file name = <vix>[FUS-B-DS02] prod-vl-mw-web01/prod-vl-mw-web01.vmdk

3:29:02.036 PM: [2224.6988] <4> tar_base::V_vTarMsgW: INF - tar message received from get_virtual_file_paths

3:29:02.036 PM: [2224.6988] <2> tar_base::V_vTarMsgW: INF - virtual file name = [FUS-B-DS02] prod-vl-mw-web01/prod-vl-mw-web01_1.vmdk

3:29:02.036 PM: [2224.6988] <4> tar_base::V_vTarMsgW: INF - tar message received from get_virtual_file_paths

3:29:02.036 PM: [2224.6988] <2> tar_base::V_vTarMsgW: INF - virtual snapshot file name = <vix>[FUS-B-DS02] prod-vl-mw-web01/prod-vl-mw-web01_1.vmdk

3:29:02.036 PM: [2224.6988] <4> tar_base::V_vTarMsgW: INF - tar message received from get_virtual_FIID

3:29:02.036 PM: [2224.6988] <2> tar_base::V_vTarMsgW: INF - virtual machine XML file: C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.10.196.0.100_1389194915.0.VM_ObjInfoXML.xml

3:29:05.219 PM: [2224.6988] <4> tar_base::V_vTarMsgW: INF - tar message received from vxms_error_handler

3:29:05.219 PM: [2224.6988] <2> tar_base::V_vTarMsgW: INF - VxMS error - severity 2.

3:29:05.219 PM: [2224.6988] <4> tar_base::V_vTarMsgW: INF - tar message received from vxms_error_handler

3:29:05.219 PM: [2224.6988] <2> tar_base::V_vTarMsgW: INF - VxMS Error message 1 = Error opening snapshot disk(s) using given transport mode(s)

message 2 = Error opening snapshot disk(s) using given transport mode(s)

I also see in the VxMS logs

        01/08/2014 15:49:01.0881 : g_vdInterfaceLogger:.\libvix.cpp:1683 <INFO> : diskLibPlugin: [2014-01-08 15:49:01.881 04960 trivia

 'DiskLibPlugin'] Attempting to open LVID:525eaa57-a779dbdc-64b8-9c8e991ef448/525eaa57-990cc8ac-24b3-9c8e991ef448/1.

        01/08/2014 15:49:01.0881 : g_vdInterfaceLogger:.\libvix.cpp:1683 <INFO> : diskLibPlugin: [2014-01-08 15:49:01.881 04960 error 

'DiskLibPlugin'] No path to device LVID:525eaa57-a779dbdc-64b8-9c8e991ef448/525eaa57-990cc8ac-24b3-9c8e991ef448/1 found.

        01/08/2014 15:49:01.0881 : g_vdInterfaceLogger:.\libvix.cpp:1683 <INFO> : diskLibPlugin: [2014-01-08 15:49:01.881 04960 error 

'DiskLibPlugin'] Failed to open new LUN LVID:525eaa57-a779dbdc-64b8-9c8e991ef448/525eaa57-990cc8ac-24b3-9c8e991ef448/1.

As the SAN administrator I know that every LUN on the ESX servers are also presented to the Backup host. The only difference would be that ESX is likely to use multipathd, and the backup host in Win2008R2 so it cant run multipathd. Do I need to install MPIO?

The storage is a mix of EMC and HDS and PowerPath and HDLM are incompatible, so I cant install those on the backup host.

[RamNagalla]

i would definatly suggest that..

what is the storage using for data stores?

what the hardware version of VM and Vcenter versions?

dod you make the successfull backup using NDB with the same logins?

### EDIT##

i just saw its a combination of EMC and HDS

[banjo67xxx]

Thanks. The user is configured as a VMWare administrator with full rights to everything.

I've checked that I can take snapshots using this user, but when I tried to download a file from the datastore I got an NFC server error, which looks like http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=101719... We cant get downtime to fix this today, if that's what the problem is, and as its ESXi its difficult to get an ssh shell to check it.

I've just installed MPIO on the backup host, and tested that it is correctly configured.

The vCenter is 5.1.0 (1065491) and the ESX servers are ESXi 5.1.0-2013...

Yes, I can make successful backups with NDB.

[banjo67xxx]

I've found the password for the ESXi servers, and discovered that VMWare KB article is not relevant on version 5.1. I've tried adding all the hostnames to /etc/hosts on the vCenter and ESXi hosts, and that doesn't work.

However, in the meanwhile I've noticed something that might be a problem. The ESXi servers have datastores defines on local and NFS storage as well as the SAN. All the VM guests have their storage on the SAN datastores, and none use the local and NFS datastores. So is this an issue?

[banjo67xxx]

I've done some further testing using account vm-admin and found that this account can access the VMDK files and download them if the Vsphere client is run on the NBU Media Server/Backup host.

So as vm-admin has full access rights to the VMDK files, why is there still a problem with access rights showing in the NBU logs???

        01/09/2014 10:25:16.0745 : g_vdInterfaceLogger:.\libvix.cpp:1683 <INFO> : SAN: Cannot access a SAN/iSCSI LUN backing this virtual disk. (Hint: If you are using vcbMounter you can use the option "-m nbd" to switch to network based disk access if this is what you want.) If you were attempting file-level access, stop the vmount Service by typing "net stop vmount2" on a command prompt to force vmount to re-scan for SAN LUNs and re-try the command.

        01/09/2014 10:25:16.0745 : g_vdInterfaceLogger:.\libvix.cpp:1683 <INFO> : DISKLIB-LINK  : "san://snapshot-1968[FUS-B-DS01] FTL-PROD-SOAHTP02/FTL-PROD-SOAHTP02.vmdk@MAL-VC01?vm-admin/XXX" : failed to open (You do not have access rights to this file).

        01/09/2014 10:25:16.0745 : g_vdInterfaceLogger:.\libvix.cpp:1683 <INFO> : DISKLIB-CHAIN : "san://snapshot-1968[FUS-B-DS01] FTL-PROD-SOAHTP02/FTL-PROD-SOAHTP02.vmdk@MAL-VC01?vm-admin/XXX" : failed to open (You do not have access rights to this file).

        01/09/2014 10:25:16.0745 : g_vdInterfaceLogger:.\libvix.cpp:1683 <INFO> : DISKLIB-LIB   : Failed to open 'san://snapshot-1968[FUS-B-DS01] FTL-PROD-SOAHTP02/FTL-PROD-SOAHTP02.vmdk@MAL-VC01?vm-admin/XXX' with flags 0x1e You do not have access rights to this file (3390).

        01/09/2014 10:25:16.0745 : vdOpen:.\VixInterface.cpp:196 <ERROR>  : VixDiskLib_Open() error. Exited

        01/09/2014 10:25:16.0745 : openLeafSnapshotDisks:.\VixGuest.cpp:452 <ERROR>  : vdOpen() error = 13. Calling closeLeafSnapshotDisks()

        01/09/2014 10:25:16.0745 : openLeafSnapshotDisks:.\VixGuest.cpp:555 <ERROR>  : Exited with failure

        01/09/2014 10:25:16.0745 : vixMapObjCtl:.\VixCoordinator.cpp:904 <ERROR>  : Exited with 23

        01/09/2014 10:25:16.0745 : vix_map_objctl:.\libvix.cpp:1138 <ERROR>  : Error 23

Do I need to start asking this question on the VMWare forums instead?

[banjo67xxx]

I've asked the same question on VMWare forum at thread https://communities.vmware.com/thread/467569

[RamNagalla · Accepted solution]

Hey,

by any change did you see the Tech note

http://www.symantec.com/business/support/index?page=content&id=TECH174128

as per tech note the vSphere 5 is not supported untill 7.1.0.3 

and also the dissuccion

https://www-secure.symantec.com/connect/forums/netbackup-7102-vsphere-5-and-san-transport

[banjo67xxx]

Aha!!! Thanks for the solution. I've found 7.1.0.4 so I guess that's the latest 7.1 patch. :)

[RamNagalla]

Great.. so now you got the way to fix it.. good luck..

and i would suggest you to upgrade to 7.5.0.5 if possible..because the 7.5 has more flexiblity for the Vmware backups..(i know 7.6 too.. but its new kid)

[banjo67xxx]

Thanks - I've confirmed the patch fixes it. If we have a 7.5 serial/license key I'll upgrade, but I suspect we only have 7.1 license.

[Marianne]

If your NBU maintenance is up-to-date you qualify for free upgrade.