Backup host upgrade from 7.1.0.4 to 7.6.0.4

[faridux]

Hi,

I would like if someone could help on a particular problem I meet with a windows backup host upgrade

We have upgraded our master to 7.6.0.4 last week

Yesterday I have upgraded my backup host from 7.1.0.4 to 7.6.0.4

Now all the backups in my flashbackup policy fail with status code 6, I got below error in the jb detail

03/20/2015 09:04:36 - Error bpbrm (pid=8360) from client GFY03KO2: ERR - Error opening the snapshot disks using given transport mode: Status 23

I copied the policy and modified to wmware policy  type

I tried host add and san transport but I still have the error

in vxms log I got this kind of error:

Begin VxMS Session...
Logmask set to 0x00000001
Current Environment Variables
VFM_ROOT = NULL
VFM_PRIVATE_ROOT = NULL
VFM_MAP_API_LIB = NULL
VFM_MAP_DIR = NULL
VFM_UTIL_LIB = NULL
   12:25:01.0473 : connectToHost:VixGuest.cpp:1008 <ERROR> : vdConnect() Error = 3.
   12:25:01.0473 : vixMapObjCtl:VixCoordinator.cpp:987 <ERROR> : Returning: 23
   12:25:01.0473 : vix_map_objctl:libvix.cpp:1258 <ERROR> : Returning: 23

in bpbrm I can see:

12:25:01.473 [3668.8744] <16> bpbrm main: from client GDN0SI06: ERR - Error opening the snapshot disks using given transport mode: Status 23

I do not understand why the san transport is no more available, while I can see in the vcenter GUI snapshot creation successful

The zoning was not modified, the vcenter credential are the same

Any help will be really appreciate

Thank you!

[Nicolai]

VMware got its own policy type now - cange policy type from flashbackupto vmware. Select NBD as transport method.

Verify name resoloution is in place - been the case in once instance before on Connect:

https://www-secure.symantec.com/connect/forums/error-opening-snapshot-disks-using-given-transport-mode-status-23-backup-failed-back-requeste#comment-7141131

[faridux]

Hi Nicolaï

Thanks for your feedback,

Actually I have already copied and updated the policy with nbplupgrade command with no error,

but both backups (with flashbackup and vmware policy type) fail with error 6

and unfortunetly nbd is not available as the ESXs are behind a firewall and I cannot reach their 902 TCP port

(by the way host files are already populated on the master and the backup host)

The only access I have is the vcenter

I am a bit lost, as it was running just before the upgrade and nothing has changed at SAN and backup up  host OS levels

[Marianne]

Port 902 has always been a requirement. No idea how things have worked previously without that.

[faridux]

Hi Marianne

the virtual server configured on my master is the vcenter not the ESXs,

The credential given by the vmware administrator can connect to the vcenter only

I can see the snapshot creation in vcenter console during snapshot backup job in the activity monitor

I don't think the backup host has to communicate with the ESX through the network when san transport mode is set in a vmware policy and the vcenter is the virtual host configured (feel free to correct me if I am wrong)

that's what I understood from this article:

https://www-secure.symantec.com/connect/blogs/nuts-and-bolts-netbackup-vmware-transport-methods-and-...

I think next week I will reinstall the backuphost from scratch and remake the datastore zoning, maybe something went wrong during the upgrade at OS level. status 6 appears just after the upgrade

[sdo]

What versions of ESX, vSphere and vCenter are you running?

I'm wondering if it might be worth checking your VDDK version compatibility:

https://www-secure.symantec.com/connect/forums/how-find-vmware-api-version-used

...also have a look at this:

https://www-secure.symantec.com/connect/forums/it-correct-infer-appliance-v2602-no-longer-supports-vsphere-esx-v41

[Marianne]

Network comms is still needed, even with SAN transport. Have you created log folders on the backup host? What is logged in bpfis and VxMS?

[faridux]

Hi,

@ sdo please find my updates below :

What versions of ESX, vSphere and vCenter are you running? The Vcenter is 5.0.0 and ESXi 5.0.0 update3

VDDK version compatibility: the VDDK embedded with my 7.6.0.4 is: 5.5.1 build-1601882

@Marianne

Please find attached bpfis,vxms,snapshot job,backup job logs for the last test I made

As far as I can see the parent snapshot job creates the snapshot

3/23/2015 10:03:26 AM - begin Parent Job
3/23/2015 10:03:26 AM - begin VMware, Start Notify Script
3/23/2015 10:03:26 AM - Info RUNCMD(pid=23265432) started            
3/23/2015 10:03:26 AM - Info RUNCMD(pid=23265432) exiting with status: 0         
Status 0
3/23/2015 10:03:26 AM - end VMware, Start Notify Script; elapsed time: 0:00:00
3/23/2015 10:03:26 AM - begin VMware, Step By Condition
Status 0
3/23/2015 10:03:26 AM - end VMware, Step By Condition; elapsed time: 0:00:00
3/23/2015 10:03:26 AM - begin VMware, Read File List
Status 0
3/23/2015 10:03:26 AM - end VMware, Read File List; elapsed time: 0:00:00
3/23/2015 10:03:26 AM - begin VMware, Create Snapshot
3/23/2015 10:03:26 AM - started
3/23/2015 10:03:28 AM - started process bpbrm (10192)
3/23/2015 10:04:51 AM - Info bpbrm(pid=10192) GDN0SI06 is the host to backup data from     
3/23/2015 10:04:51 AM - Info bpbrm(pid=10192) reading file list for client        
3/23/2015 10:04:51 AM - Info bpbrm(pid=10192) start bpfis on client         
3/23/2015 10:04:52 AM - Info bpbrm(pid=10192) Starting create snapshot processing         
3/23/2015 10:04:53 AM - Info bpfis(pid=6156) Backup started           
3/23/2015 10:04:53 AM - snapshot backup of client GDN0SI06 using method VMware_v2
3/23/2015 10:14:10 AM - end writing
Status 0
3/23/2015 10:14:10 AM - end VMware, Create Snapshot; elapsed time: 0:10:44
3/23/2015 10:14:10 AM - Info nbjm(pid=16449674) snapshotid=GDN0SI06_1427101323            
3/23/2015 10:14:10 AM - begin VMware, Policy Execution Manager Preprocessed
Status 150
3/23/2015 10:15:32 AM - end VMware, Policy Execution Manager Preprocessed; elapsed time: 0:01:22
3/23/2015 10:15:32 AM - Info bpfis(pid=6156) done. status: 0          
3/23/2015 10:15:32 AM - Info bpfis(pid=6156) done. status: 0: the requested operation was successfully completed    
3/23/2015 10:16:14 AM - end Parent Job; elapsed time: 0:12:48
3/23/2015 10:16:14 AM - begin VMware, Stop On Error
Status 0
3/23/2015 10:16:14 AM - end VMware, Stop On Error; elapsed time: 0:00:00
3/23/2015 10:16:14 AM - begin VMware, Delete Snapshot
3/23/2015 10:16:15 AM - started process bpbrm (7344)
3/23/2015 10:17:38 AM - Info bpbrm(pid=7344) Starting delete snapshot processing         
3/23/2015 10:17:39 AM - Info bpfis(pid=5012) Backup started           
3/23/2015 10:17:41 AM - Info bpbrm(pid=7344) INF - vmwareLogger: RegisterExtensionAPI: SYM_VMC_ERROR:  SOAP_ERROR      
3/23/2015 10:17:41 AM - Info bpbrm(pid=7344) INF - vmwareLogger: SOAP 1.1 fault: "":ServerFaultCode [no subcode]

Then during the backup on tape job I reach the mount tape step in bptm

...
3/23/2015 10:15:42 AM - Info bptm(pid=5584) media id C23814 mounted on drive index 10, drivepath {4,0,0,3}, drivename VTL2D031, copy 1
3/23/2015 10:15:49 AM - Error bpbrm(pid=9680) from client GDN0SI06: ERR - Error opening the snapshot disks using given transport mode: Status 23
3/23/2015 10:15:50 AM - Info bpbkar32(pid=10168) bpbkar waited 0 times for empty buffer, delayed 0 times.   
3/23/2015 10:15:55 AM - Error bpbrm(pid=9680) could not send server status message       
3/23/2015 10:15:55 AM - Critical bpbrm(pid=9680) unexpected termination of client GDN0SI06        
3/23/2015 10:15:55 AM - Info bpbkar32(pid=0) done. status: 6: the backup failed to back up the requested files
the backup failed to back up the requested files(6)

VxMS provider log displays also the error 23

Begin VxMS Session...
Logmask set to 0x00000001
Current Environment Variables
VFM_ROOT = NULL
VFM_PRIVATE_ROOT = NULL
VFM_MAP_API_LIB = NULL
VFM_MAP_DIR = NULL
VFM_UTIL_LIB = NULL
   10:14:24.0281 : vdOpen:VixInterface.cpp:492 <ERROR> : VixDiskLib_Open() error: 3014
   10:14:25.0919 : vdOpen:VixInterface.cpp:492 <ERROR> : VixDiskLib_Open() error: 3014
   10:14:25.0919 : vixMapObjCtl:VixCoordinator.cpp:987 <ERROR> : Returning: 23
   10:14:25.0919 : vix_map_objctl:libvix.cpp:1258 <ERROR> : Returning: 23

But it does not seem to acces to the snapshot
Thank you again to help me on this

[Marianne · Accepted solution]

Have a look at this TN: http://www.symantec.com/docs/TECH225372

[faridux]

Thank you, I will contact the VMware admin to check and keep you posted

[sdo]

The NetBackup v7.6.0.4 Release Notes:

http://www.symantec.com/business/support/index?page=content&id=DOC7450

...says on page 18:

"Support for VDDK 5.5 Update 1"

...and the VDDK 5.5 U1 (actually referred to as 5.5.1 on VMware's site) release notes:

https://www.vmware.com/support/developer/vddk/vddk-551-releasenotes.html

...says this:

"The VMware policy concerning backward and forward compatibility is for VDDK to support N-2 and N+1 releases. In other words, VDDK 5.5 and all of its update releases support vSphere 5.0, 5.1, and 6.0 (except new features)."

...so you should be ok... but it's the little caveat statement of "(except new features)" which might, but might not, be interesting...

...i.e. on paper it looks like you should be ok... unless the feature that NetBackup v7.6.0.4 and its embedded VDDK 5.5 U1 is trying to leverage is a "new" feature which does not exist in your instances of "vCenter 5.0.0 and ESXi 5.0.0 update3".  But I suspect that there is every possibility that this final statement of mine could be unintentionally misleading.  I guess you may end up having to engage either, or both of, Symantec Support and/or VMware Support to determine what it is that "bpbkar(32) -> bpfis -> VxMS -> VDDK (dll/lib)" is actually doing at the bottom of the software call stack.

[sdo]

I think Marianne is right...

...your 'vix' API error code from:

vdOpen:VixInterface.cpp:492 <ERROR> : VixDiskLib_Open() error: 3014

...is 3014, and this URL:

https://www.vmware.com/support/developer/vix-api/vix16_reference/errors/errors.html

...says:

3014 – VIX_E_HOST_USER_PERMISSIONS - Insufficient permissions in host operating system.

[faridux]

Hi SDO and Marianne

Sorry for the late reply I was waiting for my vmware admin feedback

I have set up new credential with a vcenter administrator account and it works correctly

Thanks again