I thought I would post a solution here to a Status 23 error from bpbrm which results in a Status 6 on the parent VMware backup job (child job never spawns).
The particulars:
NBU version: 7.5.0.1 / 7.5.0.4
Master Server: Solaris 10 (SPARC)
Media / Backup Host: Windows 2008 R2
ESXi 4.1.0 / vCenter 4.1.0
Storage: NetApp 3240
Policy Type: VMware
Storage Policy: Local DSSU
BLIB: Yes
Transport mode: NBD/NBDSSL (this issue can also apply to SAN)
If you're like me, you've already looked into the possible solutions below...
Ensure hostname resolution works for all ESXi hosts / vCenter to/from the Master Server
https://www-secure.symantec.com/connect/forums/error-opening-snapshot-disks-using-given-transport-mo...
https://www-secure.symantec.com/connect/forums/netbackup-vmware-error-opening-snapshot-disks-using-g...
Are the datastores presented/accessible to the VMware Backup Host? (not applicable for NBD)
https://www-secure.symantec.com/connect/forums/status-code-6-and-23-vstorage-api
https://www-secure.symantec.com/connect/forums/error-opening-snapshot-disks-using-given-transport-mo...
...but to no avail. Engaged tech support, turn VxMS logging on etc.
Logs produced look similar to this:
Detailed Status (from Activity Monitor):
10/23/2012 10:45:07 PM - Error bpbrm(pid=8764) from client <hostname>: ERR - Error opening the snapshot disks using given transport mode: Status 23
10/23/2012 10:45:08 PM - Info bptm(pid=4724) waited for full buffer 1 times, delayed 1619 times
10/23/2012 10:45:08 PM - Info bptm(pid=4724) EXITING with status 90 <----------
10/23/2012 10:45:08 PM - Info bpbkar32(pid=5064) bpbkar waited 0 times for empty buffer, delayed 0 times.
10/23/2012 10:45:11 PM - Error bpbrm(pid=8764) could not send server status message
10/23/2012 10:45:11 PM - Critical bpbrm(pid=8764) unexpected termination of client <hostname>
the backup failed to back up the requested files(6)
bpfis Log:
09:53:23.402 [6844.7840] <2> onlfi_vfms_logf: INF - vfm_fi_get_credentials: No disk array credentials found on server
09:53:23.621 [6844.7840] <2> onlfi_vfms_logf: INF - vfm_fi_get_credentials: No storage server credentials found on server
09:53:37.521 [6844.7840] <2> onlfi_vfms_logf: INF - vmwareLogger: RetrieveHostContents: Datastore: name = vmbackup is NOT accessible
09:53:37.521 [6844.7840] <2> onlfi_vfms_logf: INF - vmwareLogger: RetrieveDatastoreContents: datastore vmbackup is NOT accessible
VxMS Log:
18:57:28.0408 : connectToHost:xGuest.cpp:1017 <INFO> : TransportMode requested: nbd
18:57:28.0408 : connectToHost:xGuest.cpp:1022 <INFO> : Transport Modes available on this host: file:san:hotadd:nbdssl:nbd
18:57:30.0265 : vdConnect:xInterface.cpp:869 <DEBUG> : Done with VixDiskLib_Connect(): 214236752
18:57:30.0265 : vdOpen:xInterface.cpp:367 <INFO> : Calling VixDiskLib_Open()
18:57:30.0265 : g_vixInterfaceLogger:bvix.cpp:1833 <INFO> : VixDiskLibVimResolveHostName: Resolving IP address for hostname server.domain.com.
18:57:30.0265 : g_vixInterfaceLogger:bvix.cpp:1833 <INFO> : VixDiskLibVimResolveHostName: Resolved to 10.10.10.10.
18:57:30.0280 : g_vixInterfaceLogger:bvix.cpp:1833 <INFO> : VixDiskLibVim: VixDiskLibVimLogin
18:57:31.0185 : g_vixInterfaceLogger:bvix.cpp:1837 <WARN> : VixDiskLibVim: Not licensed to use this function.
This error will repeat for all available transport modes.
Steps we tried that did not work:
- Removed and re-added client to policy
- Upgraded from 7.5.0.1 to 7.5.0.4 (a suggestion from Symantec)
- vMotion VM to another ESXi host
- Storage vMotion VM to another datastore
Solution:
We cloned the VM and the clone successfully backed up. Which led us to wonder if it was a config issue on the VMware side (VMX file?).
After taking an outage for the client we did the following:
- Remove the offending VM from vCenter inventory.
- Rename the VM folder using the datastore browser. (This step is to avoid the same name folder that will be created in step 3)
- Create a new VM with the same name and specs. But no disks attached.
- Move the vmdk files from the old file folder to new folder.
- "Edit settings" on the newly created VM and attach "existing disk" by locating the VMDK files.
- Power on the new VM.
- Remove the old (renamed) file folder which should contain only the vmx, nvram and log files [house cleaning]
The client was then able to backup successfully.
