VM backup failing with status 23 and status 6

I had encountered the below error for nbu v8.1.2

23 Oct, 2019 2:27:46 PM - begin writing
23 Oct, 2019 2:27:46 PM - Info bptm (pid=980) start
23 Oct, 2019 2:28:01 PM - Error bpbrm (pid=4804) from client WINSERVER01: ERR - Error opening the snapshot disks using given transport mode: nbd:nbdssl:san:hotadd Status 23
23 Oct, 2019 2:28:02 PM - Error bptm (pid=980) socket operation failed - 10054 (at ../child.c.1287)
23 Oct, 2019 2:28:02 PM - Error bptm (pid=980) unable to perform read from client socket, connection may have been broken
23 Oct, 2019 2:28:07 PM - Info mediaserver.intranet.abc.net.it (pid=5076) StorageServer=PureDisk:mediaserver.intranet.abc.net.it; Report=PDDO Stats (multi-threaded stream used) for (mediaserver.intranet.abc.net.it): scanned: 3 KB, CR sent: 0 KB, CR sent over FC: 0 KB, dedup: 100.0%, cache disabled
23 Oct, 2019 2:28:08 PM - Error bpbrm (pid=4804) could not send server status message to client
23 Oct, 2019 2:28:10 PM - Critical bpbrm (pid=4804) unexpected termination of client WINSERVER01
23 Oct, 2019 2:28:10 PM - Info bpbkar32 (pid=0) done. status: 6: the backup failed to back up the requested files
23 Oct, 2019 2:28:10 PM - end writing; write time: 0:00:24
User backup failed (6)

i had checked firewall rules had already opened the 

  • tcp 443 & 902 for nbu master/media & vmware vcenter (able to telnet vcenter 445)
  • tcp 1556 & 13724 for nbu master/media (able to telnet master/media 1556/13724)

I had searched the knowledgebase and tried the technotes.

Please advise of any steps to resolve this issue?

Regards
Joshua

1 Solution

Accepted Solutions
Highlighted
Accepted Solution!

Re: VM backup failing with status 23 and status 6

Thank you everyone for your helpful contributions!

Finally resolved the problem after revisiting this KB article again.

VMWare backup error: Error opening the snapshot disks using given transport mode: Status 23

https://www.veritas.com/content/support/en_US/article.100010011

Rechecked my HOSTS file to verify the entries of hostnames and ip addresses of each ESX hosts carefully and spotted an ip address of a ESX server was matched to the hostname of another ESX server.

This silly typo mistake caused the NBU backup process to erroneously tried to access the snapshot files from the wrong ESX server's datastore.

After amending the HOSTS file with correct matching ip address & hostname of ESX host, i could successfully backup my VMware VMs again.

BR

View solution in original post

16 Replies

Re: VM backup failing with status 23 and status 6

Hello,

In general, this issue occurs is when the selected transport is not configured:

ERR - Error opening the snapshot disks using given transport mode: nbd:nbdssl:san:hotadd Status 23.

Please post vxms logs from your VM backup host (verbo=3) should be enough,

 

BR

 

Re: VM backup failing with status 23 and status 6

Also, you mentionned : 

i had checked firewall rules had already opened the 

  • tcp 443 & 902 for nbu master/media & vmware vcenter (able to telnet vcenter 445)

have you checked the telnet between the Vmbackup host and the vcenter over the port 902 in biderectionnel? not only the 443 port.

Re: VM backup failing with status 23 and status 6

and if you are able to telnet the ports, what is the ESX version? maybe you should look at this :

 

https://www.veritas.com/support/en_US/article.100014849

Snip :

"VMware backup failed display status code 23 and 6 by NBD method.

telnet to port 902 and 443 from backup host was working fine.
VxMS logs shows error CnxAuthdConnect: Returning false because CnxAuthdProtoReadResponse2 failed

Vcenter is not able to establish NFC connection to ESX server at port 902 as Connection is terminated by ESX server.

 
ESX 5.5 was installed on HP ProLiant BL460c Server Blade
This issue occurs with ESXi hosts running on HP hardware with below versions of AMS(Agentless Management Service):
  • hp-ams 500.9.6.0-12.434156
  • hp-ams-550.9.6.0-12.1198610
  • hp-ams 500.10.0.0-18.434156
  • hp-ams-550.10.0.0-18.1198610
 
This is a known issue affecting ESXi 5.x.
To resolve the issue, upgrade to AMS version 10.0.1.
 
For more information, please access these external links:
HP Agentless Management Service Offline Bundle for VMware vSphere 5.5
 
ESXi host cannot initiate vMotion or enable services and reports the error: Heap globalCartel-1 already at its maximum size.Cannot expand (2085618)
 

Re: VM backup failing with status 23 and status 6

Communication on port 902 is with the ESX hosts, not the vCenter. Please check the media server (backup host) can connect to the ESX hosts on 902.

Is this new or existing install? in otherwords, has it worked and stopped, or never worked?

Re: VM backup failing with status 23 and status 6

Hi hha_mea

Thank you for your reply.

Attached is the vxms log.

BR

Re: VM backup failing with status 23 and status 6

  • tcp 443 for nbu master/media & vmware vcenter (able to telnet vcenter 443)
  • tcp 902 Vmbackup host and the vcente (able to telnet 902)

the firewall rules are bi-directional.

BR

Re: VM backup failing with status 23 and status 6

I had re-checked and confirm that i can telnet from media server to ESX host port 902.

This is an existing installation and it working and stopped recently.

BR

Re: VM backup failing with status 23 and status 6

My ESX vmware v6.5 on Dell servers.

guessing the technote may not applicable to my environment.

BR

Re: VM backup failing with status 23 and status 6

Hi @jos_hua , the first error in the logs "16:13:58.0260 : vdOpen:VixInterface.cpp:486 <WARN> : Open() error: 14009" refers to these topics on vmware : 

 

https://www.dell.com/community/NetWorker/VDDK-Error-14009-The-server-refused-connection/td-p/7167244

https://communities.vmware.com/thread/547711

 

this sounds like a vmware issue  (an SSL issue)

snipp of the resolution :

 

The actual problem was SSL protocol not supported (old versions tls.protocols=tls1.0,tls1.1,tls1.2 ). This happen because one of the esxi host was version 5.1 (not supported), The client is going to migrate those VMs to the esxi version 5.5 and above . I already did the test on those esx and all work fine.

 

good luck

Re: VM backup failing with status 23 and status 6

Thank you for spotting the error.

I will check on this TLS portion.

 

BR

 

Re: VM backup failing with status 23 and status 6

Is this for all your VM's or just some/one? I think I had a similiar issues (can't look it up because I moved companies) and it only affected one giant VM. We never resolved it though. The vxms logs on level 8 only showed that the remote host closed the connection. We suspected something on the backup / media server OS was killing the connection.

It's just one, or some VM's, TLS would probably not be the issue as it would affect all backups.

Re: VM backup failing with status 23 and status 6

This NetBackup v8.1.2 issue (error status 6 & 23) is affecting all my Windows VMs backups on vmware v6.5

It was precisely for TLS 1.2 support that we had installed new NBU v8.1.2 and new vmware v6.5 during our hardware/technology refresh last year.

I had verified "Enable insecure communication with NetBackup 8.0 and earlier hosts" is turned on in Global Security Setting

I think this is for backward compatibility.

 

Re: VM backup failing with status 23 and status 6

Hello @jos_hua,

Could you please run another test with VxMs log at 8 please?

Was it working before the upgrade ? you said this is impacting all your windows Vms, how about linux/unix vms? is it working? 

BR

Highlighted
Accepted Solution!

Re: VM backup failing with status 23 and status 6

Thank you everyone for your helpful contributions!

Finally resolved the problem after revisiting this KB article again.

VMWare backup error: Error opening the snapshot disks using given transport mode: Status 23

https://www.veritas.com/content/support/en_US/article.100010011

Rechecked my HOSTS file to verify the entries of hostnames and ip addresses of each ESX hosts carefully and spotted an ip address of a ESX server was matched to the hostname of another ESX server.

This silly typo mistake caused the NBU backup process to erroneously tried to access the snapshot files from the wrong ESX server's datastore.

After amending the HOSTS file with correct matching ip address & hostname of ESX host, i could successfully backup my VMware VMs again.

BR

View solution in original post

Re: VM backup failing with status 23 and status 6

Great. Glad you sorted it out.

Re: VM backup failing with status 23 and status 6

Great news !

Glad you found the resolution, typo mistakes are the worst.. sometimes you think you configured it as it should be, but actually you didn't and at this point you get crazy and suspicious about what you learned.. (lol)..

Have a good day Smiley Happy!

BR;