Netbackup - VMWAre backup failure

[Maverick101]

Good day,

We have new backup environment that we done testing for about a month ago. The backups completed successfully. The problem started when me migrated clients to the new host. 

Netbackup V8 - Windows Server 2016

vSphere V6.7.0

Also, the backups for our prod environment works, just not our dev\QA and the versions are the same.

If there are any logs required to assist please let me know.

Extract from the Detailed status

19 Feb 2019 10:28:32 AM - Info bpbkar32 (pid=9580) INF - Backing up vCenter server cbvcvanp01, ESX host cbvmn1002, BIOS UUID 4206fbcc-fd3e-7905-dd29-d0c9edc9c09d, Instance UUID 5006da6f-7199-0497-cdb3-14f7f52e0419, Display Name CBWLNDVAPW007, Hostname CBWLNDVAPW007

19 Feb 2019 10:28:32 AM - begin writing

19 Feb 2019 10:29:32 AM - Info bpbkar32 (pid=9580) INF - Transport Type =  nbd

19 Feb 2019 10:33:43 AM - Error bpbrm (pid=7124) from client CBWLNDVAPW007: ERR - Cannot open file <vix>[UNITY905_NP_VOL10] CBWLNDVAPW007/CBWLNDVAPW007_1-000002.vmdk error = 25

19 Feb 2019 10:33:44 AM - Info bptm (pid=5032) waited for full buffer 14667 times, delayed 19837 times

19 Feb 2019 10:33:44 AM - Info bpbkar32 (pid=9580) bpbkar waited 0 times for empty buffer, delayed 0 times.

19 Feb 2019 10:33:50 AM - Error bpbrm (pid=7124) could not send server status message

19 Feb 2019 10:33:51 AM - Critical bpbrm (pid=7124) unexpected termination of client CBWLNDVAPW007

19 Feb 2019 10:33:51 AM - Info bpbkar32 (pid=0) done. status: 12: file open failed

19 Feb 2019 10:33:51 AM - end writing; write time: 0:05:19

file open failed  (12)

These errors we are seeing are for all the backups we currently running.

[Marianne]

@Maverick101 

I see this important info in your post:

The problem started when me migrated clients to the new host.

Transport Type = nbd
Cannot open file xxxxx.vmdk error = 25

Status 25: "cannot connect on socket"

Did you confirm port connectivity between "the new host " and NBU backup host? 

See: https://www.veritas.com/support/en_US/article.100011419

Problem

Firewall port requirements for the NetBackup for VMware agent.

Solution

The NetBackup backup host always requires connectivity to the VMware vCenter server at port 443 (TCP).

When using nbd as the backup or restore transport type the NetBackup backup host will need connectivity to each ESX/ESXi host at port 902 (TCP).

When using VMware Intelligent Policy (VIP), i.e. query builder, the NetBackup master server requires connectivity to the VMware vCenter server port 443 (TCP).

Note:  Ports 443 and 902 are default ports for VMware.  If these have been changed from the default in your VMware environment,the firewall requirements will change accordingly.

Note: The NetBackup backup host is also sometimes referred to as any of the following:

• Proxy server
• Access host
• VMware backup host
• Recovery host (at restore time)

[Maverick101]

Hi Marianne,

Yes have checked the ports. None of them are blocked. The backups run for about 10-12 minutes then they fail. 

[Marianne]

The job that you posted failed within 5 minutes. 

This corresponds with the default Client Connect Timeout.

Status 25 says to me that connection could not be made to port 902 from NBU backup host to this particular ESXi server where the vmdk resides (ESX host cbvmn1002).
The job details in your opening post does not reflect the name of the Backup Host.

. 
Another possible reason for status 25:  

https://www.veritas.com/support/en_US/article.100024891

Cause

The virtual machine on the VMware ESX host did not properly register on the Virtual Center Server.

The virtual image(s) or (templates in this example) did not properly register when either copied or imported from a previous system.

[Maverick101]

Thanks Marianne,

I will look into this end esnure we can have comms on the port 902 again. We could telnet to the esxi host on that part. These servers were migrated from another host so will look into that as well.