Netbackup 10-CDP Status: 4207 4726

[mehrdad64]

hi guys , 

im using Netbackup 10.1.1 -Primary server is installed on Suse EL 15 and Media server and CDP gateway are installed on RHEL 8.4

for last couple of days i've been trying to do CDP backups with no luck!

I used many configurations and changed a lot of things (standalone CDP gateway, Media server + Gateway...) and double-checked prerequisites multiple times, but no matter what CDP not working.

its important to note that VM's resides on VMWare vSAN , vCenter and ESXI  hosts version is 7.0.3 f

any help would be appreciated.

this is the error im getting :

snapshot backup of client 502ef3b3-fca6-fbb7-0cdd-b413e9ecd442 using method Hypervisor_snap
Nov 08, 2023 12:56:28 AM - Info bpbrm (pid=7462) INF - BPFIS MESSAGE: CDP gateway :xxx-bkup-ii.domain.net
Nov 08, 2023 12:56:28 AM - Critical bpbrm (pid=7462) from client 502ef3b3-fca6-fbb7-0cdd-b413e9ecd442: FTL - Cannot take backup. IOTapping not started. No PIT available.
Nov 08, 2023 12:56:28 AM - Critical bpbrm (pid=7462) from client 502ef3b3-fca6-fbb7-0cdd-b413e9ecd442: FTL - vfm_freeze: method: Hypervisor_snap, type: FIM, function: Hypervisor_snap_freeze
Nov 08, 2023 12:56:28 AM - Critical bpbrm (pid=7462) from client 502ef3b3-fca6-fbb7-0cdd-b413e9ecd442: FTL - vfm_freeze: method: Hypervisor_snap, type: FIM, function: Hypervisor_snap_freeze
Nov 08, 2023 12:56:28 AM - Critical bpbrm (pid=7462) from client 502ef3b3-fca6-fbb7-0cdd-b413e9ecd442: FTL - snapshot processing failed, status 4726
Nov 08, 2023 12:56:28 AM - Critical bpbrm (pid=7462) from client 502ef3b3-fca6-fbb7-0cdd-b413e9ecd442: FTL - snapshot creation failed, status 4726
Nov 08, 2023 12:56:28 AM - Warning bpbrm (pid=7462) from client 502ef3b3-fca6-fbb7-0cdd-b413e9ecd442: WRN - ALL_LOCAL_DRIVES is not frozen
Nov 08, 2023 12:56:28 AM - Info bpfis (pid=7474) done. status: 4726
Nov 08, 2023 12:56:28 AM - end Application Snapshot: Create Snapshot; elapsed time 0:01:57
Nov 08, 2023 12:56:28 AM - Info bpfis (pid=7474) done. status: 4726: Failed to create snapshot of the specified VM.

thanks!

[davidmoline]

Hi @mehrdad64 

Does the CDP gateway have multiple IP addresses? If so ensure the nbcctb process is listening on the correct IP address that both the primary and ESXi servers resolve the gateway to.

Another possibility is the VM was cloned and the BIOS UUID was not changed causing a possible problem if the original VM is also being protected by CDP (I do no think this would be the case here though). 

If none of the above apply, then I'd suggest logging a support case 

Cheers
David

[mehrdad64]

Hi @davidmoline 

thanks for your help, yes indeed both of my servers (Primary and CDP) have 2 IP addresses, after reading you replay i corrected dns records for all esxi hosts and vcenter. now all hosts and vcenter are resolving cdp and primary using correct ip address (192.168.110.xx same network as hosts) , although primary and cdp resolving each other on a different subnet (10.10.10.xx).

---

@davidmoline wrote:

If so ensure the nbcctb process is listening on the correct IP address 

---

How can i check that , i couldnt find anyway to find the ip binding of nbcctd proccess.

thanks

[mehrdad64]

i added host file for primary and cdp server so that now they are all resolve in the same subnet with esxi and vcenter. (192.168.10.xx) after that did a clear host cache but still getting the same error "IOTapping not started. No PIT available."

how can i test veritas vtstap I/O filter , maybe vstap is not working correctly.

[davidmoline]

Hi @mehrdad64 

Either use the lsof command with PID of the nbcctb command, or use the netstat command and look for port 33056. 

Cheers
David

[mehrdad64]

its running and listening on 0.0.0.0 so i guess its binding to all the ip addresses.

do you know where can find more details about the cdp process ,maybe a log file(other than nbcctd) in CDP or vCenter.

thanks!

netstat -p -e -l|grep 33056
tcp 0 0 xxx-bkup-ii.domain.net:33056 0.0.0.0:* LISTEN root 38869 5882/nbcctd

-------------------------------------------------------------
ps ax | grep nbcctd
5882 ? Ssl 0:12 /usr/openv/netbackup/bin/nbcctd -X
112835 pts/0 S+ 0:00 grep --color=auto nbcctd
--------------------------------------------------------------
lsof |grep 5882

nbcctd 5882 6190 nbcctd root 3w REG 253,0 164141 134292952 /usr/openv/netbackup/logs/nbcctd/root.110823_00001.log
nbcctd 5882 6190 nbcctd root 4wW REG 253,0 0 201328164 /usr/openv/netbackup/nbcct/lock/nbcctd.lock
nbcctd 5882 6190 nbcctd root 5u IPv4 38869 0t0 TCP xxx-bkup-ii.domain.net:33056 (LISTEN)
nbcctd 5882 6190 nbcctd root 6r REG 253,0 767 79755012 /etc/group
nbcctd 5882 6190 nbcctd root 7u unix 0xffff9a4428e5e300 0t0 38871 /tmp/.cct_comms type=STREAM
nbcctd 5882 6190 nbcctd root 8u a_inode 0,14 0 12544 [eventpoll]

[davidmoline]

hi @mehrdad64 

The process is listening on the local address xxx-bkup-ii.domain.net as per the netstat and lsof output (the 0.0.0.0:* is the remote address). The local address is xxx-bkup-ii.domain.net:33056.

I'd review the troubleshooting section in the VMware WebUI admin guide and see if that helps. If all else fails, log a support call. 

David

[mehrdad64]

---

@davidmoline wrote:

The process is listening on the local address xxx-bkup-ii.domain.net 

thats intersting ,  i have an idea. maybe i should add second ip address in the hosts file . 

---

---

[davidmoline]

Hi @mehrdad64 

The thing to get right is that the resolution of the hostname for the CDP Gateway defined in /etc/hosts on the CDP Gateway matches the hostname resolution for CDP Gateway defined on the NBU Primary server. This is the IP which ESXi will try to establish a network connection. 

Use the command "getent hosts <CDP gateway name>" on both the primary and CDP gateway machine to determine what each thinks the IP address is - they need to match.

David

[mehrdad64 · Accepted solution]

With the help of @davidmoline , i found the problem.

1- my NBU Servers had 2 IP addresses, one was in ESXi subnet, and another was in Backup subnet - Esxi and vcenter were resolving NBU servers to their backup subnet. adding nbu servers with esxi subnet IPs solved some of my problems.

2- CDP Gateway knew itself using Backup subnet IP, however adding CDP gateway server name and its second ip (ESXi subnet) to its host file did not solve the problem. even after clearing the OS and NBU cache.

3- but restarting both master and cdp gw did!

Thank you @davidmoline