How to create a virtual VMware Backup Host for NetBackup?

[Didi]

Hello everybody,

our company has a NetBackup environment in our Datacenter to backup our virtual infrastructure based on VMware vSphere. As we use NFS shares for our VMware datastore, VMware backups were planned to use NBD transport protocol with the known limitation in speed.

In a first step (before we start to migrate our datastores) I would like to use virtual VMware Backup Hosts to use HOTADD transport instead of NBD.

Unfortunately I was unable to find any step-by-step documentation from Veritas (competitors do have such documentation), how to create such virtual VMware Backup hosts that can read VMDKs directly from datastores via HOTADD, as long as VMs do not use IDE disks.

Here is what I found so far (correct me, if I am wrong) ...

Create a Windows VM (e.g. Windows Server 2012 R2) and install the NetBackup Client (in this case v8.0). For safety reason I would recommend to use the 'automount disable' and 'automount scrub' command in diskpart. Furthermore, I would recommend to disable the DiskManagement Snap-In via gpcedit.msc !

Finally, one has to include this new virtual VMware backup host in the host properties of the NetBackup Master Server and add it to the 'VMware backup hosts' settings.

Is that totally wrong or do I miss something?

Any reply is much appreciated!

[Mike_Gavrilov]

Do you get 4204 during backup job?

[Didi]

Yes, during backup job. NetBackup tries to make Snapshots on 2 VMs and then 4204 is the output.

[Mike_Gavrilov]

Could you share Job Details here? Could you migrate testvm to the same ESX host (for test perposes) there VM you're backing up is located?

[Didi]

testvm is on the same ESXi host, where those VMs are located that are part of the backup job. I am currently moving the VMware Backup Host VM to one of those iSCSI datastores as well.

Job details maybe later.

[Mike_Gavrilov]

Could you also enable VxMS logging on the testvm?

https://www.veritas.com/support/en_US/article.000123864

[Didi]

This is an excerpt of the Snaphot Job ...

Jul 3, 2017 4:36:19 PM - Info bpfis (pid=1320) Backup started
Jul 3, 2017 4:36:20 PM - snapshot backup of client ABC123 using method VMware_v2
Jul 3, 2017 4:36:20 PM - Critical bpbrm (pid=6736) from client ABC123: FTL - VMware_freeze: Unable to get servers at line 4787
Jul 3, 2017 4:36:20 PM - Critical bpbrm (pid=6736) from client ABC123: FTL - vfm_freeze: method: VMware_v2, type: FIM, function: VMware_v2_freeze
Jul 3, 2017 4:36:20 PM - Critical bpbrm (pid=6736) from client ABC123: FTL - vfm_freeze: method: VMware_v2, type: FIM, function: VMware_v2_freeze
Jul 3, 2017 4:36:20 PM - Critical bpbrm (pid=6736) from client ABC123: FTL - snapshot processing failed, status 4204
Jul 3, 2017 4:36:20 PM - Critical bpbrm (pid=6736) from client ABC123: FTL - snapshot creation failed, status 4204
Jul 3, 2017 4:36:20 PM - Warning bpbrm (pid=6736) from client ABC123: WRN - ALL_LOCAL_DRIVES is not frozen
Jul 3, 2017 4:36:20 PM - Info bpfis (pid=1320) done. status: 4204
Jul 3, 2017 4:36:20 PM - end Application Snapshot: Create Snapshot; elapsed time 8759:58:11
Jul 3, 2017 4:36:20 PM - Info bpfis (pid=1320) done. status: 4204: Incompatible client found
Jul 3, 2017 4:36:21 PM - Info bpbrm (pid=7004) Starting delete snapshot processing
Jul 3, 2017 4:36:22 PM - Info bpfis (pid=2968) Backup started
Jul 3, 2017 4:36:22 PM - Warning bpbrm (pid=7004) from client ABC123: cannot open C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.ABC123_1499092689.1.0
Jul 3, 2017 4:36:22 PM - Info bpfis (pid=2968) done. status: 4207
Jul 3, 2017 4:36:22 PM - end Parent Job; elapsed time 8759:58:13
Jul 3, 2017 4:36:22 PM - Info bpfis (pid=2968) done. status: 4207: Could not fetch snapshot metadata or state files

[Mike_Gavrilov]

Jul 3, 2017 4:36:20 PM - Critical bpbrm (pid=6736) from client ABC123: FTL - VMware_freeze: Unable to get servers at line 4787
Jul 3, 2017 4:36:20 PM - Critical bpbrm (pid=6736) from client ABC123: FTL - vfm_freeze: method: VMware_v2, type: FIM, function: VMware_v2_freeze
Jul 3, 2017 4:36:20 PM - Critical bpbrm (pid=6736) from client ABC123: FTL - vfm_freeze: method: VMware_v2, type: FIM, function: VMware_v2_freeze
Jul 3, 2017 4:36:20 PM - Critical bpbrm (pid=6736) from client ABC123: FTL - snapshot processing failed, status 4204
Jul 3, 2017 4:36:20 PM - Critical bpbrm (pid=6736) from client ABC123: FTL - snapshot creation failed, status 4204

Could you check if your testvm can resolve your ESXs hostnames and can connect to ports 443 and 902?

[Didi]

Well 443 is ok, but what is 902 and how can I test port 902?

[Mike_Gavrilov]

You can use telnet as always.

https://www.veritas.com/support/en_US/article.000122717

[Didi]

Connect from VMware Backup Host to ESXi server on 443 and 902 via telnet is possible, to vCenter only 443 is possible and I am unable to find, where to activate 902 in vCenter at the moment :(

[Mike_Gavrilov]

Port 902 should be opened on ESX/ESXi hosts only:

The backup host must also connect to the TCP port 902 on the ESX/ESXi hosts

Could you check resolving on vmtest?

>bpclntcmd -pn

>bpclntcmd -hn <ESXserver>

>bpclntcmd -ip <ESXserver_ip>

[Thiago_Ribeiro]

Hi,

In addition can you check this ?...It's a very interesting article

https://vox.veritas.com/t5/Netting-Out-NetBackup/Nuts-and-bolts-in-NetBackup-for-VMware-Transport-me...

Regards,

Thiago

[Didi]

I managed to get the backup job done now a couple of ours ago. So far it seems that I needed to configure the VMware Access Host with FQN. Master Server and Media Server are in a different DNS zone and iirc in a different subnet, must check that again tomorrow.

Used transport mode was HOTADD as mentioned in the job and as it was the only transport mode allowed. Throughput was disappointing though. Far beyond what I expected compared to a competitor product, which was 10x faster. Must further investigate.

Is the virtual VMware Access Host with the NetBackup Client able to do deduplication in its own RAM before sending the backup data to the MSDP pool? The VM has 4GB RAM and 4 vCPUs.

More news tomorrow.

[Mike_Gavrilov]

---

Is the virtual VMware Access Host with the NetBackup Client able to do deduplication in its own RAM before sending the backup data to the MSDP pool? The VM has 4GB RAM and 4 vCPUs.

---

Yes, you can enable client side de-dupe on the VMWare access Host. 4G is mininum but  in this case more is better

[Didi]

Quote Mike_Gavrilov: Yes, you can enable client side de-dupe on the VMWare access Host. 4G is mininum but  in this case more is better

The VMware Backup Host VM has 8GB RAM now. This machine is definitely in another IP subnet, hence the problems I had not configuring this VM with FQN under VMware Access Host in Host Properties.

Under the tab Attributes of the Policy 'Disable client-side deduplication' is not marked. Is that what you meant? Do I need to configure anything else on the VMware Backup Host? Of course the VMware Backup Host is selected in the VMware tab.

So far, I got only 60MB/s throughput with 2 VMs located on the same datastore and on the same ESXi-host. VMware Backup Host is also on the same ESXi-host and in the same datastore. The VM is connected via a vSwitch with 2x 10GBit/s LAN-interface.

Pretty bad throughput compared to NBD.

Any thoughts?

[Mike_Gavrilov]

---

Under the tab Attributes of the Policy 'Disable client-side deduplication' is not marked. Is that what you meant? Do I need to configure anything else on the VMware Backup Host? Of course the VMware Backup Host is selected in the VMware tab.

---

Yes, you need an extra step https://www.veritas.com/support/en_US/article.v27497507_nbu-msdp-web-deliverable

---

So far, I got only 60MB/s throughput with 2 VMs located on the same datastore and on the same ESXi-host. VMware Backup Host is also on the same ESXi-host and in the same datastore. The VM is connected via a vSwitch with 2x 10GBit/s LAN-interface.

---

Try to copy big file file from testvm to the media server via ftp to test if it can transfer data faster using LAN.

---

[Didi]

Quote Mike_Gavrilov: Yes, you need an extra step https://www.veritas.com/support/en_US/article.v27497507_nbu-msdp-web-deliverable

Done!

Quote Mike_Gavrilov: Try to copy big file file from testvm to the media server via ftp to test if it can transfer data faster using LAN.

Done as well!

*************************************************

I have done FTP-transfers from the virtual VMware Backup Host to both physical media servers and I can get between 70 and 600MB/s.

Using a virtual NetBackup VMware Backup Host with client-side deduplication and HOTADD transport mode, I get around 70MB/s transfer speed on SATA and 125MB/s on SAS.

Using a competitor virtual VMware Proxy Server with or without client-side deduplication and HOTADD transport mode, I get around 440MB/s transfer speed on SATA and 550MB/s on SAS.

Strange!

[Mike_Gavrilov]

---

Using a competitor virtual VMware Proxy Server with or without client-side deduplication and HOTADD transport mode, I get around 440MB/s transfer speed on SATA and 550MB/s on SAS.

---

Did you test a competitor using the same ESX? Could you create a snapshot of the test vm, log into ESX using ssh and try to read that snapshot usind dd (dd if=/path_to_the_file of=/dev/null)

 It'll show you possible read speed.