Failed to get VM Server Info List

[schrammd]

Hi,
I am trying to set up new clients in the VCB policy on a 6.5.3 server. The proxy is dedicated and is running 6.5.3 as well. The Virtual Center Server has over 75 running VM's it knows about, yet when I try to Browse Virutal Machines in Net Backup, all I get is the "file read failed (13)" error. I created the bpvmutil logs and setup the verbose logging levels. Seems as though from the command line I can login to the VCS fine but when I query names, it only returns one junky little test VM and none of the others:

C:\Program Files (x86)\VMware\VMware Consolidated Backup Framework>vcbVmName.exe -h virmgmt101-dev -u sa_vmbackup
-p ****** -s any
[2009-04-21 07:00:57.739 'App' 5996 info] Current working directory: C:\Program Files (x86)\VMware\VMware Consolidated Backup
Framework
[2009-04-21 07:00:57.739 'BaseLibs' 5996 info] HOSTINFO: Seeing Intel CPU, numCoresPerCPU 4 numThreadsPerCore 1.
[2009-04-21 07:00:57.739 'BaseLibs' 5996 info] HOSTINFO: This machine has 1 physical CPUS, 4 total cores, and 4 logical CPUs.

[2009-04-21 07:00:57.739 'BaseLibs' 5996 info] Using system libcrypto, version 90709F
[2009-04-21 07:01:13.066 'BaseLibs' 5996 warning] SSLVerifyCertAgainstSystemStore: Subject mismatch: VMware vs virmgmt101-dev
[2009-04-21 07:01:13.066 'BaseLibs' 5996 warning] SSLVerifyCertAgainstSystemStore: The remote host certificate has these prob
lems:

* The host name used for the connection does not match the subject name on the host certificate

* A certificate in the host's chain is based on an untrusted root.
[2009-04-21 07:01:13.066 'BaseLibs' 5996 warning] SSLVerifyIsEnabled: failed to read registry value. Assuming verification is
disabled. LastError = 0
[2009-04-21 07:01:13.066 'BaseLibs' 5996 warning] SSLVerifyCertAgainstSystemStore: Certificate verification is disabled, so c
onnection will proceed despite the error
[2009-04-21 07:01:15.316 'vcbVmName' 5996 warning] IP address not set.
Found VM:
moref:vm-23253
name:test4
uuid:50165156-871d-1e74-52e2-18d82d7e8584
ipaddr:
 

I'll attach the relavent log file as I am stumped and we don't know where to look for the answers. The one thing that might be biting us is the VCB proxy / master/ media servers are completely firewalled off from the ESX/VCS servers. We have port 443 open from the VCB proxy to the VCS, and the logs show the communication is working. Just a sanity check, but do the master and media server need to talk to the VMware servers or does everything pass thru the proxy as I think?
 

thanks in advance!

[Anonymous]

Read this forum post:
https://www-secure.symantec.com/connect/forums/nbu-65-vmware-intergration-failed-get-vm-server-info-list

Note:
NetBackup uses DNS lookups to populate the browse list. Make sure that the virtual machine’s FQDNs can be resolved or they are listed in the HOSTS file on the VCB proxy server. This populated list is cached in a file on the VCB proxy server in:

C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\vcbnames_vc-FQDN_Any
In a few cases, the directory may not have been created during the install. If this is true, manually create and recreate the policy. This list may be outdated if new virtual machines have been added. Simply click the refresh button to update the list.

• Also check your NetBackup Admin Console > View menu >Options > Policies tab and increase the VMware connect timeout value. I'm sure this fixed that for me in the early days.

Problem reaching the Virtual Machine from vcb_best_practices.pdf
<snippet from page 16>
When you attempt a snapshot backup, you may receive an error message saying the command cannot reach the virtual machine. There are a number of possible reasons for this error message.
Sometimes, the virtual machine is not properly registered with the VirtualCenter database, or the virtual machine might not receive an IP address before the backup begins.

To test for these problems, run the vcbVmName command from a command prompt on the backup proxy server.

The following examples illustrate how to use vcbVmName:
• vcbVmName -s powerstate:on — Lists all powered on virtual machines.
• vcbVmName -s any: — Lists all known virtual machines.
• vcbVmName -s ipaddr:vm37.company.com — Displays information about the virtual machine with the specified address.

The following example illustrates the output of vcbVmName:

# vcbVmName -s name:vm37.company.com
Found VM:
moref:192
name:Virtual Machine 37
uuid:564d78a1-8c1c-59b4-fa02-be14138797be
ipaddr:10.17.5.31

Compare the IP address reported by vcbVmName with the address shown in the VI Client. The two should be the same. If they are not, it might indicate that your VirtualCenter database is not current and that some of the virtual machine information in the database is inaccurate. If there is a mismatch, it might indicate a VMware Tools failure. Make sure the version of VMware Tools installed in the virtual machine is up-to-date.
</snippet>

• In the VI Client can you see the IP address against the Summary of all your VM's, and that VMware Tools is OK
• VCB 1.5 should be installed and update to at least NetBackup 6.5.3 also.

Also here is a good resource for VMware Backups/VCB docs
http://www.vmware-land.com/Vmware_Links.html#Backups

VMware Proven Practice Doc
Integrating Netbackup 6.5.1 with VCB 1.1

[schrammd]

Yeah, we removed that stupid test4 VM and it appears to now work. The thing that puzzles me is why the vcbvmname util would only return that one entry and then die with no other warning.
Now that my list of VMs has populated into the VCB policy, I see that about 65 out of 70 only come in with IP's only and not any names. The VMware admin says most of them are in DNS but in a completely different DNS zone than my NetBackup servers. So, there is no (easy) way for the proxy or the master server for that matter, to do reverse lookup on these IP's. This kind of sucks just seeing a list of IPs for client names instead of host/VM names.

[Anonymous]

How about populating a local hosts file for those that are not in the zone of your NetBackup servers.
Bit of work but definitely neater than list of IP numbers.

[schrammd]

Yes, that worked and now all the VM's are resolvable. So, on to the next mystery: I go into the VCB policy, right click on the clients, New, then browse for the VMs. The entire list of VMs pop in eventually, and since we want to back them all up, I select them all. Clicking OK basically freezes the console so that nothing can happen after that point. Don't see any CPU or network traffic going on, but it never returns from that function. Man, this is worse than NB 4.0.
I found that if I only select a couple VM's it returns, but hey, I want it all.

[Pravs]

if I can understand your question correctly then you are encountering VM discovery performance with 6.5.3? Am I correct? how much time you waited before you concluded that nothing is working? Can you wait for approx 1 hour to see if it has not returned any thing? 70 VMs should not be a big deal.

[schmaustech]

Create the bpVMUtil log directory under C:\Program Files\VERITAS\Netbackup\logs on the proxy server.  You can see what it happening while the update is going on rather then watch an hour glass in the gui.

Regards,

Benjamin Schmaus

[schrammd]

I don't think it is "discovery" in the sense that i am not browsing for VMs, simply adding them into the policy after they have been discovered. Last night I selected 7 of them and after 30 minutes I locked my workstation and went home. Sometime after that they all popped in since they were added when I logged in today. In any event, what I think is important here is a that there "appears" to be a performance problem (there is no hourglass BTW) it simply hangs the console and goes to "not responding" which of course is an unacceptable condition unless you don't mind opening multiple consoles to get work done.

[Anonymous · Accepted solution]

Just posted in the Digest

Improvements in virtual machine discovery and vSphere support in NetBackup for VMware 6.5.3 and 6.5.4
http://seer.entsupport.symantec.com/docs/322769.htm

This is available via Engineering Binaries for 6.5.3 right now from Support

One of the main problems sighted with discovery is when a VM has multiple network interfaces.

This is actually relevant as we have a Production interface and a Backup interface on all our VM's, and I suspect is common that a VM has more than 1 NIC adapter

This binary may well go a long way to fixing this consistent issue.

[Karthikeyan_Sun]

Thats Great man! Thanks for that Document.