After updateing our customers BEX 2014 environment (physical an virtual) all jobs defined as RAWS with transport via LAN are running smoothly.
Accidently all jobs using the AVVI with vCenter are stopped and second the local Remote Agent for Windows running at the backup server is stopped at the moment tto snap the VMware VMDKs. The log said "communication error". The vCenter is reachable like before and running well. First error is V-79-57344-65304 followed by V-79-57344-3844 because the local agent is stopped.
What is the cause? Creating a new trusted link betwenn backup server and vCenter does not help.
We use Windows Server 2012 R2 and vSphere 5.5.
Is there an Event ID 1000 logged on the BE media server when the remote agent stops during the VM backup ?
Does restarting the remote agent service allow the backup to complete ?
Just to add to VJware's comment - a VMware agent backup heavily uses the Remote Agent (beremote.exe) process on the Backup Exec Server. As such troublshooting when and why the Remote Agent is stopping could help understand why you cannot backup VMware resources.
Hi VJware thanks for your reply!
Yes event id 1000 ist logged with a reference to ..\backup exec\VMware\VixDisklib\bin\libcurl.dll.
And NO - restarting the remote agent service does not help anyway. The job aborts suddenly. But other not-VAAI-based job remain running.
Full event log entry is:
[ Name] Application Error
- EventID 1000
[ Qualifiers] 0
[ SystemTime] 2015-04-18T21:02:16.000000000Z
C:\Program Files\Symantec\Backup Exec\beremote.exe
C:\Program Files\Symantec\Backup Exec\VMware\VixDiskLib\bin\libcurl.dll
Before we installed BE 15 the BE 2014 ran without errors.
No changes at vCenter level occurred since BE install (in-place migration).
I would be very grateful if you would share the final solution. I have having the exact same problem. I have logged a support case but would appreciate the extra feedback.
@JGrogan - If you have the same issue, i would strongly suggest to continue working with support. Having multiple cases of the same issue helps us to identify the issue quicker and have a fix as well.
Up to today the case is open and under investigation by Symantec.
Support first looks after a bunch full of unneeded tasks.
The question is. why is BE 15 not able to initiate a trusted communication
with our vCenter server after the update from 2014?
I found a workaround that may be helpful. Use at your own risk. We still have a BEX 2012 server in our environment so I grabbed the libcurl.dll from that server and replaced the one on the new server (after finding a safe place to keep a copy of the new DLL). My VMWare vmdk backups are now working.
I also still have a case open with support and hope they find a better solution. I think the reason for using the newer version is due to all the SSL vunerabilities found in the past year. Using this older DLL is a risk.
Best of luck!
Thanks for your update!
Yesterday I had the sane idea to change the DLL but due other impacts I were not able to realize it. At our customer location it´s the only BE server, but testing will be possible. In our case the problem is that we have backup our complete backupserver (with OS, BE, SQL, catalogs and 15TB dedup area) to realize an full backup yet. The painful story needs estimated 60h to get on LTO5. And meanwhile I allowed not to start any incremental backups, maybe they interfere or not.
Had the same problem after upgrading from BE 2014 to BE 15. Finally stumbled on this post over the weekend and pulled the libcurl.dll from a snapshot of my system before the upgraded and replaced the one in Symantec\Backup Exec\VMWare VixDiskLib\bin folder and my VMWare backups began to work correctly. One of my other sites upgraded just fine and are using the same version of VMWare 5.5 that we are using so not sure where the issue lies between VMWare and BE 15. I did log a support ticket.
Spent a couple of hours with support on this issue, gather logs, ip settings, agent settings, vmware settings etc. Finally they wanted me to test after upgrading the VMWare tools on a VM and I had to tell it wouldn't matter becasue the back up would fail. All of the VMWare backups fail as long as I have he new libcurl.dll in place. They still didn't get it. I had to show them on a ESXI host that backed up fine with the old libcurl.dll on Saturday that it would now fail with 0 bytes backed up and then I think the support person got the picture.
They wanted to look at the trees, I'm having problems with the forest. I will worry on the specfic trees after I get to the forest solved. I probably should have just asked to to have it escalated but since I have a work around by using the libcurl.dll from BE 2014 I know have a method for at least getting something on my tapes.
Supposed to get a call back this afternoon to see if they can move the call up the ladder.
This sounds similare to the following issue:
Please log a support call so Tech support can confirm that the issue you experience matches the issue as per above Article.
Hi John - When you speak to support please give them the article number provided by RogerR in this thread, as it sounds like the engineer you are working with is not aware of the known issue. He/she can then advance against the known issue.
I was on the phone with support for a couple of hours yesterday while they looked at a lot of jobs, settings, vmware info etc. But the support tech really wanted to look at why specific VM's weren't backing up and wanted me to make changes to try and get them to work. It took me a while to convince him that it didn't matter what changes I made to the VM because BE 15 fails to communicate correctly to the ESXI host and nothing would be backed up. But I made progress after showing the support tech that renaming the libcurl.dll to the 2014 version would allow the VMWare backups to run.
So I had an email this morning that they want to have upload the logs and will move the case forward.
I did also forward them the link above this morning.
Symantec SE installed the complete C:\Program Files\Symantec\Backup Exec\VMware\VixDiskLib\bin
directory with fresh files from the BE 15 installation disk.
Unfortunateley nothing happens. The error persists.
After changing back to the former LIBCURL.DLL backups ran again.
Meanwhile I´m a little impatient.
It is beyond my comprehension that Symantec has allowed the product to go out the door with such a major show stopper bug in it. And to post a tech note that a workaround is to run Remote Agent based backups of VMs is not really a work around since you cannot restore the entire VM and VM configuration with those backups. The entire point of AVVI is to be able to backup and restore VMs. So a major portion of the funcationality in BE 15 is broken.
When can customer's expect a fix?
In the meantime I am going to recommend to my customer that he roll back to BE 2014.
Up to now I got no real solution. Last action was to disable ipv6 traffic from an to backup server. This helps that I´m able to use the original DLL. But it´s only a workaroud. Many of us use the ipv6 protocol since introduction of windows server 2008. Symantec is investigating the problem together with VMware they told me.