04-21-2015 10:10 PM
After updateing our customers BEX 2014 environment (physical an virtual) all jobs defined as RAWS with transport via LAN are running smoothly.
Accidently all jobs using the AVVI with vCenter are stopped and second the local Remote Agent for Windows running at the backup server is stopped at the moment tto snap the VMware VMDKs. The log said "communication error". The vCenter is reachable like before and running well. First error is V-79-57344-65304 followed by V-79-57344-3844 because the local agent is stopped.
What is the cause? Creating a new trusted link betwenn backup server and vCenter does not help.
We use Windows Server 2012 R2 and vSphere 5.5.
04-21-2015 10:37 PM
Is there an Event ID 1000 logged on the BE media server when the remote agent stops during the VM backup ?
Does restarting the remote agent service allow the backup to complete ?
04-21-2015 10:40 PM
Just to add to VJware's comment - a VMware agent backup heavily uses the Remote Agent (beremote.exe) process on the Backup Exec Server. As such troublshooting when and why the Remote Agent is stopping could help understand why you cannot backup VMware resources.
04-22-2015 08:37 AM
Hi VJware thanks for your reply!
Yes event id 1000 ist logged with a reference to ..\backup exec\VMware\VixDisklib\bin\libcurl.dll.
Application error.
And NO - restarting the remote agent service does not help anyway. The job aborts suddenly. But other not-VAAI-based job remain running.
04-22-2015 08:40 AM
Full event log entry is:
- System
- Provider
[ Name] Application Error
- EventID 1000
[ Qualifiers] 0
Level 2
Task 100
Keywords 0x80000000000000
- TimeCreated
[ SystemTime] 2015-04-18T21:02:16.000000000Z
EventRecordID 51869
Channel Application
Computer A650WAP0002.hbalan.ffm
Security
- EventData
beremote.exe
14.2.1180.0
54f9310e
libcurl.dll
0.0.0.0
4fb54c60
c0000409
0000000000048ff3
db0
01d07a150adcc248
C:\Program Files\Symantec\Backup Exec\beremote.exe
C:\Program Files\Symantec\Backup Exec\VMware\VixDiskLib\bin\libcurl.dll
30f5d3ed-e60e-11e4-80cb-000af70f9366
Before we installed BE 15 the BE 2014 ran without errors.
No changes at vCenter level occurred since BE install (in-place migration).
04-22-2015 10:14 AM
This indicates a service crash. Pls log a formal support case for us to analyze the crash dump.
05-01-2015 12:58 PM
I would be very grateful if you would share the final solution. I have having the exact same problem. I have logged a support case but would appreciate the extra feedback.
05-02-2015 12:29 AM
@JGrogan - If you have the same issue, i would strongly suggest to continue working with support. Having multiple cases of the same issue helps us to identify the issue quicker and have a fix as well.
05-05-2015 03:40 AM
Up to today the case is open and under investigation by Symantec.
Support first looks after a bunch full of unneeded tasks.
The question is. why is BE 15 not able to initiate a trusted communication
with our vCenter server after the update from 2014?
05-06-2015 07:04 AM
I found a workaround that may be helpful. Use at your own risk. We still have a BEX 2012 server in our environment so I grabbed the libcurl.dll from that server and replaced the one on the new server (after finding a safe place to keep a copy of the new DLL). My VMWare vmdk backups are now working.
I also still have a case open with support and hope they find a better solution. I think the reason for using the newer version is due to all the SSL vunerabilities found in the past year. Using this older DLL is a risk.
Best of luck!
05-06-2015 07:45 AM
Thanks for your update!
Yesterday I had the sane idea to change the DLL but due other impacts I were not able to realize it. At our customer location it´s the only BE server, but testing will be possible. In our case the problem is that we have backup our complete backupserver (with OS, BE, SQL, catalogs and 15TB dedup area) to realize an full backup yet. The painful story needs estimated 60h to get on LTO5. And meanwhile I allowed not to start any incremental backups, maybe they interfere or not.
05-11-2015 10:44 AM
Had the same problem after upgrading from BE 2014 to BE 15. Finally stumbled on this post over the weekend and pulled the libcurl.dll from a snapshot of my system before the upgraded and replaced the one in Symantec\Backup Exec\VMWare VixDiskLib\bin folder and my VMWare backups began to work correctly. One of my other sites upgraded just fine and are using the same version of VMWare 5.5 that we are using so not sure where the issue lies between VMWare and BE 15. I did log a support ticket.
05-14-2015 10:55 AM
Spent a couple of hours with support on this issue, gather logs, ip settings, agent settings, vmware settings etc. Finally they wanted me to test after upgrading the VMWare tools on a VM and I had to tell it wouldn't matter becasue the back up would fail. All of the VMWare backups fail as long as I have he new libcurl.dll in place. They still didn't get it. I had to show them on a ESXI host that backed up fine with the old libcurl.dll on Saturday that it would now fail with 0 bytes backed up and then I think the support person got the picture.
They wanted to look at the trees, I'm having problems with the forest. I will worry on the specfic trees after I get to the forest solved. I probably should have just asked to to have it escalated but since I have a work around by using the libcurl.dll from BE 2014 I know have a method for at least getting something on my tapes.
Supposed to get a call back this afternoon to see if they can move the call up the ladder.
05-14-2015 11:36 PM
Hi
This sounds similare to the following issue:
https://support.symantec.com/en_US/article.TECH230407.html
Please log a support call so Tech support can confirm that the issue you experience matches the issue as per above Article.
Thanks
Roger
05-15-2015 12:14 AM
Hi John - When you speak to support please give them the article number provided by RogerR in this thread, as it sounds like the engineer you are working with is not aware of the known issue. He/she can then advance against the known issue.
05-15-2015 07:02 AM
Hi Roger,
I was on the phone with support for a couple of hours yesterday while they looked at a lot of jobs, settings, vmware info etc. But the support tech really wanted to look at why specific VM's weren't backing up and wanted me to make changes to try and get them to work. It took me a while to convince him that it didn't matter what changes I made to the VM because BE 15 fails to communicate correctly to the ESXI host and nothing would be backed up. But I made progress after showing the support tech that renaming the libcurl.dll to the 2014 version would allow the VMWare backups to run.
So I had an email this morning that they want to have upload the logs and will move the case forward.
I did also forward them the link above this morning.
Thanks
John
05-19-2015 01:26 AM
At this moment we try another webex. Later I will share my new informations.
Thanks
Andreas
05-19-2015 05:16 AM
Symantec SE installed the complete C:\Program Files\Symantec\Backup Exec\VMware\VixDiskLib\bin
directory with fresh files from the BE 15 installation disk.
Unfortunateley nothing happens. The error persists.
After changing back to the former LIBCURL.DLL backups ran again.
Meanwhile I´m a little impatient.
Andreas
06-29-2015 04:29 PM
It is beyond my comprehension that Symantec has allowed the product to go out the door with such a major show stopper bug in it. And to post a tech note that a workaround is to run Remote Agent based backups of VMs is not really a work around since you cannot restore the entire VM and VM configuration with those backups. The entire point of AVVI is to be able to backup and restore VMs. So a major portion of the funcationality in BE 15 is broken.
When can customer's expect a fix?
In the meantime I am going to recommend to my customer that he roll back to BE 2014.
08-03-2015 11:19 PM
Up to now I got no real solution. Last action was to disable ipv6 traffic from an to backup server. This helps that I´m able to use the original DLL. But it´s only a workaroud. Many of us use the ipv6 protocol since introduction of windows server 2008. Symantec is investigating the problem together with VMware they told me.