Highlighted

Backup Exec 15 after Update no communication with vCenter - all AVVI jobs fail

After updateing our customers BEX 2014 environment (physical an virtual) all jobs defined as RAWS with transport via LAN are running smoothly.

Accidently all jobs using the AVVI with vCenter are stopped and second the local Remote Agent for Windows running at the backup server is stopped at the moment tto snap the VMware VMDKs. The log said "communication error". The vCenter is reachable like before and running well. First error is V-79-57344-65304 followed by V-79-57344-3844 because the local agent is stopped.

What is the cause? Creating a new trusted link betwenn backup server and vCenter does not help.

We use Windows Server 2012 R2 and vSphere 5.5.

20 Replies

Is there an Event ID 1000

Is there an Event ID 1000 logged on the BE media server when the remote agent stops during the VM backup ?

Does restarting the remote agent service allow the backup to complete ?

Just to add to VJware's

Just to add to VJware's comment -  a VMware agent backup heavily uses the Remote Agent (beremote.exe) process on the Backup Exec Server. As such troublshooting when and why the Remote Agent is stopping could help understand why you cannot backup VMware resources.

Hi VJware thanks for your

Hi VJware thanks for your reply!

Yes event id 1000 ist logged with a reference to ..\backup exec\VMware\VixDisklib\bin\libcurl.dll.

Application error.

And NO - restarting the remote agent service does not help anyway. The job aborts suddenly. But other not-VAAI-based job remain running.

Full event log entry is: -

Full event log entry is:

- System

  - Provider

   [ Name]  Application Error
 
  - EventID 1000

   [ Qualifiers]  0
 
   Level 2
 
   Task 100
 
   Keywords 0x80000000000000
 
  - TimeCreated

   [ SystemTime]  2015-04-18T21:02:16.000000000Z
 
   EventRecordID 51869
 
   Channel Application
 
   Computer A650WAP0002.hbalan.ffm
 
   Security
 

- EventData

   beremote.exe
   14.2.1180.0
   54f9310e
   libcurl.dll
   0.0.0.0
   4fb54c60
   c0000409
   0000000000048ff3
   db0
   01d07a150adcc248
   C:\Program Files\Symantec\Backup Exec\beremote.exe
   C:\Program Files\Symantec\Backup Exec\VMware\VixDiskLib\bin\libcurl.dll
   30f5d3ed-e60e-11e4-80cb-000af70f9366
   

Before we installed BE 15 the BE 2014 ran without errors.

No changes at vCenter level occurred since BE install (in-place migration).
   

This indicates a service

This indicates a service crash. Pls log a formal support case for us to analyze the crash dump.

I would be very grateful if

I would be very grateful if you would share the final solution. I have having the exact same problem. I have logged a support case but would appreciate the extra feedback.

@JGrogan - If you have the

@JGrogan - If you have the same issue, i would strongly suggest to continue working with support. Having multiple cases of the same issue helps us to identify the issue quicker and have a fix as well.

Up to today the case is open

Up to today the case is open and under investigation by Symantec.

Support first looks after a bunch full of unneeded tasks.

The question is. why is BE 15 not able to initiate a trusted communication
with our vCenter server after the update from 2014?

I found a workaround that may

I found a workaround that may be helpful. Use at your own risk. We still have a BEX 2012 server in our environment so I grabbed the libcurl.dll from that server and replaced the one on the new server (after finding a safe place to keep a copy of the new DLL). My VMWare vmdk backups are now working. 

I also still have a case open with support and hope they find a better solution. I think the reason for using the newer version is due to all the SSL vunerabilities found in the past year. Using this older DLL is a risk. 

Best of luck!

Thanks for your

Thanks for your update!

Yesterday I had the sane idea to change the DLL but due other impacts I were not able to realize it. At our customer location it´s the only BE server, but testing will be possible. In our case the problem is that we have backup our complete backupserver (with OS, BE, SQL, catalogs and 15TB dedup area) to realize an full backup yet. The painful story needs estimated 60h to get on LTO5. And meanwhile I allowed not to start any incremental backups, maybe they interfere or not.

Had the same problem after

Had the same problem after upgrading from BE 2014 to BE 15.   Finally stumbled on this post over the weekend and pulled the libcurl.dll from a snapshot of my system before the upgraded and replaced the one in Symantec\Backup Exec\VMWare VixDiskLib\bin folder and my VMWare backups began to work correctly.   One of my other sites upgraded just fine and are using the same version of VMWare 5.5 that we are using so not sure where the issue lies between VMWare and BE 15.   I did log a support ticket.

Spent a couple of hours with

Spent a couple of hours with support on this issue, gather logs, ip settings, agent settings, vmware settings etc.   Finally they wanted me to test after upgrading the VMWare tools on a VM and I had to tell it wouldn't matter becasue the back up would fail.   All of the VMWare backups fail as long as I have he new libcurl.dll in place.   They still didn't get it.   I had to show them on a ESXI host that backed up fine with the old libcurl.dll on Saturday that it would now fail with 0 bytes backed up and then I think the support person got the picture. 

They wanted to look at the trees, I'm having problems with the forest.  I will worry on the specfic trees after I get to the forest solved. I probably should have just asked to to have it escalated but since I have a work around by using the libcurl.dll from BE 2014 I know have a method for at least getting something on my tapes.

Supposed to get a call back this afternoon to see if they can move the call up the ladder.

 

Hi This sounds similare

Hi

This sounds similare to the following issue:

https://support.symantec.com/en_US/article.TECH230407.html

Please log a support call so Tech support can confirm that the issue you experience matches the issue as per above Article.

Thanks

Roger

Hi John - When you speak to

Hi John - When you speak to support please give them the article number provided by RogerR in this thread, as it sounds like the engineer you are working with  is not aware of the known issue. He/she can then advance against the known issue. 

Hi Roger, I was on the phone

Hi Roger,

I was on the phone with support for a couple of hours yesterday while they looked at a lot of jobs, settings, vmware info etc.   But the support tech really wanted to look at why specific VM's weren't backing up and wanted me to make changes to try and get them to work.   It took me a while to convince him that it didn't matter what changes I made to the VM because BE 15 fails to communicate correctly to the ESXI host and nothing would be backed up.  But I made progress after showing the support tech that renaming the libcurl.dll to the 2014 version would allow the VMWare backups to run.  

So I had an email this morning that they want to have upload the logs and will move the case forward.

I did also forward them the link above this morning.

Thanks

John

At this moment we try another

At this moment we try another webex. Later I will share my new informations.

Thanks

Andreas

Symantec SE installed the

Symantec SE installed the complete C:\Program Files\Symantec\Backup Exec\VMware\VixDiskLib\bin

directory with fresh files from the BE 15 installation disk.

Unfortunateley nothing happens. The error persists.

After changing back to the former LIBCURL.DLL backups ran again.

Meanwhile I´m a little impatient.

 

Andreas

 

It is beyond my comprehension

It is beyond my comprehension that Symantec has allowed the product to go out the door with such a major show stopper bug in it.  And to post a tech note that a workaround is to run Remote Agent based backups of VMs is not really a work around since you cannot restore the entire VM and VM configuration with those backups.  The entire point of AVVI is to be able to backup and restore VMs.  So a major portion of the funcationality in BE 15 is broken.

When can customer's expect a fix?

In the meantime I am going to recommend to my customer that he roll back to BE 2014.

Up to now I got no real

Up to now I got no real solution. Last action was to disable ipv6 traffic from an to backup server. This helps that I´m able to use the original DLL. But it´s only a workaroud. Many of us use the ipv6 protocol since introduction of windows server 2008. Symantec is investigating the problem together with VMware they told me.