**UPDATED** Quality Wins Every Time: vSphere 5.1 Support
Level 4 Drew_Meyer
Level 4
‎11-20-2012 01:26 PM
‎11-20-2012 01:26 PM

STATUS UPDATE 2/25/14: In case you somehow missed the news, on November 19, 2013 Backup Exec SP3 came out and includes support for vSphere 5.5.

 

STATUS UPDATE 4/7/13: Backup Exec 2010 and 2012 will both support vSphere 5.1 in their next service packs, due in July. Beta testing starts in May so read here for more details and a link to sign up.
 

STATUS UPDATE 1/17/13: We stated earlier that we had uncovered significant quality concerns when testing VMware’s vSphere 5.1 APIs with Symantec backup products. After close work together, VMware acknowledged these issues and announced the VDDK 5.0 Update 1 APIs.

Here’s the latest information for customers using or planning to use VMware vSphere 5.1:

  1. Backup Exec will support vSphere 5.1 in the patch releases for both BE 2010 and BE2012. More details and beta program registration are available here.
  2. NetBackup 7.5.0.4 is now qualified for use with vSphere 5.1. The details are available here.

We thank our friends at VMware for their efforts and appreciate your patience while we aligned on quality joint solutions.

 

ORIGINAL POST: We’ve seen backup complexity explode in the last few years, encompassing tape, disk, physical, virtual, de-duplication and cloud components. Data quantities are increasing exponentially and virtualization is expanding its footprint in all IT environments. Symantec is committed to making backup and recovery functions as reliable as possible during this environmental evolution.

In the last decade Symantec and VMware have jointly developed solutions for the most advanced and complex virtual environments in the world. We are humbled and grateful that customers continue to choose NetBackup or Backup Exec more often with VMware than any other option.

Symantec and VMware are unwaveringly dedicated to quality in our jointly developed solutions. During our testing with the VMware vStorage 5.1 API Symantec encountered issues* that introduce risk performing consistent backups and ensuring reliable restores. We raised these concerns with our friends at VMware, who documented them in their knowledge base and in the VMware vSphere 5.1 release notes.  VMware expects a vStorage API update, and at that time we expect to retest and once again support joint solutions.

During these weeks of close examination VMware also confirmed that any 3rd party vendor depending on the current API may be impacted by these issues. We can only conclude that compatibility and support claims from other vendors are either poorly informed or possibly represent a more cavalier “ship-and-fix” data protection approach than Symantec is willing to accept. We welcome your questions and comments.

Sincerely,

The Symantec Backup Exec and NetBackup teams

@backupexec
@netbackup
@vmware

*Three specific issues are called out in the VMware vSphere 5.1 release notes:

  • Occasional segmentation-violation crashes during library disconnect
  • Hang in connect or cleanup due to intermittent race condition
  • Disk Open fails without an SSL thumbprint from vCenter

 

 

98 Comments
Moderator
Moderator

Nice one...you need to get this pinned to both forums for better visibility. A number of forumites have asked when Symantec will support VMware vSphere 5.1, and your update here will definitely give them the information they're looking for! yes

Level 4 Drew_Meyer
Level 4

Pinned! Thanks for the help Craig.

Level 3 MMoret
Level 3

Hi,

Good to see the clear communication about this.

I spoke to another backup vendor who stated:
We use the vddk 5.0U1 to backup vSphere 5.1 as it does not state in the release notes of this vddk that 5.1 is not supported (it does not state that it is supported as well...).

I have tested this setup and did not ran into any problems.

Can you comment on this setup?

Regards,
Martijn Moret

Level 4 w-d
Level 4

Thank you for sharing this important information

Level 3 Tallwood6
Level 3
Great to get some clarification quite why it took this long though. I don't suppose vmware have indicated roughly how far off this vstorage api update is?
Level 3 BigTimeOne
Level 3

If we use vSphere 5.1 with NetBackup 7.1.0.4.

Should we still be OK since NetBackup will only use VDDK 5.0 and not 5.1?

Level 3 MMoret
Level 3

From what I heard @ Symantec Vision, vSphere 5.1 is completely not supported.
You should be ok, but do not count on getting support!

Level 3 Tallwood6
Level 3

Anyone from symantec care to comment on the following, aside from all the marketing rubbish some valid points are raised?

http://www.veeam.com/blog/symantec-admits-all-its-vmware-backup-customers-are-in-danger.html

Why do you constantly change this post? Keep your story or don't publish!

First you stated:

VMware expects a vStorage API update during the first calendar quarter of 2013, and at that time we expect to retest and once again support joint solutions.

Now that's changed. Q1 2013 is too late for your customers?

And you say in the first version:

During these weeks of close examination VMware also confirmed that any vendor depending on the current API cannot perform consistent backups and cannot ensure a reliable recovery point.

And now they "may be impacted"?

If this is this the quality you talk about....

Level 3 SeanRegan
Level 3

It is not appropriate for Symantec state or confirm shipment dates for VMware products. 

Level 3 SeanRegan
Level 3

Here is more information about VMware ready status for those unfamiliar with it. http://www.vmware.com/partners/programs/vmware-ready/application_software.html

Key Criteria: 

  • Software vendors that fully support their products on VMware virtualization solutions can qualify for the VMware Ready logo. 
  • VMware Ready logo visibly demonstrates confidence and simplifies purchase decisions for any customer using a virtualized infrastructure. 
  • Demonstrate successful operation by passing a workload test against a virtual configuration of the application and submitting the results to VMware
Level 3 Tallwood6
Level 3
This blog entry has been completely reworded! The fact of the matter is there are solutions now offering vsphere 5.1 support using both vddk 5 & 5.1 providing RELIABLE backup and restores by simply implementing what sound to be sensible safeguards. You can flash round vmware ready logos all you like fact of the matter is symantec looks like it isn't ready for very much comming out of vmware at the moment. Customers don't want excuses and logos they want commitments, timescales and solutions none of which are being provided currently.

Ahh well...Ask vRanger customers the value of VMware Ready certification...

Quest announced they had achieved VMware Ready for vSphere 5 with the vRanger 5.2 release. But customers realized after the upgrade that, despite being vSphere 5 certified, they still did not support VMFS5... Ask VMware how they granted vSphere 5 ready certification to the solution which does not support one of the biggest features of this release...

Level 6 Dushan_Gomez
Level 6

So when this vSphere 5.1 will be supported ?

http://www.veeam.com/blog/symantec-admits-all-its-vmware-backup-customers-are-in-danger.html

Veeam says that Symantec is misleading us ?

Level 3 BigTimeOne
Level 3

It looks like VDDK 5.0 has similar or same issues so how can Symantec claim support for vSphere 5.0 and not 5.1?

Also isnt Symantec part of the vmware Beta Test teams. Why was this not fixed for the GA of 5.1?

We are backing up vSphere 5.1 with NetBackup 7.1.0.4 so far no issues.

Level 3 SeanRegan
Level 3

This does not effect older versions of Symantec's VMware support. It only impacts 5.1 because of changes made specifically to 5.1. When the fix is available from VMware and we can code to it and QA it we will release support.  

There are issues in all 3rd party APIs and development kits. The product team writing to the API needs to make a quality call.  Sometimes those issues create situations that will negatively impact backup, sometimes they do not. If they do, we won't ship until they are fixed. 

Out blog made no mention of specific vendors other than to state that all vendors get the same API to work with. It is up to the vendor to make a decision how they want to handle this issue.  From a marketing standpoint it is easy to ship and fix. But, from a quality and reliability standpoint it is not the right decision. 

 

Here is a list of the 23 vendors who have received VMware VMReady designations from VMware.The vendor mentioned previously is not on this list. https://solutionexchange.vmware.com/store/categories/backup-and-availability

There are ways to rush support and do workarounds but they do not lend themselves to long term reliability and quality. Our position as noted above is to wait for the 5.1 VDDK updates from VMware and then ship support. Anthing else requires a quality compromise or workaround with the API that we don't believe is in the best interest of customers. 

(edit for spelling) 

 

 

 

 

 

 

 

Level 6 John_Santana
Level 6

Spot on :-)

Level 6 AbdulRasheed
Level 6

 

Hi Martijn,

  RE: We use the vddk 5.0U1 to backup vSphere 5.1 as it does not state in the release notes of this vddk that 5.1 is not supported 

  That is quite an amusing statement from a vendor! Why would VMware go back to release notes of an older version of VDDK and explicitly state that newer versions of vSphere will not be supported? Isn't that like expecting Ford to document that strut mount from 2012 model may not fit on 2011 model especially when 2012 had undergone a refresh? It is sad that a vendor would take an approach like this and put data at risk. 

  Having said that we had asked the same question to VMware whether we can use older version of APIs with vSphere 5.1. VMware had not tested that configuration and they were reluctant to recommend it. After realizing the impact for 5.1 customers with updated APIs, they have finally agreed to look into that option. Please stay tuned. We are working closely with VMware so as to provided enterprise grade protection for customers wanting to upgrade to vSphere 5.1. 

Level 6 AbdulRasheed
Level 6

 

Hi BigTimeOne, 

   It is recommended to wait until we sort this out with VMware. In general VMware tests current version of APIs with current version of vSphere and its predecessors just like any other software developer. This is the classic backward compatibility assurance approach. What you are proposing is forward compatibility (5.0 versions of API to work with 5.1 version of vSphere)? We had asked the very same question to VMware and originally they were reluctant to test and certify forward compatibility (which is understandable). However Symantec is indeed working with VMware to see if forward compatibility is a possibility to help customers upgrade to vSphere 5.1. Please stay tuned. 

Level 6 AbdulRasheed
Level 6

 

Hi Tallwood6,

   I work for Symantec. However opinions from me (other than those where I post as an author for Symantec official blogs) need not represent those of my employer. 

   First of all, no vendor was explicitly called out in Drew's blog. He was simply relaying what VMware had also posted in their own knowledge-base. 

   Secondly, to honor your request, I am not commenting on the 'marketing rubbish' part. 

   Finally, let us get to the real meat of what you were asking for. 

   1. There is an argument on Symantec said "will cause failures" and VMware said "may cause failures". In my opinion, that argument is plain silly. Whether it is a "may" or "will", you cannot take a chance when it comes to enterprise data protection. I have worked here in Symantec for years. Even when there is tiny fraction of a chance for data protection risk,  Symantec issues TechAlerts (Product Alerts) instead of silently fixing the issue. You have access to this information on external facing support website. Most of the competitors need you to have a specific support contract and login to access such type of vital information. This level of transparency with customers and prospects had always been a strength for Symantec. 

   2. Technical explanation: It resolved a mystery for me. I had always wondered why a VMware backup product does not get VMware Ready certification. I have seen some scripted and unconvincing responses in the past on that topic. I have direct experience in working with VMware on getting VMware Ready certification. Among many things, one thing that they would test is to make sure that callers of the API play by rules. This included port validations, SOAP call styles and many more. The goal is to ensure the product/solution in question is conforming to APIs as VMware is putting its reputation and brand behind that integration. Of course, a vendor can certainly workaround it, then the question is…

     As we do not live in a perfect world, software defects and edge cases are likely to happen. The so called claim for "immunity" is a bit premature. Let us give the benefit of doubt and assume that a solution is immune to issues in APIs because of some wrapped code. Any software developer will admit that bugs can creep in any part of the code. There is no guarantee or immunity if data loss occurs on account of the wrapper code. 

   In the unfortunate event that you did encounter a data loss issue; would you rather stand behind VMware and data protection leader like Symantec, or take the word of someone who had provided a workaround? Please don't get me wrong. I have huge respect for engineers in Symantec and other vendors who think outside the box and come up with innovative ways to solve customers' problems. What I am thinking about is the confidence in deploying a solution that is endorsed by two major brands. 

Once again, these are my own opinions! 

Level 6 AbdulRasheed
Level 6

 

Hi CvBarney, 

   Sorry to hear about vRanger customers. As I have direct experience in getting a product VMware Ready certified, I will add my 2 cents. 

   When a product undergoes tests for VMware Ready certification, among many things, what VMware is looking to validate is whether the product plays by the rules of APIs provided by VMware. This is how they validate deeper integration with vSphere. They take this very seriously as they are putting their reputation and brand behind the integrated solution. 

   VMware Ready certification is not necessarily an indication that VMware tested all aspects of the product. It looks like vRanger is indeed playing by VADP/VDDK rules and that could be how they received the certification. However, I do agree that the lack of VMFS5 support is a major hole in that story! 

Level 6 AbdulRasheed
Level 6

Hi Tallwood6,

  I had responded to one of your earlier posts. Hope that helps to clarify a few things. 

Level 6 AbdulRasheed
Level 6

Hi Dushan,

   Tallwood6 had a similar question on the blog you had referenced. Kindly see my response there. Hope that helps. 

   We are working with VMware in getting you enterprise class protection for vSphere 5.1. As Sean mentioned, we are unable to give you a solid date at this time as we are not allowed to talk about VMware's plans. 

   

Level 6 AbdulRasheed
Level 6

Hi BigTimeOne,

  Sean had already answered this question in an earlier comment. Let me copy-paste part of his comment. 

This does not effect older versions of Symantec's VMware support. It only impacts 5.1 because of changes made specifically to 5.1. When the fix is available from VMware and we can code to it and QA it we will release support.  

 

There are issues in all 3rd party APIs and development kits. The product team writing to the API needs to make a quality call.  Sometimes those issues create situations that will negatively impact backup, sometimes they do not. If they do, we won't ship until they are fixed. 

Level 4 BackupMikko
Level 4

We have been working closely with VMware ever since the beginning of the beta-process in addressing the issues that have been preventing us from providing vSphere 5.1 support.

VMware is publishing release notes for each VDDK version and updating those as needed but these notes only cover supported platforms, compatibility notices, resolved issues and known issues and workarounds. There are no details on component/design changes on the release notes.

VDDK Release Notes can be found here and SDK/documentation can be found here.

Warm Regards,

Mikko

@BackupMikko

Level 3 Tallwood6
Level 3

So we've now got to wait until:

  1. VMWARE updates the API (no timescale)
  2. Wait for symantec to test the new release (taken 2 months to test current release)
  3. Code and test new release

So we are looking at possibly 3-6 months before its supported... give it a couple more months and the next version of vsphere could be announced!

You are giving your customers 2 options at the moment, run backup exec on vsphere 5.1 unsupported or use a different product because know one who requires vsphere 5.1 is going to wait that long.  

Level 6 AbdulRasheed
Level 6

Hi Tallwood6, 

  I don't mean to sound like a broken record; but we are indeed working with VMware for a short term solution as well. Remember that other vendors who currently claim support had either shipped with back level version of APIs or had built some hacks. Both approaches are not endorsed by VMware. It just happened that Symantec came out and publicly stated the risks and decided to stand behind quality instead of 'time-to-market'. On a long run, this is what had made customers trust Symantec's offerings. I understand that there are claims in the market stating that Symantec is delaying things. The reality is that Symantec had been working with VMware even before vSphere 5.1 was beta and numerous points in release notes were based on tests Symantec had done. 

  Unfortunately, I am not able to give you a time line. We will make sure that both short term and long term solutions would have endorsement from VMware (unlike the method adopted by some other vendors). 

 

 

Level 3 Tallwood6
Level 3

A short term solution would be great! At the moment it just looks like you've kicked the ball straight into VMware’s court.

I guess what frustrates me is a decision seems to have been made by developers / quality assurance without seeming to take the immediate needs of customers into consideration. Yes backup and restore reliability is paramount but a balance needs to be struck but something looks out of kilter when quite a number of your competitors have clearly looked at the issues a decided that working round the issues and providing support for the platform is more important that burying heads in the sand. After all customers using vsphere 5.1 have little choice than to use the product unsupported at the moment which has got to be worse possible situation all round.

Level 3 BigTimeOne
Level 3

AbdulRasheed please keep us posted on backing up vSphere 5.1 with NetBackup 7.1.0.4 using VDDK 5.0 API instead of VDDK 5.1

We have been backing up hundreds of VM with this combo and no issues so far.

Level 3 BigTimeOne
Level 3

If working closely with VMware why release this news 2+ months after GA of 5.1?

Basically vSphere 5.1 can not be used since you are saying no vendor supports it.

 

Level 3 BigTimeOne
Level 3

Abdul, please keep us posted if we should be OK using NetBackup 7.1.0.4 VDDK 5.0 on vSphere 5.1.

We have been backing up hundreds of VMs with the combo above and so far no issues.

Level 3 Tallwood6
Level 3
There are vendors that support it, including one vendor on that list of "vmware ready" solutions providers.
Level 3 BigTimeOne
Level 3

According Symantec that is not possible since all use the same API. So who is telling the truth?

Level 3 BigTimeOne
Level 3

Thats backward compatibility. We are using an older VDDK 5.0 client ie. NetBackup 7.1.0.4 to connect to newer VDDK 5.1 vSphere 5.1 Servers.

Same as Symantec Endpoint Client 11.5 reporting/syncing with SEPM 12.1 Server

Level 3 SeanRegan
Level 3

Welcome to the forum. This point was raised in the original post. VMware has identified issues with the 5.1 VDDK. Those issues impact the ablity of 3rd party backup vendors to perform backups for 5.1.  All vendors get the same VDDK/API. 

It is possible to claim support.  However, given the 5.1 VDDK issue, claims of support necessate a compromise in quality/reliability or a workaround by the backup vendor. Our engineering team does not feel these solutions are appropriate and we have opted for the choice outlined in the original post.  

We will ship support for VMware 5.1 when we receive the updated VDDK from VMWare and we are able to code, QA and release a high quality of support. 

 

Level 3 BigTimeOne
Level 3

So between their update release schedule, your testing and you releasing an update that coincides with your release schedule.

We are realistically looking at 6-8 months.

 

Level 3 BigTimeOne
Level 3

6-8 months between all their releases and testing.

Thank you for posting this.  I'll hold off upgrading to ESXi 5.1 until I see it on the Backup Exec SCL.

Level 3 Paul_VanDyke
Level 3

I too am waiting for Symantec to have full vSphere 5.1 support...  However, I'm also re-thinking my backup strategies for my VMs.  I've started to migrate back to agent-based backups and away from VM-based backups.  I have been unhappy with the quality of the VM-based backups I've been doing.

If I continue to be successful moving my backup environment back to agent-based backups, I may go ahead and upgrade to vSphere 5.1 and forget about VM-based backups.

Comments?

Level 3 MMoret
Level 3

Paul VanDyke,

I am very curious what issue's made you rethink your backup strategy?

Regards
Martijn Moret

Level 3 Tallwood6
Level 3

AbdulRasheed was actually making some progress convincing us that symantec was proactively looking at something which didnt involve waiting many months before you joined this converstaion.

Again a compromise that most seem to be making and one that still seems to be winning customers and awards in one case. But thats fine symantec clearly no whats best for it customers we will just all skip vsphere 5.1!

 

Level 6 AbdulRasheed
Level 6

VMware disagrees with you (as of today). 

 

VDDK 5.0

http://www.vmware.com/support/developer/vddk/VDDK-500-ReleaseNotes.html 

 

VDDK 5.0 adds support for ESXi 5.0 and vCenter Server 5.0, and now works with the following VMware platform products:

  • ESX/ESXi versions: 5.0, 4.1, 4.0, and 3.5
  • vCenter Server versions: 5.0, 4.1, and 4.0 managing ESX/ESXi 3.5 and later
  • VirtualCenter version: 2.5 managing ESX/ESXi 3.5

(Note: 5.0U1 is a bug-fix release, no new support was added) 

 

VDDK 5.1

http://www.vmware.com/support/developer/vddk/VDDK-510-ReleaseNotes.html

 

The Virtual Disk Development Kit (VDDK) 5.1 is an update to support vSphere 5.1 and to resolve issues discovered in previous releases. VDDK 5.1 adds support for ESXi 5.1 and vCenter Server 5.1, and was tested with the following VMware platform products:

  • ESXi 5.1, ESXi 5.0, and ESX/ESXi 4.1
  • vCenter Server 5.1 and vCenter Server 5.0 managing ESX/ESXi 4.1 and later

 

In other words, VMware's approach in backward compatibility is that **newer versions of VDDK** will have support for **older versions of vSphere**. 

 

Hope this helps. 

Level 6 AbdulRasheed
Level 6

You are right. But I have also mentioned that it is pending endorsement from VMware. As Sean had mentioned, quality and reliability is of utmost importance for a data protection product and I am sure you guys would appreciate that approach. Hacking or working around API issues opens the door for long term problems. We stand behind joint customers of VMware and Symantec like yourself; we will not ship something without putting it to heavy QA. We will keep you posted on the viability of short term fix. I am unable to release more information at this time. 

As always, please do note that none of forward looking statements should be the basis for your planning as they are subject to change. (It is a standard disclaimer I have to provide). 

 

Level 6 AbdulRasheed
Level 6

Hi BigTimeOne, 

  I had responded to one of your earlier questions about VDDK versions. As of today, VMware does not officially support VDDK 5.0 on vSphere 5.1. The release notes are here: http://www.vmware.com/support/developer/vddk/VDDK-500-ReleaseNotes.html 

  Even the recent update to 5.0 was a bug-fix release, not a proliferation release to vSphere 5.1. I have learned that there are a few vendors who announced support for vSphere 5.1 using this unsupported version of VDDK. That is unfortunate as they are putting their customers data at risk. We are working with VMware on a few possiblities. We will keep you posted. 

 

 

 

 

Level 6 AbdulRasheed
Level 6

Hi Paul,

   As you might have already figured out from the post and various comments, we are actively working with VMware to get you support for vSphere 5.1. I am not sure what you mean by 'the quality of VM based backups". Have you been working with Technical Support? Send me a private note with your case number and I am happy to talk to Support team about it. 

   One of the key advantages of a unified backup solution like NetBackup or Backup Exec is indeed the possibility to protect virtual machine data in multiple ways. My personal preference is to use VADP based backups of VMware. But you do have the choice to run agent based backups as well especially in the situations like vSphere 5.1 where there are issues with the APIs at this time. You have the choice to upgrade now to 5.1 and use agent based backups. Flipping to VADP backups once support is announced is quite easy! 

 

 

Level 6 AbdulRasheed
Level 6

Hi Thundercleese, 

  We appreciate your patience. We are actively working with VMware on this matter. 

Level 3 BigTimeOne
Level 3

That is only because when VDDK 5.0 release notes were created vSphere 5.1 did not exist.

Level 3 Paul_VanDyke
Level 3

Martijn and Abdul,

A couple thoughts about VM backups:

I cannot do source-side deduplication with VM backups.  I need to run the remote agent to do source-side dedup.  I can usually get > 3,000MB/min or greater with source-side dedup.  I haven't gotten over 2,000MB/min with VM backups.  I am using NBD, not SAN as my VM transport mechanism.  I have not had the time to configure my media server to share the VMWare LUNs yet to enable SAN transport.  However, with source-side dedup, I don't have to move as much data across the wire.  I have a well-resouced VMWare cluster and can afford the extra CPU cycles in the VM guest machines.

Also, I have really enjoyed the SDR recovery methods and feel really good about the integrity of my data with an agent-based backup and SDR recovery.  I'm not sure that I feel as good using the "crash-consitent" state of a VM-based restore.

Application backups still need to be done outside the VM to get the desired results (i.e. flushing the logs on Exchange server, SQL)

Please help me understand the benefits of VM-based backups vs. agent-based backups.

Thanks,

Paul

Level 3 Paul_VanDyke
Level 3

Another reason I've shy'ed away from VM based backups is the problem on this thread: https://www-secure.symantec.com/connect/forums/be2012-beremoteexe-crashes-vmware-backups

If my VM based backup hangs, it kills the beremote.exe service on the media server and then none of my other backups will run.  I have a busy backup schedule with multiple backup jobs running at night and duplicate to tape running during the day.

Thanks,

Paul

Level 3 MMoret
Level 3

With NetBackup 7.5, you can go past the application backup for Exchange, SQL and SharePoint.
The source/client side DeDup is a great point!

One of the advantages of Vm based backup is with the right policies, you can never forget to arrange backups. They are automatically added to that policy. 

I also find the combination of DR (full vm restore) and file and application level restore a great feature.

But again, there is no right or wrong, just choices.

I am curious to the speed you would achieve with San transport mode ;-)

Regards
Martijn