Another Backup Exec Experience

[Paul_VanDyke]

I would like to share my personal experience with Backup Exec with the hope for positive change.  I have been a Backup Exec user since version 6 when the product was owned by Seagate.  I have been in the IT industry for over 20 years.  I have been responsible for running backups for 16 years.  As a network administrator, I took great pride in maintaining all aspects of the network, including my disaster recovery plan.  I did a full datacenter restore from tape once in my carreer, and I would like to never have to do that again. The restore was scheduled over a weekend and it took five days.  That experience taught me that you can never plan enough and that good backups can make or break a career.

And that is why I feel so compelled to write this.  As the internet and the dot com trend hit the industry, my data center began to grow and with it, so did my backups.  I had upgraded from my Exabyte M2 tape drive to an ExaByte VXA 10 Tape Library, but my full backups were still taking 7 tapes.  The VXA library proved to be a little unreliable and tapes would fail in the middle of a full backup.  This would cause my full backups to fail each weekend.  After several weekends of failed backups, I had to find another solution.  I purchased a 2TB Lacie Etherdisk and started writing my full backups to disk first, then duplicating them to tape.  This D2D2T process allowed me to troubleshoot tape library issues during the week and still get my full backups on the weekends that I desired.  This solution worked until virtualization crept into my datacenter and I experienced server sprawl.  I tried to maintain my existing backup schedule, but I was quickly overwhelmed.  My full backups were taking 48 hours and running all weekend.  I was stuck in a "traditional" backup schedule:  full backups on the weekend and incrementals during the week.  These jobs had all my servers in their selection list.

To combat the time that backups were taking, I decided to purchase the VMWare agent so that I could snapshot and backup my virtual machines and still give me some time on the weekends to do server maintenance (Windows updates).  I thought I was in paradise, until I tried to restore a virtual machine from backup.  It failed miserably.  Backups would run at "full" speed, but restores would drag at almost 1/100th the speed.  Completely unacceptable!  This was the first hole in the "armor" that is supposed to be protecting my datacenter.  I gave up on running VM-based backups in Backup Exec.  At the same time, other vendors were courting me with their backup solutions.  I talked with other IT admins and most of them were using VM-based backup software products (Veeam, vRanger).  My disk target (2TB Lacie Etherdisk) was filling up and was no longer big enough to store more than one full backup.

At this time a new crop of backup software packages were emerging.  The VM-based backup products had built-in deduplication and change-block-tracking.  The other traditional file based backup packages were able to deduplicate and run synthetic full backups.  These technologies reduce the amount of data needed to backup and store while providing multiple restore points.  My backup software was old, didn't adapt well to new technologies and was under-resourced.  I needed a change.  I decided to leverage my Microsoft Technet subscription and try out Microsoft's Data Protection Manager.  This provided the required file-level backups of my servers, de-deduplication and synthetic fulls.  A synthetic full backup takes one full backup and then appends incremental backups.  After the initial full-backup, each incremental is done quickly, reducing the backup window.  Full restores can be made from the last incremental.  I also adopted VMWare's VDR appliance to provide a VM-based backup of my virtual machines.  I configured four VDR appliances in my environment, running backups of my VMs.  I had the best hybrid solution.  My file-level data was backed up to disk and then written to tape.  I also had VM-based backups for full machine restores.

The experience I created with MS DPM and VMWare's VDR taught me that I could no longer schedule full backups just on the weekend.  I had to break my backup jobs into logical units and schedule them over the full week.  I had too much data to move it from production to backup just on the weekends.  I also needed my weekends for infrastucture upgrade and Windows updates.

Thru all of this tranistion, I still had maintained my Backup Exec licenses.  When I heard of the Backup Exec 2012 Beta, I was couriously interested.  I heard that Symantec was trying to change and re-invent itself.  I had some spare hardware that I could dedicate to the beta and decided to pursue it.  I signed-up, downloaded and installed the beta.  I actively commented in the beta forums and tried to provide valuable input.  I talked with folks on the beta team and thought they were doing a great job with the product.  I fully bought into the server-focused backup model.  I was a little surprised to see the product go Gold and ship.  I felt like there was still some polishing that was missing.  I understand that not all bugs are going to be caught and fixed before shipping a product, but there were some glaring flaws that were overlooked.  The user-interface is a complete redesign and I belive it's a step in the right direction.  However, Symantec undermines its efforts when the columns won't sort.  This simple user-interface flaw creates a hardship for Backup administrators who use the product.  Admins have to search for another way to get the information they need.  The product was not ready to ship when it did.  Symantec is forced to address it's users on two front, the server-centric backup model and the sub-par user-interface.  I believe these two issues distract Symantec's resources from actually fixing other bugs related to the functionality of the product.

With the new version of Backup Exec, I hoped to fully embrace virtual machine based backups.  I even used the phyical backup to virtual restore option with the beta to move a phyical machine into my virtual environment.  I quickly became disappointed with the VMWare agent when the remote agent on the media server crashed randomly once a week during VM-based backups.  When the remote agent crashes, no more backups run.  I found myself baby-sitting my backup server daily.  Calls to tech support were met with running VxGather and no follow-up.  Even VxGather, Symantec's support tool is broken; it cannot upload via FTP even though the first level support techs insist that you submit the logs via the tool.  I gave up on VM-based backups and now backup my virtual machines with the agent.  This provided me with more options and I thought that would be great.  I think the best thing I gained by switching off the VMWare agent was not having to manually delete VM snapshots.  When Backup Exec would fail, it left the VM snapshots behind.  I had to manually clean up my virtual environment behind Backup Exec.  For a network admin with a full plate, this was an unnecessary responsibility.

After I upgraded to Backup Exec 2012, I purchased the de-duplication option.  I used that option during the beta and thought it was great.  Since the VM-based backup option wasn't working well, I figured I would make the best of it and backup my VMs with the remote agent.  I have a robust virtual environment with enough resources.  I configured my backup jobs to use source-side deduplication on my physical and virtual machines.  This would allow me to reduce the amount of data moving across the network from my production environment to my backup storage.  This feature gave me great backup speeds and made me happy.  Unfortunately, I would receive media errors on my deduplication store and things would quickly unravel.  I thought that I had a hardware issue with my disk storage and replaced it with a new SAN.  I still received Backup Exec media errors with the new storage and was advised to turn off source-side deduplication.  My backup times increased and I had to reorganize my backup schedule to allow for the increased backup windows.  At one point, I installed a Backup Exec hotfix and it turned my deduplication store into a tape library.  I belive that hotfix has been pulled.  Unfortunately, this mis-step by Symantec required me to recreate my backup environment.  I was advised to reformat the server and re-install the software.  This included formating and recreating my deduplication target.

I am not afraid to work hard to accomplish something.  That datacenter restore that I mentioned previously, caused me to sleep on the floor in my office for a few days.  I know how to dig in and "get'er done!"  Unfortunately, maybe I'm blinded by my "get'er done!" attitude.  The problems that I've had with Backup Exec has caused me to spend countless hours on the phone with tech support.  This had drawn me away from my other responsibilities.  Backups are my #2 priority in my job.  My #1 priority is making sure the power is on in the datacenter.  Symantec, you needs to fix our software and make it reliable.  IT Admins need to be able to rely on this software.  Backups are important, but they cannot monolopize our time.  Earlier in my career, switching to another backup software was deemed too costly.  However, with as much time I am spending on this product to troubleshoot and tweak it, maybe it's time to switch.

I've thought about posting my experience with Backup Exec for a long time.  I want to belive in the product.  I want to think that I've picked a winner.  I've invested a lot of time into this company and this product.  However, I woke up today to see that my Backup Exec Job Engine is stopped for the second time this weekend.  It's Martin Luther King, Jr. Day and I'm supposed to be on vacation.  My wife wants to know why I'm working and not getting things done around the house.

Symantec, please fix your product or I will be spending my time replacing your product with something else.

Sincerly,

Paul VanDyke

IT Supervisor

Kodiak Island Borough

Kodiak, Alaska