BE2014 SP1 - anyone having CASO & MMS server issues where DeDupe Storage is involved

[Bayside_City_Co]

Last Thursday night was our last successfully incremental backup.  

Daily Incremental and Weekly Full (Monthly Write last Full to Tape)

Last Friday I made the mistake of upgrading to the new SP1 and have not been able to backup correctly since.

Our basic model is to Backup all jobs to the De-Dupe Store on the MMS (24 TB RAW Space @ 60 % used) and the Optimized De-Duplicate this backup Set to a off-site De-Dupe Store (24 TB RAW Space @ 53 % used)  Both Servers have 32 GB RAM.

The Two Servers are Brand New DELL R720XD Server with on board disk.  They were freshly bought 6 months ago to have a clean Windows 2012 R2 Server build with a clean install of BE2014.  

We push around 9 TB of data to these De-Dupe Devices during the Weekly Full backups.

Most Agent based backups seem to work, they write to the De-Dupe store on the MMS but the Optimized Duplicate to the Off site BE2014 Server the job goes to "Active: Queued".  

Two of the 9 VMWare Agent AVVI GRT backups seem to run through OK until it need to duplicate the set to the off site De-Dupe store it  goes to Active:Queued no Job Rate and byte count.  The Rest sit at Active : Queued for days....

Call is logged with Symantec since Monday and other than shunting me from one team to the other and then promising to escalate it to next level just today have not been any help at all.

Is there a bug or have I broken the installation during the Patching process - can anyone offer any advise or run into this type of behavior?

Thanks to anyone that can offer anything, it will be 6 Days since we have done a backup.

Thanks in advance

Stuart

[Bayside_City_Co]

Finally got the Call Escalated to the next level and due to the time difference only had 1.5 hour to work on the issue and I am sure it helped the engineer to clarify a few things, but the issue is still not resolved.  

I must say these Engineers are the ones you need if you have a complex issue as they know how their product works.

Did what I needed to do to stop any job running on the servers, fully restarted them both (with the BE services set to Manual), waited till the De-Dupe disk activity was at a minimum and then started the BE Services.

Job Queue was on hold so edited every job so that they reset the start time and didn't attempt to run any of the Jobs that were hung, recovered or waiting for when the Job Queue was released.

Tested some local created and run jobs on the MMS and its different devices (De-Dupe, B2D and Tape) with varying levels of success/speed.

Set up a new job CAS to Backup one 3.5 GB File on the MMS into the Local De-Dupe Store, duplicate this to the CAS De-Dupe Store, this ran successfully up to the point it need to "Delayed Catalog" the backup set and then it dropped to Active:Queued and stayed there until we quit/killed the services (cancel job did not work).

Does this indicate there issue with the Catalog routine?  Deferred Catalog is a new "Feature" of BE2014, does it have a bug or did a component not get upgraded properly during the installation of SP1 as it was working in the RTM version???