We've been having some problems with the speed of our backup. The setup is the following:
physical BE2012 server coonected to disk storage via fibre and connected to VmWare enviroment via Fibre as well. When backup up our file and print server to DeDupe location on our disk storage it takes a very long time (+19hr) for 900GB of data. It seems to hang on updating catalog.
History: Everything working fine and backupjob of all servers (1.4TB) runnning at speeds of between 4000MB/min to 6000MB/min. Then from one day to the other (no windows or BE updates done) the speed slowed down to between 600MB/min and 900MB/min.
Things allready tried: delete and recreate the backup job. update to SP1a. Thick all the boxes under the catalog tab and set to trucate after 4 weeks. run database maintenance at a time when no backup jobs are running. split up the job into different jobs (this alleviated the problem a bit)
After splitting the job up so that the exchange server, the SQL server and the file and print server had different jobs I noticed that the exchange and SQL were backing up at normal speeds (+4000MB/min) but the file and print server was still taking 18Hrs to backup. a bit more digging and a couple of days later I noticed that the data of file and print was backing up at speeds of +4000MB/min itself but it was hanging (going verrry slow) on updating the catalog at the end of the job while the size stayed unchanged.
My counterpart in our other location the other side of Australia is having the same problem for the exact same setup. (his problem started about 2 weeks before mine) He has been testing with going from full daily backups to incremental but the same problem stays.
We do not have a bottleneck problem with our disks, speed of our servers or any other hardware but it seems to come backup to the updating of the catalog for the file and print server. (my counterpart has also tried to uninstall and reinstall BE2012 but no sollution there either)
Does anyone have an idea how to clean, compact or do anything else with the catalog to get it back to normal speed? (catalog folder is currently 9GB with 16GB available)
Thanks in advance
Solved! Go to Solution.
We finaly have a temporary solution from Symantec.
It seems that there is a problem with a dll (pdvfsprotocol.dll) they installed a new dll and now the speed is to an acceptable 3.5GB/min backup to de-dupe (still not the 6GB/min I used to have but a lot better) They said they are still finalising the dll for general public and I'm not allowed to install any Symantec updates untill they bring out the final update and contact me again.
So for anyone out there with this issue I would advise to create a ticket with Symantec and let it roll from there.
Lets hope the final fix takes it back to original speeds and it won't take to long to get it.
Hope the rest of you get there as well.
Is BE 2012 loaded with SP1a and all other available patches?
Are you using SAN Transport mode for your backups as well? If not, check the TN below:
As mentioned in the first post sp1a is installed. We have indeed loaded all other available patches.
We have it configured exactly as the link shows.
It has always been working fine up untill one day where it decided to go slow. it hangs on the "changing catalog" at the end of the job. which makes me think that there ight be a problem with the catalogs or the database (but I have not found any info yet on how to troubleshoot the db problems.)
The "updating catalog" phase is when BE enumerates the individual items for GRT restores. If you have Exchange and files on the same VM, then each mail item and each file/folder needs to be enumerated and this can take some time.
If you suspect a corrupt catalog, you can try creating a new catalog
1) stop all BE services
2) rename the catalog directory under the BE directory to catalog.old
3) create a new catalog directory
4) start all BE services
5) try your job again.
I have the same problem. Suddenly, updating catalogs is very slow. I think it happened after a microsoft update. Do you know if there is added microsoft updates to your backup exec 2012 server in January?
Sorry I forgot to read, that you havent applyed any new windows updates.
I have tried to rename the catalog folder, with no luck. Perhaps you are in luck...... Looking forward for your result.
still the same problem, I've renamed the folder, restarted the server, ran the backup again. It backed up all data to dedupe at a rate of 6000MB/min and once the updating catalogs starts nothing happens anymore and you see the Job rate going down steadily. files in the catalog folder are populating very very slowly.
Any other idea's would be welcome.
I've noticed something else, When I go into the Job history, the start time and end time are showing in US date format instead of AU date format. (MM/DD/YYYY) all the rest in Backup exec and windows is showing AU date format. Has anyone has an idea if that might cause an issue? (grasping straws here).
I have exactly the same issue,
VMware backups via fibre channel to a media server with local dedupe store (9TB). Usual performance is 4000-6000MB/s with catalog updates taking anywhere between 1-20 mins at the end of the job, depending on which applications are being GRT processed.
After installing the latest hotfixes 199866 and 200433 (to get orphan snapshot deletion functionality) everything has turned sour. I can backup hundreds of gigabytes and all looks good then randomely performance takes a nosedive. I have one system that has been updating catalogs for 18 hours and seems to process around 25 files per second - normal performance would see 1000's of files per second cataloged.
Hardware is not the issue as both source and destinations are high performance and low utilisation. I suspect some sort of memory leak or buffer issue that is throttling performance but I can't see anything useful in the debug logs.
The only fix seems to be to cancel running jobs and restart BUE services then all work fine again for a random time before slowing right down again.
I am currently waiting on Symantec responding to my call and will update if I get anywhere. Very frustrating and not much in the way of successful backups for over a week now.
I have exactly the same issue.
I am running 7 BE2012 Servers worldwide and had the issue on two of them. On one server I solved the issue by completely delete and recreate the Dedup Store. Everything runs fine now for one week. On the other server I did exactly the same without any luck. every night, Full & incremental backups, take about 24 to 48 hours, because of a slow catalog job of one server (it is always the same server).
Did you already get an answer of Symantec Support?
I am having the same issue. Backups to dedupe folder are slow specifically in the catalog process. Job use to run in 6 hours, now runs in 24+ hours.
I have an open case with Symantec which is now going on its' 4th week (case opened 1/23). To this date, I have not been provided with any explanation or fix. All we have done is recreated the catalogs folder and generated some debug logs. Issue I have now, is that a catalog of the dedupe folder will not even complete. I am also finding that catalog jobs for tapes are running very slow as well.
Our next steps are to remove the dedupe folder and then readd it. I am told this process will not remove the backup contents in the dedupe folder. We are performing some test restores with tape to be sure we can deal with the removal or potential loss of the dedupe folder.
Anyone else have any recommendations or possible fix for the issues?
We're experiencing the same issue. But we have the slow updating catalog problem on every Disk target, not just only for the deduplication folder. Our problem came up with the installation of the latest hotfixes 199866 and 200433.
We uninstalled the Hotfix 200433, but the problem persists. The Hotfix 199866 cannot be uninstalled, so we opened a support case and we're still waiting for a solution.
The problem has been there since the beginning of January. It is very disappointing that Symantec has not found a solution yet. We have been forced to convert our backup routine for file-based backup. Vmware backup with GRT sucks. Symantec fix your software.
Hopes Symantec finds a solution as quickly as possible.
We have the same issue here. All the latest patches applied and it started around January. I ran patching last week to hopefully fix the issue but no love yet from Symantec.
We have to say that we are really disappointed about the current support. We have open the ticket now nearly 4 weeks and in the last week we didn't received any status updates about our case.
How is that a viable solution to this issue? That compromises the customer's entire backup plan and severely cuts the retention time for backup data.
This is truly an issue since hotfix 199866. The performance issue is specifically with the catalog process of a VMware Virtual Machine when using a dedupe folder as the backup destination.
My case was opened on 1/23/13. I have also worked with their SMB support which appear to be in the mode of a response or recommendation every 2-3 days. I find it very disheartening that this issue has been dealt with so poorly.