Forum Discussion

SaWa's avatar
SaWa
Level 3
12 years ago

Backup Exec 2012 backup very slow (catalog problem?)

We've been having some problems with the speed of our backup. The setup is the following:

physical BE2012 server coonected to disk storage via fibre and connected to VmWare enviroment via Fibre as well. When backup up our file and print server to DeDupe location on our disk storage it takes a very long time (+19hr) for 900GB of data. It seems to hang on updating catalog.

History: Everything working fine and backupjob of all servers (1.4TB) runnning at speeds of between 4000MB/min to 6000MB/min. Then from one day to the other (no windows or BE updates done) the speed slowed down to between 600MB/min and 900MB/min.

Things allready tried: delete and recreate the backup job. update to SP1a. Thick all the boxes under the catalog tab and set to trucate after 4 weeks. run database maintenance at a time when no backup jobs are running. split up the job into different jobs (this alleviated the problem a bit)

After splitting the job up so that the exchange server, the SQL server and the file and print server had different jobs I noticed that the exchange and SQL were backing up at normal speeds (+4000MB/min) but the file and print server was still taking 18Hrs to backup. a bit more digging and a couple of days later I noticed that the data of file and print was backing up at speeds of +4000MB/min itself but it was hanging (going verrry slow) on updating the catalog at the end of the job while the size stayed unchanged.

My counterpart in our other location the other side of Australia is having the same problem for the exact same setup. (his problem started about 2 weeks before mine) He has been testing with going from full daily backups to incremental but the same problem stays.

We do not have a bottleneck problem with our disks, speed of our servers or any other hardware but it seems to come backup to the updating of the catalog for the file and print server. (my counterpart has also tried to uninstall and reinstall BE2012 but no sollution there either)

Does anyone have an idea how to clean, compact or do anything else with the catalog to get it back to normal speed? (catalog folder is currently 9GB with 16GB available)

Thanks in advance

2012
Backup and Recovery
Backup Exec
Deduplication Option
File System Archiving Option
  • SaWa's avatar
    SaWa
    12 years ago

    We finaly have a temporary solution from Symantec.

    It seems that there is a problem with a dll (pdvfsprotocol.dll) they installed a new dll and now the speed is to an acceptable 3.5GB/min backup to de-dupe (still not the 6GB/min I used to have but a lot better) They said they are still finalising the dll for general public and I'm not allowed to install any Symantec updates untill they bring out the final update and contact me again.

    So for anyone out there with this issue I would advise to create a ticket with Symantec and let it roll from there.

    Lets hope the final fix takes it back to original speeds and it won't take to long to get it.

    Hope the rest of you get there as well.

    G.

37 Replies

Sort By
  • CraigV's avatar
    CraigV
    Moderator
    12 years ago

    Hi,

     

    Is BE 2012 loaded with SP1a and all other available patches?

    Are you using SAN Transport mode for your backups as well? If not, check the TN below:

    http://www.symantec.com/business/support/index?page=content&id=TECH155831

    Thanks!

  • SaWa's avatar
    SaWa
    Level 3
    12 years ago

    As mentioned in the first post sp1a is installed. We have indeed loaded all other available patches.

    We have it configured exactly as the link shows.

    It has always been working fine up untill one day where it decided to go slow. it hangs on the "changing catalog" at the end of the job. which makes me think that there ight be a problem with the catalogs or the database (but I have not found any info yet on how to troubleshoot the db problems.)

    Cheers.

  • pkh's avatar
    pkh
    Moderator
    12 years ago

    The "updating catalog" phase is when BE enumerates the individual items for GRT restores.  If you have Exchange and files on the same VM, then each mail item and each file/folder needs to be enumerated and this can take some time.

    If you suspect a corrupt catalog, you can try creating a new catalog

    1) stop all BE services

    2) rename the catalog directory under the BE directory to catalog.old

    3) create a new catalog directory

    4) start all BE services

    5) try your job again.

  • SaWa's avatar
    SaWa
    Level 3
    12 years ago

    I'll give that a try and update when done.

    thanks.

  • Lejre's avatar
    Lejre
    Level 2
    12 years ago

    I have the same problem. Suddenly, updating catalogs is very slow. I think it happened after a microsoft update. Do you know if there is added microsoft updates to your backup exec 2012 server in January?

    Sorry I forgot to read, that you havent applyed any new windows updates.

    I have tried to rename the catalog folder, with no luck. Perhaps you are in luck...... Looking forward for your result.

  • SaWa's avatar
    SaWa
    Level 3
    12 years ago

    still the same problem, I've renamed the folder, restarted the server, ran the backup again. It backed up all data to dedupe at a rate of 6000MB/min and once the updating catalogs starts nothing happens anymore and you see the Job rate going down steadily. files in the catalog folder are populating very very slowly.

    Any other idea's would be welcome.

  • SaWa's avatar
    SaWa
    Level 3
    12 years ago

    I've noticed something else, When I go into the Job history, the start time and end time are showing in US date format instead of AU date format. (MM/DD/YYYY) all the rest in Backup exec and windows is showing AU date format. Has anyone has an idea if that might cause an issue? (grasping straws here).

    cheers.

  • SaWa's avatar
    SaWa
    Level 3
    12 years ago

    My counterpart in the other location has tried to format the Dedupe storage and backup exec hard drive as well with no positive outcome. the problem stays the same.

    cheers

  • _grimm's avatar
    _grimm
    Level 2
    12 years ago

    I have exactly the same issue,

    VMware backups via fibre channel to a media server with local dedupe store (9TB). Usual performance is 4000-6000MB/s with catalog updates taking anywhere between 1-20 mins at the end of the job, depending on which applications are being GRT processed.

    After installing the latest hotfixes 199866 and 200433 (to get orphan snapshot deletion functionality) everything has turned sour. I can backup hundreds of gigabytes and all looks good then randomely performance takes a nosedive. I have one system that has been updating catalogs for 18 hours and seems to process around 25 files per second - normal performance would see 1000's of files per second cataloged.

    Hardware is not the issue as both source and destinations are high performance and low utilisation. I suspect some sort of memory leak or buffer issue that is throttling performance but I can't see anything useful in the debug logs.

    The only fix seems to be to cancel running jobs and restart BUE services then all work fine again for a random time before slowing right down again.

    I am currently waiting on Symantec responding to my call and will update if I get anywhere. Very frustrating and not much in the way of successful backups for over a week now.

  • JBAS's avatar
    JBAS
    12 years ago

    Hi _grimm

     

    I have exactly the same issue.

     

    I am running 7 BE2012 Servers worldwide and had the issue on two of them. On one server I solved the issue by completely delete and recreate the Dedup Store. Everything runs fine now for one week. On the other server I did exactly the same without any luck. every night, Full & incremental backups, take about 24 to 48 hours, because of a slow catalog job of one server (it is always the same server).

     

    Did you already get an answer of Symantec Support?