Verify job is very slow in Backup Exec 2012 deduplication storage

[sean_lee]

Hello.

We have build up new Backup Exec 2012 environemnt. 1 Media server and 20 Remote agents.

We are using Deduplication storage with client-side deduplication enabled for main backup.

While backup performance is OK (finished within 8 Hours), verify takes much more time comparing with backup (more than 24 hours)

Backup Exec Media Server

- Windows 2008R2 Enterprise with SP2

- Xeon E5620 CPU

- 24GB Memory

- IBM DS3500 for deduplication storage

- All BE updates installed except last one (HOTFIX 201596) 

We configured vefiry job as separate schedule, but still it takes too much time (all weekend)

And I could monitor that while Verify jobs are running, if some backup job started, the backup job is also extremly slow. (less than 100MB/min)

Is there any idea that we can improve verify performance?

[Gurvinder]

•Disable the option Allow this job to have direct access to this device if you use client-side deduplication with a verify template. Ensure the above checkbox is not selected for verify job

[sean_lee]

Thanks for reply.

The option "Enable direct access" already deselected for all Verify jobs before I post this question.

Any other suggestion?

[CraigV]

...if you have jobs that run while the veriy is running you could end up with a slow verify along with your slow backups. If can stagger this, do so.

That said, make sure that no antivirus on your media server is actively scanning the dedupe folder (exclude it). Your hardware spec looks right too!

THanks!

[sean_lee]

This slow verify jobs can be seen even there are only verify jobs (no backup).

The reason why I made separate verify job schedule is to minimize overlap with backup window.

For antivirus, I already exclude Deduplication Storage Folder from antivirus software.

By the way, when I see the document "Best practices for Backup Exec 2012 Deduplication Option", the only thing I didn't meet is "Disable RAID caching", is it possible RAID caching can be the cause of slow Verify performance?

Thanks

[pkh]

"Disable RAID caching" is unlikely to be the cause of your problem.  However, you should still disable the caching because this affects the integrity of your backup if there is an outage of the media server and/or the dedup folder.

[OP]

Hi sean lee,

I've got the same problem with every single task that is "read related" from the dedup storage - in my case duplicating to external media like tape or usb-disks.
Do you have similar experiences?

How is the speed during a restore? As slow like during the verify process ?

Thx

Olaf

[teiva-boy]

Verify from the dedupe store will ALWAYS be slow, there is no way around it. BE needs to read the store, reassemble the data/file to its original state, read the data, verify the data and move on to the next file. Its a slow process designed by Symantec. Not an issue with other competing products... Frankly verify from tape is more important than from disk. Or verify only on the replicated copy.

[teiva-boy]

Battery backed cache. Its been around for more than a decade.

[pkh]

They do fail at the wrong time.  I had them failed twice when they are needed.  Once, this resulted in a server rebuild.  Imagine restoring your dedup folder, that is if you had taken a backup of it.

[sean_lee · Accepted solution]

Thanks for lots of information

After I update all NICs and FC HBAs device driver and firmware to most recent version, the extremely slow performance when read/write operation at same time is getting better. So I don't need to run separate verify job.

I've monitored backup performance over a month and found that the overall backup performance was 3000MB/min ~ 4000MB/min with 8 concurrent operations on deduplication storage.

By the way when I increased the number of concurrent operations to 16, the overall backup performance was less than 1000MB/min.

I've varied the vaule of concurrent operations and found that, in my case, the optimum setting is 8 concurrent   job and overall backup performance is 3000MB/min ~ 4000MB/min. (even in read/write operation like backup jobs and verify jobs are running at same time)

If I found a better way to increase deduplication performance, I'll post it.

To Olaf

In my case, duplication performance to LTO6 tape library is almost same with backup performance (30000MB/min ~ 4000MB/min).

And I will intensive restore test until next week, so I'll let you know what the restore performance from deduplication storage is.

Thanks

Sean Lee