BackupExec causes horrendous fragmentation

[Ross_Smith]

I am seeing incredible fragmentation on the drive we use to store our BackupToDisk files. I'm posting more details below, but if you are using B2D, could you please check the fragmentation level for me and let me know if you are also having this problem.

We have a 570Gb NAS device used as the main target for all of our backups. This device has been in use for approximately 6 months and is used almost exclusively for B2D backups.

After numerous problems with BackupExec, I happened to run a defrag analysis on this device, with the following results: http://www.averysilly.com/Defrag%20report.jpg.

Even after deleting 350Gb of B2D files, we are still getting 99% fragmentation reported with files scattered across the entire disk, and with a 1Gb b2d file in as many as 40,000 fragments.

I am still speaking with Veritas, we've no clear idea what caused this, but on at least one occasion I have set BackupExec to allow concurrent operations to this B2D device, and we suspect this may be the cause of this fragmentation.

We are defragmenting this device today to see if it fixes our problems, but I would be very interested to know if anyone else is seeing similar effects on their B2D devices?

RossSubject line updated 20/6/05 by:
Ross Smith

[Ross_Smith_2]

Ok, I think we're all well aware that BackupExec causes this fragmentation and it's nothing to do with drives / configuration or NTFS.

I think the issue now is; Does this have an actual effect on the backup performance?

Veritas right now don't seem to see this as a priority. I know they're aware of it, but it would seem that they don't really see this as a problem so long as BackupExec is actually taking backups.

I think we really need to start collecting some figures on BackupExec's performance before & after defragging these drives so we've got some concrete figures we can take to Veritas.

I fully expect that in the long term performance will suffer as B2D files get more and more fragmented and I'm planning to keep an eye on the performance of our most used NAS device.

[Lars_Lange]

> I think the issue now is; Does this have an actual
> effect on the backup performance?
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Yes! After 2-3 months, the backup was so slow so the time windows I have to make backups was to small to Veritas :(

> Veritas right now don't seem to see this as a
> priority. I know they're aware of it, but it would
> seem that they don't really see this as a problem so
> long as BackupExec is actually taking backups.
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Sorry to say that i am not surpriesed.

> I think we really need to start collecting some
> figures on BackupExec's performance before & after
> defragging these drives so we've got some concrete
> figures we can take to Veritas.
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
I agree, but i leaving or vacation, so i will first be able to deliver some data in 3 weeks :)

/Lars

[Joshua_McKenzie]

Same results here. Just inherited the backup systems and in attempting to troubleshoot sub-100mb/min B2D performance, I ran a quick defrag scan and sure enough - 99%.

Running on a Dell NAS 720 w/Windows 2003 Appliance w/SP1, SATA RAID 5 w/4-250GB drives, 2 gig RAM, etc.

Upon completely deleting the B2D files and re-creating the device/media set, the backups jumped up to ~580mb/min, but after three days of incrementals, fragmentation was back up to ~80% and climbing.

Cleared the B2D files again, ran the backup from a non-fragmented system to make sure fragmentation of target systems wasn't an issue, and BackupExec fragmented the file all over the free space on the drive.

Running with 512mb vs. 1 gig vs. 5 gig B2D file size has made no difference working with this problem.

Just thought I'd toss in my .02

~Josh

[Vin_Latus]

I too am experiencing awful fragmentation problems, that were brought to my attention by increasingly slow backups.

[Vin_Latus]

Sorry, forgot to give you config info.

Windows 2003 BE 10.0

Proliant DL380 with MSA20 direct attached.

Analysis Report

1,630 GB Total, 1,446 GB (88%) Free, 49% Fragmented (99% file fragmentation)

The above is the result of wiping the drive array and starting over with one nights backup.

PS. I'm new to this forum, at the top of the page it says:
This question is not answered. Helpful answers available: 2, correct answers available: 1.

But I don't see how/where the correct answer can be accessed. If there is an answer to this problem, I'd like it. ;)Message was edited by:
Vin Latus

[Ken_Putnam]

The way that this forum is set up, the original poster can say that two responses were "Helpful" and the responders get 5 points each. The OP can also say that one response is "Correct" and that responder gets 10 points

A response that was helpful will have an amber light bulb in the header like this one http://forums.veritas.com/discussions/thread.jspa?threadID=52398&tstart=15
and the correct answer will have a green light bulb like this one http://forums.veritas.com/discussions/thread.jspa?threadID=52523&tstart=90

[Ross_Smith_2]

A little off-topic but I started this thread and I've never marked anyone's reply as helpful or correct. I assume the responses from Veritas were automatically marked as one or the other.

I don't think there is a solution for this until Veritas develop one, and last time I mentioned it to them they didn't seem too bothered. This thread was started to raise awareness of the problem and to get some facts & figures together so I can bring it to Veritas' attention.

Right now I'm most keen to get figures on any performance probems caused by this. That would seem to be the area that's going to give us the most leverage to pressure Veritas into fixing this. I'm going over my old logs today to see if performance has been dropping.

[Peter_Ludwig]

Actually it should not be necessary to mention it, but I left the defragmentation run and forgot about it. In the evening the backups tried to start und failed with strange error messages (database in use or something like that), the jobs did not even produce log files.

I will have a look at Raxco PerfectDisk

greetings
Peter

[Tommy_Gronbeck]

To Peter: Interesting info about running defrag and BE at the same time, I haven't left defrag running during backup-time yet, I downloaded a 30-day trial version of Diskeeper and scheduled a continous defrag during off-backup times, but definately something worth thinking about while considering a defrag-strategy.

To Ross: We have had the exact same problems as discussed earlier, we're running Windows 2003 Server, BE 10.0 (5520) with 4 300 GB scsi discs in raid 0 and a fragmentation level of 99% with our worst files being split into more than 10.000 fragments...

We noticed that there was some sort of problem with our B2D solution when the backup job started to take increasingly longer time to finish.
We're backing up approximately 250 GB to disc and tape each night and when the backup-job (it starts at 21:00) didn't finish until 12:00 the following day we ran a two-day defragmentation on the volume. After defragging the volume the backup now finishes 06:00 - i.e. it finishes 6(!) hours earlier than before running daily defragmentations.

If this doesn't constitute a problem that Veritas should look into, I don't know what does... We have today taken the decision to purchase a full license of Diskeeper Server Enterprise Edition, just to be able to get the performance out of BE that we thought BE alone would provide us with.

[Peter_Ludwig]

Thanks Tommy,

I've downloaded Diskeeper and try it for the next month. It seems that it is the first tool which can deal with the problem.

greetings
Peter

[Tommy_Gronbeck]

To me it was pretty clear from the start that the fragmentation was a problem. We discovered the horrendous fragmentation when our backup-to-disc started taking longer and longer time to finish. When we saw the red display in defrag we were fairly certain of what was causing our problems.

Once we got the disc defragmented completely and started defragmenting every day right after BE had fragmented the disc beyond recognition, we experienced a drop in the time our backups took:
Before defragmentation the backup took: 15h
After defragmentation the backup took: 9h
Performance gain from defragmentation: 6h (40% performance gain)

To me this pretty clearly states that Backup Exec's B2D-functionality is poorly designed and not functioning as it should. And for reasons stated earlier in this thread, the problem does not lie within the operating system or the file system used neither does it contain itself to certain hardware configurations. Instead, the only common software/hardware used by every single one describing their configurations in this thread is Backup Exec...

Fortunately we have found a solution that solves our problem. Now if only Symantec/Veritas could acknowledge this problem and come up with a viable solution that does not include their customers being forced to buy a software from another company, everything would be fine...

[Vin_Latus]

Just thought I would give a little update. It's been almost a month since I wiped my backup array and started over. Since that time, I've been running the built-in Win2k3 defrag on a daily basis. Backup performance is again starting to suffer.

I'd really rather not buy Diskkeeper either, it would be nice if Symantec/Veritas would resolve this issue. We sold this idea of B2D to our management telling them it would be faster. Now we just look incompetent.

Volume Data (D:)
Volume size = 1,630 GB
Cluster size = 4 KB
Used space = 1,471 GB
Free space = 159 GB
Percent free space = 9 %

Volume fragmentation
Total fragmentation = 49 %
File fragmentation = 99 %
Free space fragmentation = 0 %

File fragmentation
Total files = 28
Average file size = 70.02 GB
Total fragmented files = 17
Total excess fragments = 2,345,886
Average fragments per file = 83782.64

Pagefile fragmentation
Pagefile size = 0 bytes
Total fragments = 0

Folder fragmentation
Total folders = 5
Fragmented folders = 1
Excess folder fragments = 0

Master File Table (MFT) fragmentation
Total MFT size = 23 MB
MFT record count = 23,047
Percent MFT in use = 99 %
Total MFT fragments = 2

[Arjan_van_Doorn]

>Support is (were) helping me but there is no solution, I have to make an Enhancement call.
>
>So I advice / ask you all to make also an enhancement call at: http://enhancement.veritas.com/
>
>Thx.
>Arjan

Pls all make an enhancement call so they (hopefully) will do something about it.

I use Diskkeeper aswel.
Also I use 2 volumes on 2 controllers, Volume one for the Full backup wich will be defragged within 3 days, Volume two a smaller volume for the Daily backup, wich will be defragged during working hours and is ready before the next daily cycle.
The speed is back were it should be

[Michael_Rogers_]

We have a vtrack 15100 with 3 1.5 tb arrays each dedicated to b2d. We experience the exact same problems with fragmentation.

[Lars_Lange]

Hi all,

Just to let you know what my next move is.
I have been told that netvault from www.bakbone.com has a great backup-to-disk option. That works, so I am gonna give it a try. And for now only use Veritas to tape-backups. If its nessary with Netvault.

If anybody wish I can post a feedback in one week or so.

[Paul_Yip]

> We sold this idea of B2D to our management telling
> g them it would be faster. Now we just look
> incompetent.
>

I am in the same position. We have over a terabyte worth of space and it is useless as my full backups run over to Monday now when they used to be done Saturday morning. I may have to take out my backup to disk jobs.

[Roland_Apacway]

The fragmentation problem is an inherent problem of BACKUP TO DISK. Not necessarily Backup Exec. Larger backup files will help reduce fragmentation but over time the fragmentation will get worse no matter what size the backup files are.

This is becasue the filesystem is changing rapidly & blocks are being overwritten regularly. Remember the blocks on the hard disk are written to randomly & not in sequence. Fragmentation will adversly affect any write intensive application - backup to disk included.

[Roland_Apacway]

I might add that there are hardware options to overcome this problem....

NetApp filers (with the aid of their WAFL filesystem) write to disk in a manner that reduces fragmentation.

Overland Data has a backup to disk product - The REO - that rezises its volumes according to the size of the data being written - which helps it overcome the fragmentation problem.

[Ross_Smith_2]

This has already been discussed. As far as I'm concerned there's no need for this level of fragmentation to occur. The best example of this is to compare BackupExec with an Exchange Server.

We see around 10Gb of data a week passing through our exchange log drive yet there is never any level of fragmentation - it's reported as 0% fragmentation constantly. I'm confident I can leave that server running for several years without any concerns over fragmentation.

Compare that to BackupExec: after a single backup, BackupExec has 90% fragmentation or worse. Leave it a month and it's having noticable effects on performance - a 20% drop in my case after one month.

Right now I'm leaving the system running - I'm very curious to see just how bad this is going to get...

My point is that any backup system dealing with large volumes of file traffic should be designed so that performance does not degrade over time. Microsoft obviously considered fragmentation when designing Exchange and managed to avoid it completely. Veritas should have done similar when planning BackupExec.

I'm not aware of any other program that can split a single file into several thousand fragments on it's first run on a clean drive.

Oh, and by the way... one of the users here has already reported that ArcServe doesn't demonstrate the same behaviour.

[Peter_Ludwig]

I agree completely with Ross Smith, it is a problem which could have been avoided.
No I also know what the Overland REO is good for (compared with a simpler solution)

I tried DiskKeeper and PerfectDisk, both of them can not keep up with the fragmentation. And they seem to cause other troubles, so I removed them again.

greetings
Peter