BackupExec causes horrendous fragmentation

[Ross_Smith]

I am seeing incredible fragmentation on the drive we use to store our BackupToDisk files. I'm posting more details below, but if you are using B2D, could you please check the fragmentation level for me and let me know if you are also having this problem.

We have a 570Gb NAS device used as the main target for all of our backups. This device has been in use for approximately 6 months and is used almost exclusively for B2D backups.

After numerous problems with BackupExec, I happened to run a defrag analysis on this device, with the following results: http://www.averysilly.com/Defrag%20report.jpg.

Even after deleting 350Gb of B2D files, we are still getting 99% fragmentation reported with files scattered across the entire disk, and with a 1Gb b2d file in as many as 40,000 fragments.

I am still speaking with Veritas, we've no clear idea what caused this, but on at least one occasion I have set BackupExec to allow concurrent operations to this B2D device, and we suspect this may be the cause of this fragmentation.

We are defragmenting this device today to see if it fixes our problems, but I would be very interested to know if anyone else is seeing similar effects on their B2D devices?

RossSubject line updated 20/6/05 by:
Ross Smith

[A_Russell]

Hello. I just wanted to post that we have the same problem as well.

The media server is a Win2k server, with all updates.
We have a 750GB hard drive that is only used to store the Veritas backup files. We've only been using it for about 6-8 weeks. It was 99% fragmented.

Even with over 200 GB of free space, last night's backup-to-disk file had over 6000 fragments.

We use Veritas ver. 10.0 Rev. 5520

[Brad_Jacobsen_2]

Just checking in with the same problem. Using an MSA1500 attached to our server on its own SCSI controller, independent from the controller for our tapeloader. I formatted the drive only a few days ago, fragmentation of this level is not acceptable. Veritas has a real issue here but I have yet to see any acknowledgement of it.

Volume (D)
Volume size = 957 GB
Cluster size = 4 KB
Used space = 495 GB
Free space = 462 GB
Percent free space = 48 %

Volume fragmentation
Total fragmentation = 49 %
File fragmentation = 99 %
Free space fragmentation = 0 %

File fragmentation
Total files = 557
Average file size = 920 MB
Total fragmented files = 515
Total excess fragments = 38,467
Average fragments per file = 70.06

Pagefile fragmentation
Pagefile size = 0 bytes
Total fragments = 0

Folder fragmentation
Total folders = 9
Fragmented folders = 4
Excess folder fragments = 44

Master File Table (MFT) fragmentation
Total MFT size = 800 KB
MFT record count = 778
Percent MFT in use = 97 %
Total MFT fragments = 2

--------------------------------------------------------------------------------
Fragments File Size Most fragmented files
7,040 1.00 GB \Exchange\Brick\B2D000132.bkf
3,797 1.00 GB \Exchange\Brick\B2D000151.bkf
3,253 1.00 GB \Exchange\Brick\B2D000142.bkf
1,829 1.00 GB \Exchange\Brick\B2D000161.bkf
1,425 273 MB \Exchange\Brick\B2D000263.bkf
913 1.00 GB \Exchange\DBs\B2D000137.bkf
868 1.00 GB \Exchange\DBs\B2D000134.bkf
866 1.00 GB \Exchange\DBs\B2D000135.bkf
796 1.00 GB \Exchange\DBs\B2D000138.bkf
794 1.00 GB \Exchange\DBs\B2D000133.bkf
779 1.00 GB \Exchange\DBs\B2D000141.bkf
768 1.00 GB \Exchange\DBs\B2D000136.bkf
687 1.00 GB \Exchange\DBs\B2D000140.bkf
597 1.00 GB \Exchange\DBs\B2D000139.bkf
507 210 MB \Exchange\DBs\B2D000220.bkf
491 1.00 GB \Exchange\DBs\B2D000157.bkf
487 1.00 GB \Exchange\DBs\B2D000160.bkf
465 1.00 GB \Exchange\DBs\B2D000154.bkf
461 1.00 GB \Exchange\DBs\B2D000147.bkf
461 1.00 GB \Exchange\DBs\B2D000145.bkf
460 1.00 GB \Exchange\DBs\B2D000153.bkf
452 1.00 GB \Exchange\DBs\B2D000158.bkf
448 1.00 GB \Exchange\DBs\B2D000150.bkf
444 1.00 GB \Exchange\DBs\B2D000149.bkf
419 198 MB \Exchange\Brick\B2D000590.bkf
414 1.00 GB \Exchange\DBs\B2D000159.bkf
408 1.00 GB \Exchange\DBs\B2D000146.bkf
385 1.00 GB \Exchange\DBs\B2D000163.bkf
384 1.00 GB \Exchange\DBs\B2D000144.bkf
383 1.00 GB \Exchange\DBs\B2D000155.bkf


If Veritas does not address this and the slow B2D issues soon, I will be moving right back to ArcServe and not looking back. The benefits that BE10 had over AS have not met my expectations. Synthetic backups are pretty much worthless without the ability to backup databases and shadow copy components, B2D is 8x slower than backing up to tape, the list goes on...

[Ross_Smith_2]

Brad, out of curiosity, what speed are you getting to your NAS box? I've also had speed problems backing up to NAS in the past but have managed to get it all sorted now.

Ross

PS. I have a strong impression that Veritas rarely acknowledge problems until they have a fix ready. It's one of their worst failings and has caused me grief on several occasions. The only reason I'm still sticking with BackupExec is that I've got a decent relationship with the advanced support team now and I've had just as bad an experience with Arcserve in the past...

[Neil_Goldstein]

Has anyone with a support contract & who is experiencing this problem opened an offical support case (phone) for this?

[Brad_Jacobsen_2]

Ross, I'm getting 300MB/min, ±10MB/min. Backups to tape, an LTO-3 drive, scream along at 1.5 - 2.0GB/min. I have indexing, antivirus and OFO disabled but 300MB/min is the best I can get. As we recently switched from Arcserve to BE for the B2D and synthetic backup capabilities which are not panning out, I may abandon BE until this problem is solved OR indefinitely. Arcserve's B2D capabilities, while limited, will still allow my backups to finish within the windows I am allowed.

[Ross_Smith_2]

I've mentioned the fragmentation to the advanced support team in the UK, so they are aware of the problem and have the URL of this discussion. I've never actually opened a case for it though, it just got mentioned on the back of another case. I may do that now since it's obviously not an isolated problem.

With regards the speed issues, I've had problems with Maxtor Windows Powered NAS boxes that used software RAID 5, switching these to RAID 0 made a huge difference.

I suspect however you won't be using software RAID considering the rest of your spec :) I think it far more likely you're hitting another problem I've found - BackupExec is far slower when dealing with large numbers of files.

On our system, we can get 766Mb/min backing up a single 300Mb zip file to one of our NAS boxes. Backing up a set of files (22,000 files taking up 170mb) however runs far slower at 158Mb/min. A straight file copy of that 300Mb file in explorer runs at 1233Mb/min.

Disabling software compression does up the rate somewhat - to 1058 and 233Mb/min respectively.

I raised this issue with veritas when I found that Winzip is actually 6 times faster than BackupExec:
http://forums.veritas.com/discussions/thread.jspa?threadID=48065&start=0&tstart=0

If ArcServe actually gives you better performance, I would be very interested in the figures - we also struggle to fit our backups into the time windows allocated.

[Brad_Jacobsen_2]

Ross,
I set up Arcserve 11 on the same server, deployed agents, formatted my MSA1500 volume and set about backing up the same data as before. Afterwards, I have a single file that consists of 2 fragments, all other files are a single fragment. The backup completed in 6 hour window, which is acceptable. However, one server which has 80GB of data in 1 million files backed up at 360MB/min with Arcserve, and backed up at nearly twice that speed with BE. In other areas, however, I saw Arcserve far exceed BE's transfer rates for the same volumes across the network, so it is clear that the 2 programs handle data very differently. I have no doubts that Veritas would work much better for me as a backup solution once the fragmentation bug and the 300MB/min problem I get on remote backups are resolved.

[Arjan_van_Doorn]

I have the same problem, opened a case...2 allready. since 8 juli. The fragmentation problem also exist when using NTbackup. No fragmentation on file copy's

Latest reaction from Veritas:
"There is nothing which I can do form my side. This is the behaviour of your disk. There is no solution for that. Even this happens with NTBACKUP. You can call up your disk vendor and ask them whether they have any solution for it."

Played with a lot of b2d settings, no change in fragmentation.

BE 10.0 5484
Windows 2003 sp1
HP DL380G03

Looks like we are drowning...

[Ross_Smith_2]

Funny how it doesn't happen with ArcServe though...

It doesn't happen with Microsoft Exchange either - I know that's not a backup program, but BackupExec is not using any special system call to write these files, we're simply talking about an operation that writes a lot of data to the NTFS file system.

Our Exchange server log files handle just as much data on a daily basis as BackupExec, and they do so with effectively zero fragmentation.

This is all about whether you take due consideration of how NTFS works when designing a system to write to it, or whether you just fudge your tape driver so it can stream to NTFS...

Veritas can try to pass this off all they like. We have confirmation now that ArcServe does not behave in the same way.

[Sandeep_Swain_2]

Hi,

We had the same result with NTBACKUP, where the disk was getting dragmented after the backup job.

[Ross_Smith_2]

I'm not doubting that Ntbackup has this problem, it's just not something I expect from a fully fledged backup solution. Especially not one that touts backup to disk as a major feature.

[Thomas_McGeown]

For information only:
I have also noticed that this problem also occurs when using the Desktop and Laptop Option. On files over 1gb the fragmentation is nearly always over 50000.
The file fragmentation on the disk is 64%.
I will soon be migrating the data to a new NAS and will see if the fragmentation continues after the transfer and a defragment is run.

Tom

[Lars_Lange]

Does anybody (besides me) think that Veritas is ignnoring this tread ? I have tried to increase the size of the backupfiles to 10 GB. Same result :(

So Veritas, pretty please - with a cherry on top, HELP US!

/Lars

[Arjan_van_Doorn]

Support is (were) helping me but there is no solution, I have to make an Enhancement call.

So I advice / ask you all to make also an enhancement call at: http://enhancement.veritas.com/

Thx.
Arjan

[ross_tuininga]

We do our backups on a win2k3 server with a 1.4 terabyte SATA array. The full and diffs go to the hard drive array and we copy to tape weekly. The fragmentation was so bad we scheduled a dfrag daily. We also scheduled an array consistency check weekly.

Backups are now faster to drives than to tape but we still plan on implementing an enhancement suggested by our hardware vendor, silicon mechanics. We will install more RAID controllers to expand the pipe. This is part of increasing the size of our backup server.
One controller will handle just C: with the veritas software on a small mirror. For the backup array we will use a software stripe to access two other controllers in "parallel". We have already paralleled the network interfaces on a Gbit switch. It currently seems our backup server keeps up with everything our regular servers can throw at it, but we are adding more load. You might want to try this kind of setup if you expand your backup servers.
rt

[Peter_Ludwig]

I do use a very similar setup W2k3 server, 1.3TB array (19 disks á 70GB). I just checked and I also have 99% fragmentation. I actually never wondered about that. And I thought that the OS is in charge for that.
I just run a defragmentation (hopefully it finishes before the first backup jobs start).

And: my BKF files are up to 230Gb large, this has absolutely no influence, so this hint was just misleading.

I may just setup up defragmenation with the scheduler to perform at the backup server's idling time.

Thanks a lot for bringing this issue up.

greetings
Peter

[Lars_Lange]

Have tried to call Veritas support, twice.
No solution to this problem, and they had never heard of it.

Only suggestion was to increase the filesize.

/LarsMessage was edited by:
Lars Lange

[Andy_Newell]

"Me Too!" One of several reasons I am looking to leave Veritas.

Data shown is a B2D-dedicated 0.5 TB LaCie iee1394 Disk, after 3 days' use. B2D files are 10G each. The drive is formatted with largest available block size. Win2K3sp1.

I wonder, have we run afoul of the old rule-of-thumb about not filling NTFS disks beyond 80% full?

Anybody using Raxco PerfectDisk or another real-time defrag engine under thier B2D?
-------------------

Volume fragmentation
Total fragmentation = 47 %
File fragmentation = 95 %
Free space fragmentation = 0 %

File fragmentation
Total files = 63
Average file size = 5.27 GB
Total fragmented files = 42
Total excess fragments = 16,828
Average fragments per file = 268.11

Pagefile fragmentation
Pagefile size = 0 bytes
Total fragments = 0

Folder fragmentation
Total folders = 7
Fragmented folders = 1
Excess folder fragments = 0

Master File Table (MFT) fragmentation
Total MFT size = 384 KB
MFT record count = 151
Percent MFT in use = 39 %
Total MFT fragments = 2

--------------------------------------------------------------------------------
Fragments File Size Most fragmented files
4,167 10.00 GB \VERITAS\B2D\B2D000477.bkf
3,398 3.79 GB \VERITAS\B2D\B2D000474.bkf
3,276 10.00 GB \VERITAS\B2D\B2D000520.bkf
2,290 10.00 GB \VERITAS\B2D\B2D000496.bkf
1,636 10.00 GB \VERITAS\B2D\B2D000479.bkf
546 10.00 GB \VERITAS\B2D\B2D000481.bkf
377 1.34 GB \VERITAS\B2D\B2D000499.bkf
....snip.

[Ross_Smith_2]

Andy, it's not about filling the disks over 80% - I wiped a 1.2Tb drive to test this - it happens right from the start.

[Lars_Lange]

Here is my "top 20" of files with most fragments

Fragmentes / Filesize /Filename
55483 10240.0 MB B2D002557.bkf
42226 10240.0 MB B2D002558.bkf
3834910240.0 MB B2D002577.bkf
36557 7673.1 MB B2D002603.bkf
27603 10240.0 MB B2D002727.bkf
26968 7601.0 MB B2D002712.bkf
26802 10240.0 MB B2D002559.bkf
26776 10240.0 MB B2D002556.bkf
24048 10240.0 MB B2D002596.bkf
22189 10240.0 MB B2D002583.bkf
19235 4209.4 MB B2D002751.bkf
16894 10240.0 MB B2D002538.bkf
15232 10240.0 MB B2D002605.bkf
14559 10240.0 MB B2D002535.bkf
13083 10240.0 MB B2D002706.bkf
12242 10240.0 MB B2D002721.bkf
10547 10240.0 MB B2D002607.bkf
9218 10240.0 MB B2D002723.bkf
8954 7495.0 MB B2D002549.bkf
8906 10240.0 MB B2D002761.bkf

The setup is SATA array with 5 400 gb disk in raid 5. on a intel raid controller. The system disk is on another raid controller (onboard) This drive is _only_ used by Veritas backup files.

/Lars