BackupExec causes horrendous fragmentation

[Ross_Smith]

I am seeing incredible fragmentation on the drive we use to store our BackupToDisk files. I'm posting more details below, but if you are using B2D, could you please check the fragmentation level for me and let me know if you are also having this problem.

We have a 570Gb NAS device used as the main target for all of our backups. This device has been in use for approximately 6 months and is used almost exclusively for B2D backups.

After numerous problems with BackupExec, I happened to run a defrag analysis on this device, with the following results: http://www.averysilly.com/Defrag%20report.jpg.

Even after deleting 350Gb of B2D files, we are still getting 99% fragmentation reported with files scattered across the entire disk, and with a 1Gb b2d file in as many as 40,000 fragments.

I am still speaking with Veritas, we've no clear idea what caused this, but on at least one occasion I have set BackupExec to allow concurrent operations to this B2D device, and we suspect this may be the cause of this fragmentation.

We are defragmenting this device today to see if it fixes our problems, but I would be very interested to know if anyone else is seeing similar effects on their B2D devices?

RossSubject line updated 20/6/05 by:
Ross Smith

[Ross_Smith]

Bounce.

Has anyone else had this kind of problem?

[Grant_Wood]

Yes, our backup to disk solution also fragments data beyond exceptable limits. Just noticed it when reading your thread and checking my own backup disks. I think others are having the same issue.

[Ross_Smith]

Could you post a screenshot or copy/paste the figures from your backup report. I think it would be useful to help highlight to Veritas just how bad this problem is.

Ross

[Amruta_Purandar]

Hello,

How do you mean the disk is getting fragmented?
- Is it creating multiple .bkf files?
- Are these files not getting overwritten?
- Are you choosing to overwrite these files?

Also which device are you backing up to?
- Is it a IOMEGA drive or a hard disk? Please specify.

NOTE : If we do not receive your reply within two business days, this post would be marked assumed answered and would be moved to answered questions pool.

[Ross_Smith]

I mean in the normal sense of the word when applied to hard disks... you'll also get a pretty clear idea if you take the time to view the screenshot in my post...

I'm going to answer your questions, just to be thorough, even though they have absolutely no bearing on this issue, and even though nothing in my post is asking for support from Veritas. It's simply a post on the board asking other users if they have seen this.

- Is it creating multiple .bkf files?
* Yes. My original post actually referenced 350Gb of bkf files. Not only does this explicitly state 'files', plural, but if you understood your software properly, you would know that Veritas don't recommend using file sizes larger than 2Gb for network operations.

- Are these files not getting overwritten?
* No, they are overwriting fine, this has nothing to do with my statement.

- Are you choosing to overwrite these files?
* Yes, we are using a normal retention policy. This again has NOTHING to do with my statement.

The device we are backing up to is a 640Gb Iomega MaxAttach 4300 NAS server. I doubt however that this is what you mean by an Iomega drive...

Bearing in mind that you don't seem to have paid much attention to my post, let me explain:

BackupExec is scattering it's bkf files all over the disk. Each file is being split into multiple parts, just one of our 1Gb .bkf files was actually split into 10,000 separate entries on the drive. This has a huge effect on performance since the server has to scan right across the hard drive to read files. We also believe it may have affected the reliability of this server.

It's taken me two days so far to correct this just so that I can begin diagnostics on our backup server again. My post here is not asking for help, this issue is already logged with Veritas' advanced support team in the UK. My sole reason for posting is to see how many other Veritas customers are also affected by this problem so I can pass that information back to the advanced support guys.

Ross

[Grant_Wood]

I am using a Promise VTrak 15100 Disk Array

My Defrag report is as follows:

Volume DiskBackup (F:)
Volume size = 2,235 GB
Cluster size = 4 KB
Used space = 749 GB
Free space = 1,485 GB
Percent free space = 66 %

Volume fragmentation
Total fragmentation = 48 %
File fragmentation = 96 %
Free space fragmentation = 0 %

File fragmentation
Total files = 459
Average file size = 1.66 GB
Total fragmented files = 415
Total excess fragments = 7,229
Average fragments per file = 16.74

Pagefile fragmentation
Pagefile size = 0 bytes
Total fragments = 0

Folder fragmentation
Total folders = 13
Fragmented folders = 5
Excess folder fragments = 38

Master File Table (MFT) fragmentation
Total MFT size = 763 KB
MFT record count = 509
Percent MFT in use = 66 %
Total MFT fragments = 4

I am still on my first 4 week rotation but your post has me watching for a solution. I have no concurrent operations on this divice. I will be defraging the array tonight to also see if that helps control fragmentation as files are overwritten. I am starting the next 4 week rotation this Sunday.

Never seen a disk so fragmented!

[Ross_Smith]

Good god! 96% fragmentation after 4 weeks? Surely this can't be right. Thanks for the extra info, I'll pass that on to Veritas support.

You say you've never seen a disk so fragmented... you should see what it looks like after 6 months:

Volume Data (E:):
Volume size = 574 GB
Cluster size = 4 KB
Used space = 219 GB
Free space = 355 GB
Percent free space = 61 %

Volume fragmentation
Total fragmentation = 49 %
File fragmentation = 99 %
Free space fragmentation = 0 %

File fragmentation
Total files = 2,125
Average file size = 111 MB
Total fragmented files = 271
Total excess fragments = 388,440
Average fragments per file = 183.79

Pagefile fragmentation
Pagefile size = 0 bytes
Total fragments = 0

Directory fragmentation
Total directories = 48
Fragmented directories = 7
Excess directory fragments = 123

Master File Table (MFT) fragmentation
Total MFT size = 10,346 KB
MFT record count = 6,579
Percent MFT in use = 63 %
Total MFT fragments = 2

Ross

[Charles_Graham]

we are having the same problem as well

Volume BACKUPS (D)
Volume size = 74.51 GB
Cluster size = 4 KB
Used space = 37.86 GB
Free space = 36.65 GB
Percent free space = 49 %

Volume fragmentation
Total fragmentation = 49 %
File fragmentation = 99 %
Free space fragmentation = 0 %

File fragmentation
Total files = 51
Average file size = 841 MB
Total fragmented files = 38
Total excess fragments = 134,837
Average fragments per file = 2644.86

Pagefile fragmentation
Pagefile size = 0 bytes
Total fragments = 0

Folder fragmentation
Total folders = 8
Fragmented folders = 1
Excess folder fragments = 0

Master File Table (MFT) fragmentation
Total MFT size = 1 MB
MFT record count = 677
Percent MFT in use = 52 %
Total MFT fragments = 2

--------------------------------------------------------------------------------
Fragments File Size Most fragmented files
9,386 1.00 GB \Backup-to-Disk 2\B2D000157.bkf
6,893 1.00 GB \Backup-to-Disk 1\B2D000122.bkf
6,725 1.00 GB \Backup-to-Disk 1\B2D000137.bkf
6,170 1.00 GB \Backup-to-Disk 1\B2D000132.bkf
6,149 1.00 GB \Backup-to-Disk 2\B2D000145.bkf
6,079 1.00 GB \Backup-to-Disk 2\B2D000144.bkf
5,987 1.00 GB \Backup-to-Disk 1\B2D000133.bkf
5,842 1.00 GB \Backup-to-Disk 2\B2D000155.bkf
5,655 1.00 GB \Backup-to-Disk 1\B2D000129.bkf
5,608 1.00 GB \Backup-to-Disk 1\B2D000134.bkf
5,522 1.00 GB \Backup-to-Disk 1\B2D000125.bkf
5,414 1.00 GB \Backup-to-Disk 2\B2D000142.bkf
5,079 1.00 GB \Backup-to-Disk 1\B2D000131.bkf
4,718 1.00 GB \Backup-to-Disk 1\B2D000127.bkf
4,453 1.00 GB \Backup-to-Disk 1\B2D000124.bkf
4,413 1.00 GB \Backup-to-Disk 2\B2D000150.bkf
4,372 1.00 GB \Backup-to-Disk 2\B2D000143.bkf
4,316 1.00 GB \Backup-to-Disk 2\B2D000156.bkf
3,527 1.00 GB \Backup-to-Disk 2\B2D000148.bkf
3,497 819 MB \Backup-to-Disk 1\B2D000138.bkf
3,311 1.00 GB \Backup-to-Disk 1\B2D000126.bkf
3,306 1.00 GB \Backup-to-Disk 2\B2D000151.bkf
3,131 1.00 GB \Backup-to-Disk 1\B2D000121.bkf
2,835 1.00 GB \Backup-to-Disk 2\B2D000154.bkf
2,611 1.00 GB \Backup-to-Disk 1\B2D000136.bkf
2,285 1.00 GB \Backup-to-Disk 1\B2D000128.bkf
1,837 1.00 GB \Backup-to-Disk 1\B2D000123.bkf
1,565 1.00 GB \Backup-to-Disk 1\B2D000130.bkf
785 1.00 GB \Backup-to-Disk 2\B2D000149.bkf
770 1.00 GB \Backup-to-Disk 1\B2D000140.bkfMessage was edited by:
Charles Graham

[Ross_Smith]

Thanks for that Charles, it's useful to know it's not an isolated problem.

I've now found that this problem mainly occurs on our Windows powered NAS devices, which run a cut down version of 2000 server. A home-made NAS box running XP workstation doesn't show anywhere near this level of fragmentation.

Our NAS boxes are both from the Maxtor MaxAttach range, and there are articles stating that they cannot be patched beyond SP2. I've now found an article on the MS knowledgebase which seems to indicate there were fragmentation problems with Windows 2000 SP2:
http://support.microsoft.com/default.aspx?scid=kb;en-us;297268

I think this may be relevant to this problem, can you let me know what operating system is running there on your B2D target?

Ross

[Renuka_-]

Hello,


What i feel you can try is to prevent too many .bkf files being created for each backup, go to the backup to disk properties and under the configuration tab increase the value for 'Maximum size for backup to disk files' which will create larger and leeser files/backup.

Secondly you may also limit the number of backup files backup to disk file.

This will ofcourse depend on the average size of your backups, the frequency of backups and frequency of the files being overwritten as in the retention period desired by you for each of your backups.

Hope this helps, do revert with your views on the same.

NOTE : If we do not receive your reply within two business days, this post would be marked assumed answeredand would be moved toanswered questions pool.

[Bob_Babcock]

I just checked 2 of our USB disks - first time I've ever seen the fragmentation display use only red. But we have typically 50 fragments per file, not thousands, and no evidence that it's causing problems. The disks are full enough that defrag says it won't work well.

[Ross_Smith]

As I have stated before on this thread, I am not looking for an answer from Veritas, this thread is purely to gauge how many other people are also seeing this problem.

The issue is not a question of too many .bkf files being created, it's a case that as each .bkf file is created it is being fragmented for some reason.

If you refer to the Veritas knowledgebase, you will find that changin the maximum size for backup to disk files is not recommended for network backups. The maximum recommended size is 2Gb. We currently have this set to 1Gb and have a 574Gb drive.

I am also not likely to be restricting the number of backups per backup to disk file since we don't allow backups to append to files so there is only ever one backup per file.

Ross

[Ross_Smith]

Bob, what operating system are you running there? I've only seen this on Windows 2000 SP2 so far, I assume you'll be more up to date than that if you're using USB disks?

I raised it with Veritas Advanced Support a few weeks back, they also were not sure whether this would indicate a problem or not, I had the impression it was something they'd not seen before.

I would agree that it doesn't seem to cause any problems. Now I'm aware of this all I'm going to do is keep a closer eye on the data transfer performance of our NAS boxes, and invest in some defrag software if necessary.

Ross

[david_chaika_2]

We have a similar problem with 1GB backup file fragmented into 4000 parts, Windows 2003.

[Bob_Babcock]

> Bob, what operating system are you running there?
> I've only seen this on Windows 2000 SP2 so far, I
> I assume you'll be more up to date than that if
> you're using USB disks?

XP workstation, 400GB USB drives. As an experiment, I just deleted the backup that would have been overwritten tonight to free up some space and I'm running a defrag. It's going to take a long time to defrag a disk this large, and since Windows doesn't defrag free space, I fear future backups will fragment too.

[ray_littlefie1]

I think this is more of a NTFS format issue than with the way Veritas writes files to the disk. It would be interesting if you tried to just copy some of your BKF files from one location to another folder on the same disk.
One way to reduce fragmentation is to increase the cluster size (default allocation) when creating (formatting )the partition. Bigger clusters work well for writing/re-writing big files.

[Ross_Smith]

I beg to differ Ray, I agree that NTFS has it's part to play, but I think it's the way Veritas are writing to the file system that's causing these problems.

It varies for each case, but if you do the math on the numbers of fragments per file, Veritas' .bkf files are being fragmented approximately every 250k. That is rediculous for a 1Gb file, especially when there are no other transactions to the file system as that file is being written (BackupExec here is set to write to one file at a time, and there are no other operations targetted at this device.).

The KB article I referred to earlier may point to an explanation. I suspect that Microsoft did not update NTFS when they fixed that, instead I suspect they updated the way they restore files when they discovered the problem.

As an example of how NTFS does not need to be fragmented like this, we have an exchange server here with a turnover on the log drive of around 500Mb per day. That drive has almost zero fragmentation. At the moment it contains 427 files, only one is fragmented. That one file is the main database, and contains 17 fragments.

In comparison, our NAS device now contains 2,368 files split into 2,644,060 fragments. Even if you do the math to give a fair comparison, that would mean 476,779 fragments per 427 files.

476,799 fragments with BackupExec compared to 17 with Exchange!! This is not a NTFS problem.

I suspect the difference is that exchange allocates log files in 5Mb chunks. Every single file is created exactly the same size, and data is only written into that file *after* the file has been created. This means that each file is created with no fragmentation and does not become fragmented as data is written. In contrast, BackupExec expands the file as it goes, writing a constant stream of tiny updates to the file system.

This is what happens when you fudge a tape driver to allow you to write to NTFS without putting any real thought into the long term effects. The contrast between the professionalism of the Microsoft solution and the quick bodge favoured by Veritas could not be more apparent.

Ross

[Phil_Bryan]

I'm first backing up to an EMC Clarion fiber attached disk array then duplicating to an HP tape library. I'm also get horrible fragmentation. I expected faster backup to disk but backup to tape is actually faster. I'm actually using large b2d files so that only 1 file is created for each job and I still get fragmented drives. Here's my frag report on the SAN array.
Volume SanDisk (S:):
Volume size = 1,499 GB
Cluster size = 4 KB
Used space = 935 GB
Free space = 564 GB
Percent free space = 37 %

Volume fragmentation
Total fragmentation = 49 %
File fragmentation = 99 %
Free space fragmentation = 0 %

File fragmentation
Total files = 91
Average file size = 15,453 MB
Total fragmented files = 10
Total excess fragments = 1,967,169
Average fragments per file = 21618.24

Pagefile fragmentation
Pagefile size = 0 bytes
Total fragments = 0

Directory fragmentation
Total directories = 40
Fragmented directories = 2
Excess directory fragments = 17

Master File Table (MFT) fragmentation
Total MFT size = 23,248 KB
MFT record count = 9,299
Percent MFT in use = 39 %
Total MFT fragments = 4

--------------------------------------------------------------------------------
Fragments File Size Most fragmented files
3 12 KB \EMC_RM\SQL
4 17 KB \Changer.cfg
594,179 367 GB \Backup-to-Disk Folder 1-Weekly\B2D000249.bkf
473,985 152 GB \Backup-to-Disk Folder 2-Weekly\B2D000250.bkf
93,787 28,011 MB \Backup-to-Disk Folder 3-Weekly\B2D000251.bkf
589,314 171 GB \Backup-to-Disk Folder 4-Weekly\B2D000252.bkf
54,602 43,780 MB \Backup-to-Disk Folder 1-Daily\B2D000253.bkf
103,191 146 GB \Backup-to-Disk Folder 2-Daily\B2D000254.bkf
19,926 10,478 MB \Backup-to-Disk Folder 3-Daily\B2D000255.bkf
26,763 15,118 MB \Backup-to-Disk Folder 4-Daily\B2D000256.bkf
11,428 3,022 MB \Backup-to-Disk Folder 5-Daily\B2D000257.bkf

[Lars_Lange]

Hi all,

I have the same problem. Due to wipping the disk drive i can not provide the exact numbers. But close.

We use a 1,45 TB disk drive to store the backup files. After 3 months usage the backups are running so slow that we are desperate for a solution.

The media server is a Win2k server, with all updates. The TB drive is raid 5 (Intel hardware all the way) . The baddest 1 gb file had more then 13.000 fragments and in generel all the backup files were between 8000-10000 fragments, and this drive is _only_ used to store the Veritas backup files. So this must be a matter that Veritas must handle. We tried to run the OO Defrag server edtion software, after 36 hours only 34% of the drive was defraget! And this was only with approx. 400 gb on the entire drive.

We use Veritas ver. 10.0 Rev. 5484

/Lars
"This message was transmitted on 100% recyclable electrons"