Low Compression Ratio

[Daniel_Bennett]

Hi all,

I am hoping that someone can help. We are having a problem with backups and are getting a compression ratio of less then 1:1 typically around 0.98:1 using hardware compression and LTO2 tapes (HP drive).

Nothing has changed in the server or backup setup. Two weeks ago every backup would run through at between 1.25:1 and 1.3:1 and all of a sudden that has ceased.

I have visited the forums and seen this topic come up but I can't see a solution here. I have tried updating to the HP driver. Installed Veritas updates and also cleaned the tape drive. Yet there has been no improvement.

My next step will be to call in the supplier to check for hardware issues unless I can find a fix for this problem.

Cheers and thanks in advance.

Daniel Bennett

[Daniel_Bennett]

Further information that I didn't post originally is that we are using BE 9.1 on a Windows 2003 server.

Hope this helps.

Thanks,

Daniel.

[Amruta_Bhide]

Hello,
Hardware compression is done by the device itself. When you use hardware compression option on Backup Exec (BE), BE commands the hardware to go ahead with the compression. However the hardware compression depends on the device. The device should support Hardware Compression. We advice you to contact your Vendor for more information of the same.

Software compression is done by BE. It is said to achivachievepression of 2:1 but that ratio is hardly ever achieved. Compression can be highly variable depending on your input data, as the software compression depends on the type of file or data you are trying to back up. For example: Already compressed files cannot be compressed any further. And the compression for normal '.ttext files will be more than the compression of '.gif' or database files.


You can refer to the following article to understand this concept better.



http://seer.support.veritas.com/docs/199542.htm

-------------------------------------------------------------------------------------
You can also go through the following article which will help to check that the data is being properly written to the media.




http://seer.support.veritas.com/docs/246958.htm

******************************************************************
*****************************************************************

Note : If we do not receive your reply within two business days, this post would be marked ‘assumed answered’ and would be moved to ‘answered questions’ pool.


Thanks.

[Thomas_Bianco]

low compression ratios are not uncommon if the input data is sufficiently random (i.e. complex data or encrypted files) or sufficiently dense (i.e. compressed data, such as zip, compressed video or audio, etc).

a ratio of .98:1 probably means that your data is precompressed, and there is a ~2% overhead for the compression headers.

great source for theoretical information on compression http://www.data-compression.com

[Daniel_Bennett]

"Hardware compression is done by the device itself."

I understand this

"The device should support Hardware Compression."

It does

"Compression can be highly variable depending on your input data, as the software compression depends on the type of file or data you are trying to back up."

Complete Agreeance here. The files are typical office files. Word, Excel, AutoCAD files. Yes there are PDFs and ZIP files and but these would maybe make up 25% of the data. The Shared NNetwork drives are not compressed.

"You can refer to the following article to understand this concept better..... http://seer.support.veritas.com/docs/199542.htm "

Been there read it check / confirmed it. Hardware compression is enabled on both the Job Properties and the Device driver.

"You can also go through the following article which will help to check that the data is being properly written to the media."

Not much help really, I know the data is ok I can restore from it and use it.

"low compression ratios are not uncommon if the input data is sufficiently random "

I realise I probably will never achieve 2:1 i'd settle for the 1.3: we were getting before.

"a ratio of .98:1 probably means that your data is precompressed, and there is a ~2% overhead for the compression headers."

Again the data is not pre-compressed (can you give me a way to confirm this anyway).

As previously stated... Backups were working with a compression ratio of about 1.3:1. Nothing has chaged, the selection sets are the same as they were before. Hardware compression is enabled on both the Job properties and the device driver.


I used the HP L&TT software and performed the Compression Test. The result was 0.71:1. I have a tech coming in this after noon to look at the drive. I'll post the results.

I am also testing indidual shared drives to determine if I can get compression from any of them. The first drive that has almost nothing but Excel and word files, maybe 10% jpegs yeilded a 1:1 compression ratio.

Daniel.Message was edited by:
Daniel Bennett

[Daniel_Bennett]

Well I have been doing a lot of testing and had a tech in. Still no closer. So here are the results for all my testing, please note that this is lifted from emails I have been sending to people so there may be a few places where it doesn't flow coherently.....

[Daniel_Bennett]

Just to keep all updated I am adopting the following short term strategy to managed backups: Monday night will run a full backup of the server, this will spill over to Tuesday before the job is complete. I will then run Incremental backups every night (including the weekend) to pick up any changed files. These backups will fit on a single tape and will therefore complete within a couple of hours. I will martian this strategy until the problem is resolved.

I have also done a bit more testing and reading. I heard about the HP L&TT (Library and Tape Tools) software for the drive which has a test specifically for Hardware compression. According to the literature

"This option will test the hardware compression capability of your tape drive. You will need a writable tape to complete this test. A file is created on tape that is designed to have a compression ratio of approximately 2:1, anything less than 2:1 could mean that your hardware compression is disabled."

I ran this test several times and got a Compression Ratio of 0.71:1 every time. Unfortunately you can't seem to export the test result in any way (other than copy each individual line) which is a bit limiting. Greg or Justin could you run this test and let me know what result you get?

Also yesterday the Tape drive went off-line and I could only get it back on line by rebooting the server. This is the first time this has ever happened.

Finally I am going to commence a trail of data compression on each different volume that we backup as I have had a couple of results that have had small compression ratios greater than 1:1.

[Daniel_Bennett]

The tech for the tape drive should be here around 3pm (Darwin). In the meantime I have undertaken a test to determine what volumes if any are getting file compression.

The procedure I used is identical for each job and is as follows:
Make a Standard Selection set called "Testing"- 1st time only
Make a Backup job using the "Testing" Selection set - 1st time only
Erase the contents of the tape
Modify the Selection set "Testing" for the necessary a particular volume
Run the "Testing" Backup job.
View the and record the results.
Repeat from step 3.

In this manner each job should be Identical. The results are as follows:

Test VolumeSizeCompression RatioTimeComments
================================================================================
Admin15GB0.998:10:31:11
Drafting3.6GB1.57:1 0:23:44Took a long time to start
PPI3.8GB1.42:10:08:14
Programs18.9GB0.86:10:21:16
Projects133GB0.969:13:26:56
Standards2.6GB1.39:1 0:27:16Took a long time to start
User41GB0.913:11:03:22Took a long time to start - 30 min
Lotus22GB1.06:10:14:55Started straight away

Interestingly 3 jobs did get good compression, they also both took a long time to start.... Any comments?

[Daniel_Bennett]

Tech from CSG has been and gone, though my problem remains. I think he is leaning towards the data being the Issue. I am not convinced. I will concede I did make an error when I recorded some test results earlier. The PPI line should read 1.42:1 compression ratio which is mighty fine. I'm going to re-run the test for the Admin drive and recheck it's compression ratio too.

However I showed him and I will state it here clearly:

The last good backup with a compression ratio of greater than 1:1 (1.28:1 actual amount) was done on 242 GB of data. The next 3 backups failed to record any data why I don't know. The next backup was 246 GB of data the compression ratio was 0.98:1. What could possibly change in a measly 4 GB of data to have such a dramatic affect?

The next test I will undertake is to try and prove that the data content is innocent. To do this I am going to restore the entire U:\ drive. This is about 41 GB of data and yielded a compression ratio of 0.913:1. I will then use Winzip to compress this data and I bet my left nut that we can get around 80% (1.2:1) compression on it.

Any way I am no closer to a solution than I was two hours ago and I feel I taught the tech more about Veritas and Tape drives than he did me. His only other idea at the moment is to talk directly with HP and or Veritas (he will ask another guy but I don't hold too much hope).Message was edited by:
Daniel Bennett

[Daniel_Bennett]

Alright just finished this test and I don't recommend undertaking it using a simple program like a freeware version of Power Archiver! It was just to darn Laborious. (it would appear that zips should be kept under 1GB or somewhere there abouts). Anyway the results are in.....

User drive ZIP compression test:
VolumeSizeSize on Disk
==========================================
Users37.7 GB37.8 GB
Users (zipped)29.5 GB29.5 GB
==========================================
Compression Ratio1.28:1 or 78%
==========================================

Therefore as many JPEGs and MP3s etc as there is on this volume (some people have collections of 2+ GB and I think this will be an additional issue to address, I hope they are not being played from there) there is still a lot of room for compression. (and I get to keep my left nut).

I re-ran the Admin drive backup to check the result and it was pretty much spot on. The new figure is 0.998:1.

I forgot to mention earlier that there may be a possibility of getting loan drive to test the backup and compare the results. Other than that have you guys any ideas yet because I am all out?

[Daniel_Bennett]

Refer next postMessage was edited by:
Daniel Bennett

[Daniel_Bennett]

Let me apologise in advance if I sound a negative here at all.


I think I have probably read most of these links over the last week, but I'll check them all later today anyway.


"The fact that some volumes are seeing a successful compression ration and others are not, tends to make me think that the problem isn't with the tape drive itself. "


Hmmm.... but why using Hardware Compression could I only get a compression ratio in the user volume of 0.913:1 and using common freeware software I was able to get 1.28:1 compression on the same data. Fair enough there would be some difference in performance but we are talking about a 140% gain over using the Hardware compression!


I also thought that maybe it would be a good idea to validate the software compression after my above rant. So here is the results for the PPI drive, if you recall I was happy with the compression ratio here at 1.42:1.

PP drive ZIP compression test:
VolumeSizeSize on Disk
================================================
PPI3.79GB3.81 GB
PPI (zipped)2.58GB2.58GB
================================================
Software Compression Ratio1.48:1 or 68%
Hardware Compression Ratio1.42:1 or 70%
================================================

As I suspected they are in the same ball park with only a 2% difference between the two. I might add that the drive also contains many jpegs (about 0.75 GB found with a quick search)

Furthermore and this is the clincher for me how can a 4 GB change in 246 GB a 1.6% increase in data size make such a difference to compression? If that's the case this could happen to any other office over night and unless you can find the file causing the problem every backup you are ever going to do from that day on is screwed. We are talking about going from 80% compression one night to 2% expansion the next when 98% of the data is unchanged.

"I am keen to see the results of using a different tape drive."

Me too.

If you have the means we could subject the project drive to the same software compression test I have already used, But I need something more robust that Power Archiver to handle the 130 GB of data. I'll bet my left nut again that I can achieve 80% or 1.2:1 compression.

And finally I don't know but I hope this is a big clue..... Sometimes at the moment jobs seem to be taking a long time to start up to 30 or 40 minutes. Usually I have data being written within 2 minutes. I have a restore going at the moment that has been idle for 40 minutes, normally these are done and dusted in 10 minutes.Message was edited by:
Daniel Bennett

[Daniel_Bennett]

All right lets make it fun. If compressing data that is already compressed should make a larger back eg have a compression ratio of less than 1:1 lets test it.

Since I spent 5 hours zipping things up last night to see what compression I could get lets back that up and check the result. Interested, me too!

[Daniel_Bennett]

Alright more test results.

Test VolumeSizeCompression RatioTimeComments
======================================================================================================
PPI3.8GB1.42:10:08:14
PPI Zipped2.59GB1.02:10:13:39
User41GB0.913:11:03:22
User zipped29.6GB0.770:10:50:53
================================================================================================

The PPI drive actually retained compressed albeit it's a fraction over 1% but the plus is it doesn't expand. I guess not surprisingly the User drive volume continued to expand.

In hind sight this probably doesn't prove or dispel anything unless someone can make a different interpretation of it.

[Daniel_Bennett]

Still not answered. Posting to keep it active.

[Thomas_Bianco]

i'm keeping my eye on this, but it seems to be an issue with the tape drive. have you tried running a test backup with MSBackup? at the very least it would clear veritas from any current wrong doing.

[andrew_black]

Found this thread because I was having the same problem. As a bit of trouble shooting I went into BE (9.1) and changed the compression settings to software. Achieved compression went from 1:1 to 1:1.2. This would suggest the compression issue is at a hardware level not software.

[Daniel_Bennett]

Despite my sceptisism I am leaning towards data being the issue. However that is not to say that I am satisfied with the result or performance of the software / hardware combination.

I have been testing individual directories to see what compression I could achieve. On some directories I can achieve up to 1.5:1 on others it drops back to 0.9:1.

Through using the procedure I setup and outlined in an earlier post and elimination (select a directory split it into 2,3 or 4 logical blocks) I am honing in on the problem. (done about 30 or so different backup sets). I then repeat the process on folders that show bad compression.

The results are interesting. At the root level dividing up the directories gives an almost identical compression ration to the parent directory. ie making a comparison between the compression ratio of the parent directory versus a calculated compression ratio of the sum of the child directories (Bytes Written : Used Capacity from the Device/Media properties screen).

At first I was a little detered about the result but then thought maybe that is actually reflecting what is happening. The problem was for the first 3 levels I was finding the bad compression in the largest folders and hence the weight of that folder tainted the overall compression ratio.

Then I got down to the forth level.... My bad compression is starting to show up in smaller folders now and that shows that if I do the backup in blocks (absoultley not practical) that I would be starting to achive compression rations of about 1.1:1 (whilst still including the bad data) rather than 0.9:1. Big difference.

I am part way through the fith level now on one of the most suspect directories. I expect that when I finish this one the result will be staggering.

Andrew, How much data are you backing up? Are you able to try doing something similar? This would help validate my results if it can be replicated.

And in a long way around to answer your advice, I am not sure that longer that doing a software compression job will make a difference (I'll try anything at the moment though) but when time permits (IT isn't my only work priority) I'll give it a shot.

[Jake_Norman]

We are having the same problem and no, software compression really doesn't make a difference on the compression time. Unforunately, our servers still backs up in a viable amount of time so my boss doesn't see this as a big problem yet.

Jake
F&R

[Daniel_Bennett]

Luckily my manager is taking this seriously. Our backup has gone from a single tape in 3 hours to 2 tapes that takes about 8 or 9 hours as it can't start the second tape until I arrive in the morning to swap it.

I might cull the server a bit to reduce the problem over the christmas break.