Catalog backup size not reduced after catalog compression

[nick80]

Hi,

Our catalog backup is about 98GB in size and runs for about 3 hours. We are looking at ways of speeding this up and have gone down the compression route. (Netbackup 6.5.0.6 - Windows 2003 R2 Master)

I have appled catalog compression using the recommended "staggered approach" and we have now compressed all images older than 32 days and we have seen the db/images folder reduce in terms of the size on disk from 92GB to 56GB. The problem is that the Catalog backup itself is stil at 98GB and has actually gone up in duration to 3 and half hours. (I know the increase could be dependant on how busy the environment is but sure we should at leaset have seen the backup reduce in size since the compression)

I know its not a complicated process but just want to make sure I havent missed something :

>> Ensure catalog backup isnt running

>> Set catalog compress interval

>> run bpimage -cleanup -allclients

>> Check the db/images folder on completetion and can see a reduction of size on disk and image files in blue to indicate compression.

I cant understand why the catalog backup is still 98GB. Maybe I need to restart Netbackup ? I dont want to suggest this unless I am pretty confident it will resolve the problem as we have to force a window and puts us a behind a little with Vaulting.

Has anyone had something similar and can you advise?

Thanks

Nick

[Mark_Solutions]

Lots of things to say here but let me summarise ...

1. 6.5.x is way out of support so you really need to upgrade soon

2. It is best if you are going to upgrade not to have your catalog compressed

3. catalog compression is not usually reccomended - i would advise you uncompress it again using the correct proceedure

4. When you backup compressed files they will uncompress during the backup and so will make the backup size exactly the same and will actually slow down the backup

Trust me - many people would love their catalog to be only 98GB - find out where your bottleneck is (network, buffer tuning, disk fragmentation etc.) to help speed you catalog backup up but dont compress it and get upgraded as soon as you can

Hope this helps

[nick80]

Hi Mark, Thanks very much for you advice.

1 ) Yep we have another environment which we are currently migrating over to. (Im new here)

2) n/a

3 and 4) I appreciate its a small catalog and I have an idea where the bottleneck is (master doesnt have access to tape so backing up over ip) The problem however is that the tape library is quite flaky (Media write errors) and because we are moving away from this environment they no longer have hardware support and we were looking for a way of speeding up the backup to give it a better chance of finishing befor an error occurred.

Picking out the second point from the below technote surely this goes against the point you made about it uncompressing the files as part of backing them up ? 

Compressing the image catalog accomplishes the following objectives:

• Reduces greatly the disk space that is consumed.

• Reduces the media that is required to back up the catalog.

http://www.symantec.com/business/support/index?page=content&id=HOWTO68018&actp=search&viewlocale=en_...

[Mark_Solutions]

I see what they say in that tech note .. but not sure I agree with it - in my experience compressed files get uncompressed when they are backed up which makes the backup slower but still uses the same amount of media ... perhaps i am wrong?

[mph999]

How about catalog archiving.

This backs up the .f files and then deletes them.

Just be sure to make a duplicate of the tape(s) used for the archiving (just in case ...).

M

[jim_dalton]

Your version is old BUT dont forget the catalog might not just consist of the images, theres other stuff like the BMR db and theres a know 'feature' with BMR db that requires one to  fix it up. It can take a while to just verify that fella. This might explain some of the time taken. You say writing over the network...if its not 1Gbit then its not going to be too good...I'd hack  together some kind of tape setup if I could. Depending on current solution the compress may not change the media used since if its compressed once already your tape drive with built in compression wont be able to repeat the feat.But net traffic will be less.

Jim 

[Mark_Solutions · Accepted solution]

This is something you may need to test more as it depends on how things work with the catalog compression.

I have dug through my old notes for a customer i dealt with a few years ago who used compressed files on their systems (windows compression)

During the tests we backed up a folder containing their compressed files - it backed up 5 GB of data in 33 minutes

We then uncompressed the files and ran the backup again - this time it backed up exactly the same amount of data but it only took 11 minutes - so three times as fast.

This suggested that that type of compression caused the files backed up to be uncompressed in the backup stream causing a overhead on the backup and really slowing it down - and still reporting the full backup size

As those folders showed up a blue in Windows explorer the same as the catalog files do that you have then i assume it uses the same type of compression method and from that i assume it will do the same during the backup

I do not see that documented anywhere but i do have my test results to work from

Hope this helps

[Yasuhisa_Ishika]

I agree with Mark. Please look under images folder and check how files compressed.

I'm not sure how catalog files are compressed(I have never used catalog compression on Windows). If compressed with Window's built-in comptession feature, disk space is saved but amount of data served from file system to applications are same as before. Thus, performance will be degraded as compression and decompression is done inline each time when applications access to blocks in compressed files.

[sdo]

On Windows, when catalog compression is enabled, then NetBackup leverages NTFS native file system compression - so, probably not the fastest. If you view the folders and files they'll appear blue (if the option to display compressed file/folders in a separate colour is enabled). On a speed front - I have never had a catalog backup run very fast, be it from SAN or SSD direct to FC or SAS tape even - let alone across a network to a media server - and certainly never as fast as plain client backups. What are you seeing, about 18 MB/s for your catalog backup?

[nick80]

Thanks guys, I think im going to uncompress it as it hasnt reduced the amount of data written to tape which is what the documentation indicates.

Yet again the decompression docs leave me with a few questions about the process and im hoping some one can help ?

http://www.symantec.com/business/support/index?page=content&id=HOWTO86598&actp=search&viewlocale=en_...

1 ) Is there no way of staggering the decompression by days (I know you can do it by client using bpimage -decompress but at the same time i have read in another tech note that it is not recommended to compress/decompress manually usaing this command)

2) We have vault/duplication jobs running continuously, when i stop the request deamon service will these continue.

3) Once I have set the "Set catalog compress interval" to 0 and ran bpimage -decompress -allclient or a specific client, can I then restart the request deamon or does it have to remain down until the decompress has finished?

4) How will I know when its finished and will it take as long as it took to compress in the first place. (I split the compression into about 4 different time intervals (each taking about 25 minutes) so if its going to do the same again then i potentially have to arrange a quiet period of 2 hours for the decompression to happen without causing issues to backups/duplications)

Unfortunately this all has to be done on live so i cant test it. And I need to make sure I dont do anything that effects the duplication jobs. Hope you can help

[Mark_Solutions]

To be honest it is not a huge catalog so i would find that bit of quiet time and go for it

Another thread on here did the same thing and it uncompressed very quickly - cant see the thread at the moment

[nick80]

OK Thanks, but can I leave the duplications running ?

[Mark_Solutions]

As i said i would do it at a quiet time - nothing running - just put them on hold for a while while you get it done

[sdo]

Just a final point - someone mentioned this above too, but I'll add to it... because the compression on Windows is actually native NTFS file system, when anything reads the compressed folders and files they get uncompressed hence the catalog backup landing zone (tape or disk) is never as small as the compressed catalog appears to be on Windows disk. However, my understanding is that catalog compression in unix/Linux land is accomplished using gzip - thus catalog backups on target are smaller because they are not uncompressed on the fly by the file system stack.

[mph999]

Close, I think unix/ linux uses the compress command.

The compressed files have a .Z prefix, which is 'compress' not 'gzip'.

[sdo]

:p Clarity always being the end point. Thanks for that. Now an entry in my notes. :)