netbackup incremental slow - but full & catalog fast

[bradley]

I am looking for ideas why the incremental backups on local hard drives are slow - taking 75 min to backup a few gig while our full and catalog backups run quite fast.  Data transfer rates are measured in K per second, not megabytes.

We have two systems setup with near identical hardware / software platforms.  The slowness is only observed in one of the two computer centers.

Data to be backup - local on a 3511 raid boxes – no network access is required to retrieve data.

OS: Solaris 8

hardware: v240 (rack mount)

netbackup 6

The netbackup catalog location is on internal Sun hard drives – mirrored.

The total amount of space used on 3511 - approx 8 - 9 terabytes - most of the data is in "long term" storage and not accessed once written.  No errors were reported on hard drive arrays when using Sun Storage tools

We have imported the policies from the main computer system which has no issues backing up.  So the problem does not seemed to be policy related.

No error messages were observed in /var/adm/messages - - none were expected because full backups run "fast" on the same hardware.  Tapes have been swapped out, drives cleaned with no improvement in overall speed.

Data is staged to the local drives - using custom scripts which run approx once every 10 min - with a maximum time to backup approx 5 or less mins.   

Brad

[Ver-Cap]

Is the incremental backup defferential??

Is it online catalog backup ?

How frequently data is accessed..?

Best Regards,

Kitty

[bradley]

backup should not be defferentail - the files themselfs are not updated, new versions are created so a defferential would not speed things up.

Is it online catalog backup? - not sure of the question - but the catalog is online on higher speed drives - which is where the OS boots from.

Incremental backups are run once an hour - catalog backups are run once per week.

Once data is written to the hard drive the backup is the next time the file is read.

However; the information can be read via an nfs read only mount to audit the data.

This task is very infreq - might happen once a month.

Data collection is performed on a 6 cron jobs -- running approx 10 mins apart. 

00:00  -- 5 servers are inspected - new files are staged to the 3511

00:10 --  5 additional servers are inspected - new files are staged on the 3511

.

.

.

00:50 -- last 5 servers are inspected -- new files are staged on the 3511

The real odd thing is - we have two data centers - and the slow backups are only observed in our

backup site.  The amount of data that is logged is very small vs the main complex.  I was wondering if the problem is some how related to the netbackup catalog.  Both systems went online about 1.5 years ago.

brad

[bradley]

Correction on the catalog backups - they happen once an hour after the incremental has finished. 

I should do more research and figure out a better method to backup this catalog and move the backup from tape to an empty partition.  No use bogging down the tape drive unless I need to move the files off site.

Brad

[Ver-Cap]

Brad thank for detailed info....

In current schenerio the full backup has no problems running and only incremental goes slow then it can be difficult to trace down where exactly the issue is observed as the data backed up is locally.

Hence i would suggest you couple of alternates which migh provide indepth clarity on the issue:

1) Select the incremental backup policy and Right click on the policy --> select copy into new policy then fill in the required attributes on in the tabls then file a incremental backup and just make record of the data backed up flow (KBPS) and the time taken to complete.

2)Enable logging for both incremental and full backup and trigger the backups.Check for any details that has specific time interval details.

Best Regards,

Kitty