Netbackup Catalog online full backup is very slow

[ajalboush]

I have netbackup 7.1 installed on windows 2003 server, when I do catalog full backup i suffer from bad performance, about 280 GB took about 18 hours, any help please. 

[Mark_Solutions]

That is quite slow but a little more information would be useful.

Does the Master perform this or is the data passed over the network to a Media Server.

It is basically just a backup like any other so the BUFFER tuning, paging memory etc. all applies to the catalog backup along with the fragmentation of the drive and network performance (if going to a media server)

So where is the backup going to (disk or tape) and is it local or to a Media Server - plus what tuning have you done for NetBackup

[mph999]

What part is slow, the staging of the DB, or one of the actual backup jobs ?

Is the media serevr also the master server.

Please enable bptm log on the media server and bpbkar process on the master.

Allow the backup to run and finsh (it has to finish).

Look in the log for lines like these :

bptm:

log.011112:09:36:33.951 [28257] <2> io_init: child delay = 10, parent delay = 15 (milliseconds)

log.010912:14:39:48.500 [18546] <2> write_data: waited for full buffer 112 times, delayed 1275507 times

bpbkar 

log.011112:09:36:45.730 [28256] <2> bpbkar write_eot: JBD - bpbkar waited 0 times for empty buffer, delayed 0 times

For the media server that creates the backup - does it run any other backup jobs, and, are these 'fast' ?

Can you create a test job, where the media server backs up itself to it tapes drive / disk stoage unit - how fast is this.

Thanks,

Martin

[ajalboush]

To Mark_Solution:

Thank you,

I use the master as media server for itself, and  I send the backup to VTL (datadomain), i use the follwoing values:

NUMBER_DATA_BUFFERS = 64

SIZE_DATA_BUFFERS = 262144

most of tim is on backing up the follwoing path: /D/VERITAS/NetBackup/db/images/*

To Mph999:

the actual backup is slow, yes i use the master as a media for itself, and this master server only do backup for itself with two dedicated tape drives. 

[ajalboush]

the actual backup is slow, yes i use the master as a media for itself, and this master server only do backup for itself with two dedicated tape drives. 

[ajalboush]

i Have uploaded the log file that contains:

bptm:

log.011112:09:36:33.951 [28257] <2> io_init: child delay = 10, parent delay = 15 (milliseconds)

log.010912:14:39:48.500 [18546] <2> write_data: waited for full buffer 112 times, delayed 1275507 times

[Mark_Solutions]

This is showing poor performance

Your buffers look fine (assuming they are on the Master in \netbackup\db\config directory)

So it is down to either fragmentation of the disk (D drive in yoru case) or anti-virus scanning the files as they are being backed up.

Make sure AV excludes all NetBackup folders and processes.

Also make sure you are OK for Memory and you paging file is tuned:

HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management\

 DWORD - PoolUsageMaximum  - Decimal value of 40

 DWORD - PagedPoolSize Hex value of FFFFFFFF (this is 8 x F)

Both need a reboot to take effect

Hope this helps

[ajalboush]

tHANK YOU, I WILL CHECK AND UPDATE YOU. 

[ajalboush]

i attahced the d drive fragmentation report, please check it. 

[Mark_Solutions]

Looks pretty fragmented to be - and some big image files there!

You have a few images files that are up to a GB in size with over 16000 fragments

16,804          1.03 GB         \VERITAS\NetBackup\db\images\fleetctl-app\1314000000\catstore\MASTER_MSWINDOWS--fleetctl-app_1314790861_FULL.f_imgRecord0
16,788          1.03 GB         \VERITAS\NetBackup\db\images\fleetctl-app\1313000000\catstore\MASTER_MSWINDOWS--fleetctl-app_1313269176_FULL.f_imgRecord0
16,786          1.02 GB         \VERITAS\NetBackup\db\images\fleetctl-app\1307000000\catstore\MASTER_MSWINDOWS--fleetctl-app_1307278881_FULL.f_imgRecord0
16,741          1.02 GB         \VERITAS\NetBackup\db\images\fleetctl-app\1309000000\catstore\MASTER_MSWINDOWS--fleetctl-app_1309081672_FULL.f_imgRecord0
16,693          1.03 GB         \VERITAS\NetBackup\db\images\fleetctl-app\1322000000\catstore\MASTER_MSWINDOWS--fleetctl-app_1322488880_FULL.f_imgRecord0
14,472          904 MB          \VERITAS\NetBackup\db\images\jawbilling\1261000000\catstore\JAWBILLING_UNIX_1261767602_FULL.f_imgRecord0

The summary is fairly clear too:

Volume fragmentation
    Total fragmentation                        = 50 %
    File fragmentation                         = 97 %
    Free space fragmentation                   = 3 %

If you have any available down time it would be good to defragment it - just be careful of the timing when doing it - wouldnt want to do it during a Catalog Backup or busy backup window

Did you check AV?

[mph999]

OK, 

1275507 x 15ms  = 19132 seconds

= 5.3 hours

So, for 5.3 hours, the 'backup ' (well, bptm/ media serever part) is doing nothing, while it waits for the cliemt side of the backup to send data.

As this is a master/ media server,we can more or less discount/ ignore a network issue, so this leaves one of two things (probably)

You are unable to read the data off the disk quickly as the server is busy writing / reading the catalog

or

You are unable to read the disk quickly due to a 'config' issue with the disk or filesystem

It would be good to run a test backup off the server, using a separate area when the server is not too busy - how fast does this go.

Martin

[mph999]

Hmm, quite bad - if you defrag this, you will need downtime and must stop NetBackup whilst you do it ...

(Yes, you probably could leave it running, but I hate to think of what could happen ...)

Could be worth considering to 'Archive' the catalog to make it smaller - this writes the xxx.f files in the catalog to tape, then deletes them from the server.  If you need to restore, the .f file is restored from the tape first (manual process, but easy ...).

Martin

[ajalboush]

i disabled the antivirus now. 

how the archiving can be done? i need this becuase i will migrate the master to new hardware.

[mph999]

The suggestion of archiving was meant as a 'suggestion' not necessarly a fix.

Depending on the cause - filesystem problem/ disk config issue - archiving the catalog would only be a workaround, as the real issue is still there.

The correct course of action is to fully investigate the cause, and fix if possible.  Then, you may consider archiving.

Archiving is covered in the NBU Admin manual :

Here is a link to the 7.1 manuals :

 http://www.symantec.com/docs/TECH154178

For exmple, in the manual :

NetBackup Administrator's Guide for UNIX and Linux, Volume I  

http://www.symantec.com/docs/DOC3650

Archiving the Catalog is found on page 759.

You should make multiple duplicateions of these tapes, kept in separate places.

Regards,

Martin

[ajalboush]

what do you mean by test backup off the server? 

[mph999]

Run a backup of the master server, but not the catalog area.  I want to see how fast it can go ...

Hence my previous question, does the master server run any other backups, and, are these fast ?

Thanks,

Martin

[ajalboush]

the master is media for itself now, if you wish i can run backup using the master as media also i will do the test using another media server.

[mph999]

Yes, excellent idea. Martin

[ajalboush]

HI All,

i did many tests,

1. I did backup for G:\ drive (D is for Veritas), the backup is fast when using the master as a media and when using anohter server as media.

2. I did backup for the D drive using he master as media and using another media both are slow.

3. I did catalog backup using the master as media and using another media, both are slow.

So I concluded that the backup become slow when starts taking backup for D:\veratis\netbackup\db\....

[Mark_Solutions]

Looks like you need some down time to do the defrag then - as we said previosuly it would not be a great idea to do it whilst NetBackup is active

After all, according to the defrag report it is the images directory that is in the worst state.