01-11-2012 02:49 AM
I have netbackup 7.1 installed on windows 2003 server, when I do catalog full backup i suffer from bad performance, about 280 GB took about 18 hours, any help please.
01-11-2012 02:59 AM
That is quite slow but a little more information would be useful.
Does the Master perform this or is the data passed over the network to a Media Server.
It is basically just a backup like any other so the BUFFER tuning, paging memory etc. all applies to the catalog backup along with the fragmentation of the drive and network performance (if going to a media server)
So where is the backup going to (disk or tape) and is it local or to a Media Server - plus what tuning have you done for NetBackup
01-11-2012 03:03 AM
What part is slow, the staging of the DB, or one of the actual backup jobs ?
Is the media serevr also the master server.
Please enable bptm log on the media server and bpbkar process on the master.
Allow the backup to run and finsh (it has to finish).
Look in the log for lines like these :
bptm:
log.011112:09:36:33.951 [28257] <2> io_init: child delay = 10, parent delay = 15 (milliseconds)
log.011112:09:36:45.730 [28256] <2> bpbkar write_eot: JBD - bpbkar waited 0 times for empty buffer, delayed 0 times
01-11-2012 03:15 AM
To Mark_Solution:
Thank you,
I use the master as media server for itself, and I send the backup to VTL (datadomain), i use the follwoing values:
NUMBER_DATA_BUFFERS = 64
SIZE_DATA_BUFFERS = 262144
most of tim is on backing up the follwoing path: /D/VERITAS/NetBackup/db/images/*
To Mph999:
the actual backup is slow, yes i use the master as a media for itself, and this master server only do backup for itself with two dedicated tape drives.
01-11-2012 03:16 AM
the actual backup is slow, yes i use the master as a media for itself, and this master server only do backup for itself with two dedicated tape drives.
01-11-2012 03:28 AM
i Have uploaded the log file that contains:
bptm:
log.011112:09:36:33.951 [28257] <2> io_init: child delay = 10, parent delay = 15 (milliseconds)
01-11-2012 04:02 AM
This is showing poor performance
Your buffers look fine (assuming they are on the Master in \netbackup\db\config directory)
So it is down to either fragmentation of the disk (D drive in yoru case) or anti-virus scanning the files as they are being backed up.
Make sure AV excludes all NetBackup folders and processes.
Also make sure you are OK for Memory and you paging file is tuned:
HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management\
DWORD - PoolUsageMaximum - Decimal value of 40
DWORD - PagedPoolSize Hex value of FFFFFFFF (this is 8 x F)
Both need a reboot to take effect
Hope this helps
01-11-2012 04:16 AM
tHANK YOU, I WILL CHECK AND UPDATE YOU.
01-11-2012 04:20 AM
i attahced the d drive fragmentation report, please check it.
01-11-2012 04:32 AM
Looks pretty fragmented to be - and some big image files there!
You have a few images files that are up to a GB in size with over 16000 fragments
16,804 1.03 GB \VERITAS\NetBackup\db\images\fleetctl-app\1314000000\catstore\MASTER_MSWINDOWS--fleetctl-app_1314790861_FULL.f_imgRecord0
16,788 1.03 GB \VERITAS\NetBackup\db\images\fleetctl-app\1313000000\catstore\MASTER_MSWINDOWS--fleetctl-app_1313269176_FULL.f_imgRecord0
16,786 1.02 GB \VERITAS\NetBackup\db\images\fleetctl-app\1307000000\catstore\MASTER_MSWINDOWS--fleetctl-app_1307278881_FULL.f_imgRecord0
16,741 1.02 GB \VERITAS\NetBackup\db\images\fleetctl-app\1309000000\catstore\MASTER_MSWINDOWS--fleetctl-app_1309081672_FULL.f_imgRecord0
16,693 1.03 GB \VERITAS\NetBackup\db\images\fleetctl-app\1322000000\catstore\MASTER_MSWINDOWS--fleetctl-app_1322488880_FULL.f_imgRecord0
14,472 904 MB \VERITAS\NetBackup\db\images\jawbilling\1261000000\catstore\JAWBILLING_UNIX_1261767602_FULL.f_imgRecord0
The summary is fairly clear too:
Volume fragmentation
Total fragmentation = 50 %
File fragmentation = 97 %
Free space fragmentation = 3 %
If you have any available down time it would be good to defragment it - just be careful of the timing when doing it - wouldnt want to do it during a Catalog Backup or busy backup window
Did you check AV?
01-11-2012 04:32 AM
OK,
1275507 x 15ms = 19132 seconds
= 5.3 hours
So, for 5.3 hours, the 'backup ' (well, bptm/ media serever part) is doing nothing, while it waits for the cliemt side of the backup to send data.
As this is a master/ media server,we can more or less discount/ ignore a network issue, so this leaves one of two things (probably)
You are unable to read the data off the disk quickly as the server is busy writing / reading the catalog
or
You are unable to read the disk quickly due to a 'config' issue with the disk or filesystem
It would be good to run a test backup off the server, using a separate area when the server is not too busy - how fast does this go.
Martin
01-11-2012 04:35 AM
Hmm, quite bad - if you defrag this, you will need downtime and must stop NetBackup whilst you do it ...
(Yes, you probably could leave it running, but I hate to think of what could happen ...)
Could be worth considering to 'Archive' the catalog to make it smaller - this writes the xxx.f files in the catalog to tape, then deletes them from the server. If you need to restore, the .f file is restored from the tape first (manual process, but easy ...).
Martin
01-11-2012 04:51 AM
i disabled the antivirus now.
how the archiving can be done? i need this becuase i will migrate the master to new hardware.
01-11-2012 05:02 AM
The suggestion of archiving was meant as a 'suggestion' not necessarly a fix.
Depending on the cause - filesystem problem/ disk config issue - archiving the catalog would only be a workaround, as the real issue is still there.
The correct course of action is to fully investigate the cause, and fix if possible. Then, you may consider archiving.
Archiving is covered in the NBU Admin manual :
Here is a link to the 7.1 manuals :
http://www.symantec.com/docs/TECH154178
For exmple, in the manual :
NetBackup Administrator's Guide for UNIX and Linux, Volume I
http://www.symantec.com/docs/DOC3650
Archiving the Catalog is found on page 759.
You should make multiple duplicateions of these tapes, kept in separate places.
Regards,
Martin
01-11-2012 06:38 AM
what do you mean by test backup off the server?
01-11-2012 07:23 AM
Run a backup of the master server, but not the catalog area. I want to see how fast it can go ...
Hence my previous question, does the master server run any other backups, and, are these fast ?
Thanks,
Martin
01-11-2012 09:39 AM
the master is media for itself now, if you wish i can run backup using the master as media also i will do the test using another media server.
01-11-2012 10:26 AM
01-12-2012 12:54 AM
HI All,
i did many tests,
1. I did backup for G:\ drive (D is for Veritas), the backup is fast when using the master as a media and when using anohter server as media.
2. I did backup for the D drive using he master as media and using another media both are slow.
3. I did catalog backup using the master as media and using another media, both are slow.
So I concluded that the backup become slow when starts taking backup for D:\veratis\netbackup\db\....
01-12-2012 01:25 AM
Looks like you need some down time to do the defrag then - as we said previosuly it would not be a great idea to do it whilst NetBackup is active
After all, according to the defrag report it is the images directory that is in the worst state.