bptm process - causing performance bottleneck

Question

Noticed performance drop when single bptm process spawn multiple duplication jobs.
Problem Description:
ServerA is backed up using "_Test_XYZ1Day" SLP. As per the SLP config, the backup goes to disk staging pool (Advance disk pool XYZStagingpool1) and then duplicated (3copies) to tape. Throughput of the duplication job to tape drive drops directly in proportion to the number of duplication per SLP.&nbsp;&nbsp;(NOTE: each duplication job uses a seperate Tape drive)
Test Results

SLP _Test_XYZ1Day: Set with 1x Duplication job. Job uses 1 Tape drive and runs at throughput of 120 – 140MB/s. 120MB/sec sustained throughput most of the time.
	SLP _Test_XYZ1Day: Set with 2x Duplication job. Jobs will use 2 Tape drives and each of the jobs run at throughput of 80 – 85MB/s. 83MB/sec sustained throughput per drive most of the time.&nbsp;
	SLP _Test_XYZ1Day: Set with 3x Duplication job. Jobs will use 3 Tape drives and each of the jobs run at throughput of 53 – 58MB/s. 54MB/sec sustained throughput per drive most of the time.

NOTE: Combined throughtput of all the duplication jobs seems limited to 160 - 170MB/sec.
Test Conditions: All the above tests are stats are under below conditions

Dedicated Staging Array used as a Advance Disk Pool.( 270+ MB/sec Sustained read throughtput )
	No other writes/read on the Array.
	To the same Tape library. (Stk SL500 with 5 LTO4 Drives)
	Single port dual HBA's connected to two different fabrics at 2gbit speeds. (3 tape drives on 1 fabric and 2 and controller on other)

&nbsp;
Storage lifecycle policy details:
#nbstl _Test_XYZ1Day -L
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Name: _Test_XYZ1Day
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Data Classification: (none specified)
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Duplication job priority: 0
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; State: active
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Version: 5
	Destination&nbsp; 1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Use for: backup
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Storage Unit: XYZStagingPool1
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Volume Pool: (none specified)
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Server Group: (none specified)
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Retention Type: Capacity Managed
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Retention Level: 10 (1 day)
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Alternate Read Server: (none specified)
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Preserve Multiplexing: false
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; State: active
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Source: (client)
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Destination ID: (none specified)
	Destination&nbsp; 2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Use for: duplication
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Storage Unit: media3-hcart-tld-2
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Volume Pool: temp_cp_test1
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Server Group: Any
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Retention Type: Fixed
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Retention Level: 10 (1 day)
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Alternate Read Server: (none specified)
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Preserve Multiplexing: false
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; State: active
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Source: (primary)
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Destination ID: (none specified)
	Destination&nbsp; 3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Use for: duplication
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Storage Unit: media3-hcart-tld-2
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Volume Pool: temp_cp_test1
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Server Group: Any
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Retention Type: Fixed
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Retention Level: 10 (1 day)
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Alternate Read Server: (none specified)
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Preserve Multiplexing: false
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; State: active
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Source: (primary)
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Destination ID: (none specified)
	Destination&nbsp; 4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Use for: duplication
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Storage Unit: media3-hcart-tld-2
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Volume Pool: temp_cp_test2
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Server Group: Any
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Retention Type: Fixed
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Retention Level: 10 (1 day)
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Alternate Read Server: (none specified)
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Preserve Multiplexing: false
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; State: active
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Source: (primary)
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Destination ID: (none specified)
&nbsp;
Any ideas would be appreciated.
Thanks.

mark_solutions · Accepted Answer

Fully with you and your setup and tuning tests now.
The issue here is that you use inline copy and its handling of the bptm process which is a know problem.
This tech note explains it better http://www.symantec.com/docs/HOWTO56160
It maybe that you can think about this to see if there is a bottleneck anywhere that could help imporve things but the issue stems from using inline copy
Hope this helps

aalmroth · Answer

Well, the figures you mention seems to correlate well with 2Gbit FC, approx 170-180MB/s.
I think your problem may be that you use all three drives on one HBA port. A simple test here would be to "down" 1-2 drives on the first port, and see if the same behaviour occur when the jobs are forced to run to diffferent ports. If it is a HBA port bandwidth problem, then you should in this case have two duplication jobs, but each running to drives on separate ports, and you should get the full speed. If not, then perhaps the I/O bus on the media server is the limiting factor.
Also, you don't mention how the disk system is connected? Is it connected using the same HBA ports, or dedicated? Even if the I/O direction is different, that is, read from disk, write to tape, there will be overhead handling all I/O on the HBA, the device driver, and OS kernel. In this case it may help to use a higher I/O block size. Exact configuration varies depending on OS.
/A

mark_solutions · Answer

Hi
If this is not all dropped on a single HBA (and hopefully it isn't) then when you do more than one duplication at a time (inline copies) then you can find that the bptm process handles the data through its buffers in such a way that this has been seen to occurr.
Tuning your data buffers can help - just try different numbers.
As you are using LTO4 drives the size for best performance in SIZE_DATA_BUFFERS is 262144
If you are not already using this value you will need to bplabel any previously (BUT EXPIRED!) tapes, otherwise NBU will jsut read the header and change the buffer size to match the old setting.
So then try 16, 32 and 64 in the NUMBER_DATA_BUFFERS file to see what gives you the best performance.
All buffer files in 
etbackup\db\config\
32 is usually good but this will vary when using inline copy
Of course you can also tune your DISK buffers which may improve things further
Hope this helps

omar_villa · Answer

Just seeing the behavior u every drive under the same hba, confirm the zoning or binding of your drives and u will know where u are

errrlog · Answer

Thanks guys for the comments.
I want to clarify few more things:
-2Gbit port: &nbsp;I am able to get 193MB/sec sustained throughput on a 2 gbit port.
- Most of the tests are done with 2xduplication jobs. I started testing 3xduplication to correlate the 160MB/sec - 170MB/sec bptm limitation.
- During the 2xduplication job tests, I made sure that I use the tape drives on a different HBA, which are connected to differnt fabric switchs.
ALL TESTED SCENARIOS:

1x duplication job goint to 1xTP Drive on Hba1 on Hba2 - Throughput 120MB/sec - 140MB/sec
	2x duplication jobs going to 2xTP drives on different hba port - Throughput 80 - 85MB/sec on each drive
	2x duplication jobs going to 2xTP drives on the same hba port&nbsp; - Throughput 80 - 85MB/sec on each drive
	3x duplication jobs going to 2xTp drives on hba1 and 1xTP drive on hba2 - Throughput 53 - 58MB/sec on each drive
	3x duplication jobs going to 3xTp drives on hba1&nbsp; - Throughput 53 - 58MB/sec on each drive

- Disk is a Storage LUN from Storagetek SAN array dedicated for backup Staging. The array is connected using another set of two single port hba's. (bpbkar to null test provided 270MB/sec sustained throughput)
- All HBA's are PCI-E (64 bit)&nbsp;and so no way close to saturating bus limitation.
- Disk Buffers: - Tested with 128k, 256k, 512k, 1024k. 1024k seems to be slightly better but 512K seems to balance the delayed waits.
- Number data buffers: - 256. (the server is loaded with 32GB RAM) and no more than 200 concurrent jobs running at the same time.
- Tuning: The point to be note is we are getting 120MB/sec sustained to a LTO4 drive. Which is a optimal value for LTO4 and anything over 120MB/sec is just due to compression.
SCENARIO (1): - 1x duplication job goint to 1xTP Drive on Hba1 on Hba2 -&nbsp; (processess involved bptm controlling 1xtape drive , bpdm reading from disk)
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; I get Throughput 120MB/sec - 140MB/sec
SCENARIO (2): considering (1) when 2xduplication jobs are forked from single bptm writing to -&gt; Different Tape Drives -&gt; on Different Hba ports -&gt; connected to different fabric switches. (processess involved bptm controlling 2xtape drive , bpdm reading from disk) 
I get Throughput 80MB/sec - 85MB/sec. But, Ideally I should get the same throughput as in Senario (1). i,e 120MB - 140MB.
The only change between the SCENARIO (1) and SCENARIO (2) is NUMBER OF DUPLICATION JOBS.
This clearly seems like some kind of bptm process contention and not resource contention

mark_solutions · Answer

Hi
Your disk buffers look quite small and your number of buffers quite high
An example of what i have found works really well in multiple drive LTO4 or 5 environments is as follows:
SIZE_DATA_BUFFERS 2162144
NUMBER_DATA_BUFFERS 32
SIZE_DATA_BUFFERS_DISK 1048576
NUMBER_DATA_BUFFERS_DISK 32
Do bear in mind that you need to test this using new tapes or any available expired tapes will need relabelling to overwrite the header with the new block size
Hope this helps

Forum Discussion

bptm process - causing performance bottleneck

12 Replies

Related Content

LTO9 tape performance poor - bottlenecks?

High CPU load on primary server - java process

Re: VM Backups over SAN is slow......or about right??

Unable to perform file restore on bitlocker encrypted VM's

Re: How to identify the bottleneck for slow backups?

Recent Discussions

command: bperror

MS-SharePoint policy restore error (2804) .

How to restore a backup

How to configure RBAC

10 years old netbackup appliance database service down, ssl certification out date