shared memory configuration for netbackup 7.5.0.4

[Sam_sd93]

Hi,

Can anyone help me out in configuring the shared memory on solaris 10. Currently, netbackups are happening with 7mb/sec, so i've created SIZE & NUMBER buffer files to speed up the backups.However when the increase the value in SIZE file backups fail with "problems encountered during setup of shared memory (89). Currently no shared memory is configured on the machine.

2) Will there be any performance degradation with these BUFFER values when restoring

Environment.

Netbackup master server : solaris 10

Backup Ver: 7.5.0.4

RAM : 32GB

SWAP: 25GB

NETWORK: 3 NICS with each speed 1GB.So created an aggregate which comes to 3GB speed.

SIZE_DATA_BUFFER: 3670016

NUMBER_DATA_BUFFER: 512

TAPE LIBRARY : DRIVES(2); MPX=1; TAPES =LT05

Nummer of tape drives  * MPX * SIZE_DATA_BUFFERS * NUMBER_DATA_BUFFERS = TOTAL AMOUNT OF SHARED MEMORY

2*1*3670016*512 = 3758096384 ( almost 4GB)

I followed the steps to create 8GB shared memory as per KB TECH62633. As i am using solaris 10 i am bit confused with steps 3 & 4 in KB TECH138168

#/usr/sbin/projadd -U root -c "NetBackup resource project" -p 1000 NetBackup
 # /usr/sbin/projmod -a -K 'project.max-msg-ids=(privileged,256,deny)' NetBackup
 # /usr/sbin/projmod -a -K 'project.max-sem-ids=(privileged,1024,deny)' NetBackup
 # /usr/sbin/projmod -a -K 'project.max-shm-ids=(privileged,1024,deny)' NetBackup
 # /usr/sbin/projmod -a -K 'project.max-shm-memory=(privileged,8589934592,deny)' NetBackup

[mph999]

Set NUMBER to 128 SIZE to 262144 test Then set NUMBER to 256 test, if faster, try NUMBER at 512 If it decreases in performance, go back to 256 ... Typically, NUMBER is set to 64 / 128 / 256 / 512 etc ... 256 is probably the most common. SIZE should be (64 x 1024) or (128 x 1024) or (256 x 1024) Typically, SIZE at 262144 gives good performance, but it is a case or try and see. Martin

[Sam_sd93]

Martin,

I've verified with different values and currently backups are running fine.However I want to increase NUMBER value more than 512 as we've sufficient resources on the server but its complaining to configure the shared memory. so I am not sure what are all the parameters that needs to be changed on solaris 10 machine to configure the shared memory.

Regards,

Sam

[revarooo]

If backups are running fine just leave it be.

[Sam_sd93]

Revaroo,

I mean backups are running fine but they are slow not throughput as expected.

Regards,

[mph999]

You can't just increase the NUMBER - it's a tuning value, you have to experiment. It will get to the point at which the drives can't write any quicker, no point in increasing from then on. I don't think I've ever seen more than 512 buffers (not saying people don't do this, just that I don't recall seeing it ... I'd expect a combination or 256 or 512 buffers along with 262144 size should give you 100 or more MB/s - if not I'd suggest something else is wrong. Look in act mon on a completed job to see if there are more delays either waiting for full or empty buffers. Tape drive speed is not just about the tuning settings, drivers, firmware (HBA / Switch / Drive), SAN, potential drive faults all have an effect (along with disk read speed and network speed). M

[Sam_sd93]

Martin,

From the completed job I see "waited for full buffer 954 times, delayed 12037 times " when the NUMBER VALUE is set to 512,however I still see the speed as 14MB/s and its taking more than 8hrs to complete 300GB of data to backup. As I said earlier I've created an aggregate with 3 NICs of 1GB speed each and HBA's are qlogic with 8GB speed. Much appreciated if you can help me out.

Regards,

Sam

[mph999]

bptm stopped receiving data and as delayed 12037 times, each delay be default is 15 milliseconds so: (15/ 1000) x 12037 = 180 seconds Providing that the backup that this line came from took say 20 mins or more, that delay is fairly insignificant. If the backup was one that should have only taken a miunte or so, then the delay is very significant (as scaled up for a longer backup we would be looking at massive delays). I don't know which it is (short or long backup). If the backup was 'short' then I would suggest first running a test backup of the media server to itself to see how fast the drives can go, that is eliminate the network as much as possible. There should have been another line like that one, but saying: bpbkar waited xx times delayd yy times - what were these values . If there were many 1000's of delays for bpbkar, this would show the drives cannot write quiickly, which is usually a driver/ hba/ firmware / drive issue and buffer tuning will make no difference. So take this back to basics, and see what the drives can do backing up the media server itsef. Providing you can get a decent read speed off the disks, the tape drives should give decent performance.

[Marianne]

waited for full buffer.....

It seems to me that your buffer size is too big, causing long delays in filling up the buffers.

I agree with Martin - a buffer size of 262144 has been found to give best performance.

What is network buffer set to?

 

[Sam_sd93]

Marianne,

Currently, there is no Network buffer set. If so, what should be the value and is it at both server & client side?

Regards,

Sam

[mph999]

Forget the network buffer for the moment ...

Set the buffers like I explained earlier, run a backup of the media server and lets see what the performance is.

If we start changing all the different buffers at once we're going to get nowhere ...

M

[Sam_sd93]

Martin,

It took like 7 hrs 15mins to backup 205GB of data after changing the NUMBER buffer to 512 & SIZE to 3145728 and as per Status log "full buffer 66748times,delayed 1701111 times". Will check with SIZE value set to 262144

Regards,

SAM

[Sam_sd93]

Moreover the speed is like 7MB/sec

[Ron_Cohn]

One item has been missed from the discussion:  Number of files on the server.  You can have the fastest SAN on the fastest network, but it really boils down to number of files.  Even though my environment is Windows, I had a 25GB drive that took 13 hours to backup.  Looking at the drive, I had 3+ million files on it.

Once I changed from a "standard" backup to a "flash", run time dropped to 30 minutes.  Just a thought to consider...

[jim_dalton]

2 things

1 you have set up the project settings for netbackup but have you made sure that netbackup start script is made to be part of the project?Theres a netbackup document for it.

2 what ron says is correct. Do yourself a favour: rule out the data for now, use a synthetic policy to write out random data from server to itself if you can ie no network and a small number of multi Gbyte files. If thats slow then the bottleneck is there already without involving networks, data profiles etc. A local backup reading synthetic data ie no disk will be as fast as you can get data. With lto5 and large blocks if you dont get the manufacturers speed then we will learn something. I would say 150M/sec, maybe more.

Jim