SAN media servers backups are runnig slow

[Rajesh_s1]

Hi,

We are experiencing a huge throughput issue post upgrade of NetBackup 7.5.0.4 from 7.1. Please let me know if any changes or recommendation we need to perform.

Regards,

Rajesh

[Marianne]

There are no known performance issues with 7.5.x.

Please post bptm logs from before and after upgrade as File attachments.

[Rajesh_s1]

Attaching the logs of before 7.5 upgrade

[Rajesh_s1]

Attaching the logs of after 7.5 upgrade

[Rajesh_s1]

Hello Marrine, can you please help me if any tuning guide is available or any tuning we need to do, i wont think so its released for 7.5. We are using windows 2008-R2 OS and all the SAN media servers are upgraded to 7.5.0.4. Regards, Rajes

[revarooo]

Logs are for bpbrm, would need the bptm log from the media server and bpbkar from the client (if this is a SAN media server logs would be from the SAN Media server for both media and client as they are the same machine).

Logs for just bpbrm do not help with troubleshooting performance issues.

[Rajesh_s1]

Hi Revaroo, I am talking about SAN media server performance issue and i have uploaded BPTM logs . Ours is an SSO environment and the SAN media servers backups are going to tape through its own.

Previous week we have done upgrade from Netbackup 7.1 to 7.5.0.4 and on the same time we have configured Symantec End point protection (AV) earlier we were using CA , post to this changes we are experiencing this performance issue.

Please let me know if anything required. Rgds, rajesh

[revarooo]

Yes SAN Media server, a server that is only allowed to backup itself. 

> Please let me know if anything required.

I already mentioned, bpbkar and bptm logs.

[Marianne]

I can see big differences in transfer rates in the 2 log files, but we are looking at 2 different policies as well - do they have diferent backup selections as well?

Oh, and a matter of interest - seems SAN media server was upgraded from 6.5.1.

This is what I see in the logs :

Before:

2:35:00.048 [12048.16964] <2> bptm: INITIATING (VERBOSE = 0): -w -pid 13908 -c MSDB5 -den 6 -rt 8 -rn 1 -stunit MSDB5-hcart-robot-tld-1 -cl MSDB5 -bt 1355000648 -b MSDB5_1355000648 -st 0 -cj 2 -p Weekly -reqid -1354219791 -jm -brm -hostname MSDB5 -ru root -rclnt MSDB5 -rclnthostname MSDB5 -rl 2 -rp 1814400 -sl Sec_Weekly_Full -ct 13 -maxfrag 1048576 -tir -eari 0 -mediasvr MSDB5 -no_callback -connect_options 0x01010100 -jobid 364485 -jobgrpid 364477 -masterversion 750000 -bpbrm_shm_id 

02:35:03.915 [12048.16964] <2> io_init: using 65536 data buffer size

02:35:03.915 [12048.16964] <2> io_init: CINDEX 0, sched Kbytes for monitoring = 60000

02:35:03.915 [12048.16964] <2> io_set_recvbuf: setting receive network buffer to 263168 bytes

02:35:03.916 [12048.16964] <2> io_init: using 30 data buffers

02:36:03.041 [12048.16964] <2> manage_drive_attributes: Host Attributes: Vendor [SYMANTEC], Name [NetBackup BPTM], Version [6.5.1]

02:50:04.989 [12048.16964] <4> write_backup: successfully wrote backup id MSDB5_1355000648, copy 1, fragment 1, 14469440 Kbytes at 20537.662 Kbytes/sec

03:06:46.718 [12048.16964] <4> write_backup: successfully wrote backup id MSDB5_1355000648, copy 1, fragment 2, 47914112 Kbytes at 47893.326 Kbytes/sec

03:23:48.491 [12048.16964] <4> write_backup: successfully wrote backup id MSDB5_1355000648, copy 1, fragment 3, 66235392 Kbytes at 64892.189 Kbytes/sec
....
....

04:45:35.085 [12048.16964] <2> write_data: waited for full buffer 165339 times, delayed 327544 times

04:45:35.085 [12048.16964] <2> write_data: Total Kbytes transferred 336418887

After:  (backup starting during working hours??)

08:44:55.229 [7344.7956] <2> bptm: INITIATING (VERBOSE = 0): -w -pid 7504 -c MSDB5 -den 6 -rt 8 -rn 1 -stunit MSDB5-hcart-robot-tld-1 -cl MSDB5 -bt 1356664459 -b MSDB5_1356664459 -st 0 -cj 2 -p Daily -reqid -1354311913 -jm -brm -hostname MSDB5 -ru root -rclnt MSDB5 -rclnthostname MSDB5 -rl 1 -rp 1209600 -sl Full -ct 13 -maxfrag 1048576 -tir -eari 0 -mediasvr MSDB5 ...

08:45:00.761 [7344.7956] <2> manage_drive_attributes: Host Attributes: Vendor [SYMANTEC], Name [NetBackup BPTM], Version [7.5.0.4]

08:44:58.923 [7344.7956] <2> io_init: using 65536 data buffer size

08:44:58.923 [7344.7956] <2> io_init: CINDEX 0, sched Kbytes for monitoring = 600000

08:44:58.923 [7344.7956] <2> io_set_recvbuf: setting receive network buffer to 263168 bytes

08:44:58.923 [7344.7956] <2> io_init: using 30 data buffers

08:59:58.751 [7344.7956] <4> write_backup: successfully wrote backup id MSDB5_1356664459, copy 1, fragment 1, 8596864 Kbytes at 9631.693 Kbytes/sec

09:14:58.636 [7344.7956] <4> write_backup: successfully wrote backup id MSDB5_1356664459, copy 1, fragment 2, 8271424 Kbytes at 9222.915 Kbytes/sec

09:29:58.403 [7344.7956] <4> write_backup: successfully wrote backup id MSDB5_1356664459, copy 1, fragment 3, 10112640 Kbytes at 11277.102 Kbytes/sec

....

23:48:03.258 [7344.7956] <4> write_backup: successfully wrote backup id MSDB5_1356664459, copy 1, fragment 46, 7390208 Kbytes at 10594.156 Kbytes/sec

We don't see the end result as backup ran into next day with rest of info in next day's bptm log.

II would blame SEP for slow throughput rather than NBU upgrade. You may want to add NBU executables to SEP exclude lists. See these previous discussions: 

https://www-secure.symantec.com/connect/forums/symantec-endpoint-protection-netbackup-7103 

https://www-secure.symantec.com/connect/forums/netbackup-7-and-anti-virus