2 tape drives and I do not seem to get proper writing speed

[Unknown77]

Okay thank you, I have 2 tape drives and I do not seem to get proper writing speed (1:1 backup advanceddisk to tapes lto5) 120MB/sec, this is what speeds are, this is linux appliance, can you please advise ?

[24787] <4> write_backup: successfully wrote backup id bitcservdc1 copy 2, fragment 6, 65536000 Kbytes at 74994.536 Kbytes/sec
 [11897] <4> write_backup: successfully wrote backup id bitcservdc2 copy 2, fragment 1, 65536000 Kbytes at 80521.443 Kbytes/sec

[sdo]

hopefully Marianne will split this into a new question.

Anyway speed issues are notoriously difficult to pin down.

1) Did you not mean duplication from Advanced Disk to LTO5 ?

2) What Appliance model and Appliance version ?

3) Please show disk pool configuration from Appliance ?  We need to see shelves and layouts and sizes of Advanced Disk versus MSDP, and where they physically are.

4) Any other concurrent workloads whilst the duplications to tape were taking place ?  i.e. any backups taking place whilst the duplications were running?

5) Your two line log snippet does not have time stamps... so it's difficult to tell whether you had two concurrent duplications running, i.e. one duplication to each of the two tape drives at the same time ?  Or were those two duplications sequential, one after another?  Or at different times?

[sdo]

6) Tape drive vendor ?  (yes, it can make a difference in very subtle situations)

7) I assume fibre tape drives, so please show SAN switch port link speed ?

8) Are all SAN swicth ports appliance-to-switch AND switch-to-tape all at fixed speed, or auto-negotiate ?

9) I assume you have zero'd SAN switch port counters and checked for fibre errors ?

10) Are tape drive vendor and tape media vendor the same ?   i.e. HP tapes in HP drives ?  or IBM tapes in IBM drives ?

11) Any tape encryption taking place ?  If yes, please outline how implemented.

[Unknown77]

1) - Yes

2) 5220

3) Drives are in ibm tape library and it is directly connected to the appliance

4) No other load,

5) 2 tape drives are writing at the same time

[Unknown77]

cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 02 Id: 00 Lun: 00
Vendor: Intel Model: MegaSR Rev: 1.0
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi1 Channel: 00 Id: 24 Lun: 00
Vendor: Promise Model: J630s Rev: 060=
Type: Enclosure ANSI SCSI revision: 06
Host: scsi1 Channel: 02 Id: 00 Lun: 00
Vendor: INTEL Model: RS2PI008 Rev: 2.12
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
Vendor: ESG-SHV. Model: SCA HSBP M9..... Rev: 2.17
Type: Enclosure ANSI SCSI revision: 03
Host: scsi2 Channel: 02 Id: 00 Lun: 00
Vendor: INTEL Model: SROMBSASMP2 Rev: 1.41
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi3 Channel: 00 Id: 00 Lun: 00
Vendor: IBM Model: ULT3580-HH5 Rev: F991
Type: Sequential-Access ANSI SCSI revision: 06
Host: scsi3 Channel: 00 Id: 00 Lun: 01
Vendor: IBM Model: 3573-TL Rev: A.60
Type: Medium Changer ANSI SCSI revision: 05
Host: scsi4 Channel: 00 Id: 00 Lun: 00
Vendor: IBM Model: ULT3580-HH5 Rev: H977
Type: Sequential-Access ANSI SCSI revision: 06
Host: scsi4 Channel: 00 Id: 00 Lun: 01
Vendor: IBM Model: 3573-TL Rev: A.60
Type: Medium Changer ANSI SCSI revision: 05
Host: scsi4 Channel: 00 Id: 01 Lun: 00
Vendor: IBM Model: ULT3580-TD4 Rev: C7QH
Type: Sequential-Access ANSI SCSI revision: 03
5220PL:/usr/openv/netbackup/logs #

lspci:

00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 22)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 22)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 22)
00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 22)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 22)
00:09.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 22)
00:0a.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 10 (rev 22)
00:10.0 PIC: Intel Corporation 5520/5500/X58 Physical and Link Layer Registers Port 0 (rev 22)
00:10.1 PIC: Intel Corporation 5520/5500/X58 Routing and Protocol Layer Registers Port 0 (rev 22)
00:11.0 PIC: Intel Corporation 5520/5500 Physical and Link Layer Registers Port 1 (rev 22)
00:11.1 PIC: Intel Corporation 5520/5500 Routing & Protocol Layer Register Port 1 (rev 22)
00:13.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller (rev 22)
00:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management Registers (rev 22)
00:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 22)
00:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 22)
00:14.3 PIC: Intel Corporation 5520/5500/X58 I/O Hub Throttle Registers (rev 22)
00:15.0 PIC: Intel Corporation 5520/5500/X58 Trusted Execution Technology Registers (rev 22)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1
00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5
00:1c.5 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 6
00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 RAID bus controller: Intel Corporation 82801JIR (ICH10R) SATA RAID Controller
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
01:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
01:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
02:00.0 RAID bus controller: LSI Logic / Symbios Logic LSI MegaSAS 9260 (rev 05)
03:00.0 PCI bridge: Integrated Device Technology, Inc. PES24T3G2 PCI Express Gen2 Switch (rev 02)
04:02.0 PCI bridge: Integrated Device Technology, Inc. PES24T3G2 PCI Express Gen2 Switch (rev 02)
04:04.0 PCI bridge: Integrated Device Technology, Inc. PES24T3G2 PCI Express Gen2 Switch (rev 02)
05:00.0 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)
05:00.1 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)
07:00.0 PCI bridge: Integrated Device Technology, Inc. PES24T3G2 PCI Express Gen2 Switch (rev 02)
08:02.0 PCI bridge: Integrated Device Technology, Inc. PES24T3G2 PCI Express Gen2 Switch (rev 02)
08:04.0 PCI bridge: Integrated Device Technology, Inc. PES24T3G2 PCI Express Gen2 Switch (rev 02)
09:00.0 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
09:00.1 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
09:00.2 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
09:00.3 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
0a:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
0a:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
0f:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 1078 (rev 04)
10:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200e [Pilot] ServerEngines (SEP1) (rev 02)

[Unknown77]

what speeds do other get?

[sdo]

Well, the 5220 went out of support over a year ago.  I assume that those two snippets mean that you got say 155MB/s out of the Advanced Disk pool - i.e. from two concurrent disk reads, which means the Adv.Disk was actually peforming random reads (as the controller and spindles see it).

What version of Appliance firmware is your 5220 running?  You might have the nbperfchk tool available for you to perform some sustained sequential disk IO to see what you can get from the Adv.Disk pool when all else is quiet.  Have you tried any disk read performance tests?

Has tape writing speed only recently become a problem for you?

Cheers for PCI dumps etc... but I was after the output from the CLIsh that shows pool(s) layout(s) across shelves... to see if you perhaps have any part of your Adv.Disk pool spread across some of the physical spindles in the appliance head unit.

[Marianne]

@Unknown77 

Looking at your opening post - in all honesty, most other users are only dreaming of getting 120MB/sec with tape drives. 
With tape drives (or any kind of data transfer), the maximum throughput is only as fast as the slowest component in the data  chain. 
bptm log is good for troubleshooting this - look for 'waited for full (or empty) buffers ....' at the end of each job. 

The throughput in the next 2 lines seems like backups to dedupe storage - not all data is really written. Have a look at all details in the job like this:
scanned: ######## KB, CR sent: #### KB, CR sent over FC: 0 KB, dedup: ##.# %, cache disabled

The transfer rate that you posted (e.g. 74994.536 Kbytes/sec ) is taking the 'scanned' Kbytes into acount, not the actual data that was written (CR Kbytes). 

[Unknown77]

Hi again,

Please advise:

1) appliance is connected through the fibre cable to ibm tape library that has 2 IBM drives HH5

2) had a check and turns out that transport type is LAN and not Fibre

3) How can i change from LAN to Fibre and is it worth it ? OVPASS drivers needed ? This is IBM library

[Marianne]

---

@Unknown77 wrote:

Hi again,

Please advise:

1) appliance is connected through the fibre cable to ibm tape library that has 2 IBM drives HH5

2) had a check and turns out that transport type is LAN and not Fibre

3) How can i change from LAN to Fibre and is it worth it ? OVPASS drivers needed ? This is IBM library

---

---

@Unknown77 wrote:

Hi again,

2) had a check and turns out that transport type is LAN and not Fibre

---

For backup, this is normal. 

Backup path is client -> network -> MSDP

---

3) How can i change from LAN to Fibre and is it worth it ? OVPASS drivers needed ? This is IBM library

---

IBM drivers are only needed on AIX media servers.
The Linux OS on the Appliance have all the necessary drivers for all types of supported LTO tape drives.

When data is duplicated from MSDP to tape, no data transfer is going over the network (unless the tape is connected to a different media server and MSDP and tape are not connected to the same media server). 

Bear in mind that duplication from dedupe disk to tape needs to rehydrate the data.
With that in mind, the write speed of 120MB/sec during duplication is good.

@Unknown77 

It has been a month - have you tried to look at bptm logs as I have suggested in my previous post? 
You can track read speeds (from dedupe disk) and write speeds (to tape) in bptm logs, and see the 'waits' while the read process needs to wait for data to be rehydrated. 

[Unknown77]

Hi again,

so the duplications speed is not 120MB/s but only ~60MB/s,slow writing throughput from advanced disk to tapes (no encryption, 1:1 backup) is still an issue, library and appliance are connected through the fibre cable and set on 4Gbps speed, please advise, report is below:

Avg (Kb/sec)	Library	Media Server	Drive Name	Disp	Monday	Tuesday	Wednesday	Thursday	Friday	Saturday	Sunday
62579	00L4U78C8087_LL0	n5220uk	IBM.ULT3580-HH5.000	D
58720	00L4U78C8087_LL0	n5220uk	IBM.ULT3580-HH5.001	D
60654	ALL DRIVES AVERAGE