Read error on media id A00032, drive index 5, reading header block, cannot allocate memory

[higsc]

We are using Netbackup 7.1 running on SLES11 SP1. Master and Media server hardware is on a Cisco UCS Blade B200. We are using quantum i6000 LTO5 tape library.Jobs are failing with EC-84. If anybody have faced a similar issue in netbackup, please help us.

Here is our bptm log :

18:42:25.560 [14045] <2> io_write_media_header: drive index 2, writing media header 18:42:25.560 [14045] <2> io_ioctl: command (6)MTREW 1 0x0 from (bptm.c.9521) on drive index 2 18:42:25.700 [14045] <2> io_ioctl: command (5)MTWEOF 1 0x0 from (bptm.c.9585) on drive index 2

18:42:32.217 [14045] <4> write_backup: begin writing backup id uschabck1003-b_1308178829, copy 2, fragment 1, to media id A00289 on drive HP.ULTRIUM5-SCSI.003 (index 2)

18:42:32.219 [14045] <2> write_data: absolute block position prior to writing backup header(s) is 2, copy 2

18:42:32.219 [14045] <2> io_write_back_header: drive index 2, uschabck1003-b_1308178829, file num = 1, mpx_headers = 0, copy 2

18:42:32.221 [14045] <2> write_data: completed writing backup header, start writing data when first buffer is available, copy 2

18:42:32.221 [14045] <2> write_data: first write, twin_index: 0 cindex: 0 dont_process: 1 wrote_backup_hdr: 1 finished_buff: 0

18:42:32.221 [14045] <2> write_data: received first buffer (262144 bytes), begin writing data

18:42:41.204 [14045] <2> write_data: writing block shorter than BUFF_SIZE, 98304 bytes

18:42:41.205 [14045] <2> write_data: writing short block, 98304 bytes, remainder 0

18:42:41.205 [14045] <2> write_data: Total Kbytes transferred 2151008

18:42:41.206 [14045] <2> io_terminate_tape: writing empty backup header, drive index 2, copy 2

18:42:41.206 [14045] <2> io_ioctl: command (5)MTWEOF 1 0x1 from (bptm.c.8778) on drive index 2

18:42:42.855 [14045] <2> io_write_back_header: drive index 2, empty_file, file num = 2, mpx_headers = 0, copy 2

18:43:11.787 [14045] <2> io_terminate_tape: absolute block position prior to writing empty header is 8405, copy 2

18:43:11.787 [14045] <2> io_terminate_tape: block position check: actual 8405, expected 8407

18:43:11.787 [14045] <16> io_terminate_tape: FREEZING media id A00289, External event caused rewind during write, all data on media is lost

Thanks

Suresh

[Nicolai]

This is the issue:

io_terminate_tape: block position check: actual 8405, expected 8407

Something is changing the absolute block number. Netbackup keep track of the block written. If the starting  block + the written block does not match actual block number, Netbackup will of security purposes freeze the tape as further writing may lead to corrupt backup images.

Things to check:

Cannot allocate memory : This often caused by a very high number in NUMBER_DATA_BUFFERS (placed in /usr/openv/netbackup/db/config). Reduce the number to 256 and re-try the operation. Refer to The Netbackup performance & tuning guide. http://www.symantec.com/docs/TECH62317

External Commands: If the drives are connected by SAN, check zoning. Do not put multiple drives in the same zone. Zone one HBA to one tape drive. The HBA can be a member of many zones. This config is much more resilient to external events. 

[higsc]

28 zones are created as shown in the attach. One zone per tape drive and host is created.

We also tried downgrading the firmware on tapedrives as per quantums' suggestion. Quantum is telling that it could be an issue with an unsupported hba or tape driver on use linux 

What we have is...netbackup master and media servers running on Cisco Ucs B200 hardware using CNA.

CNA tranfers all FC traffic and ethernet trafficthru one single interface.

Please suggest.

[Nicolai]

Does you till see issues ?

What does /var/log/messages say ?