cancel
Showing results for 
Search instead for 
Did you mean: 

NetBackup 8.3 not writing to tapes

jpope
Level 3

We have an issue where all off a sudden the Media/Master server Netbackup 8.3.0.2 we have will not write to any tapes.

It fires off the job, it says it has selected a tape to mount, I see the tape move in the library into the drive (Can see that on the web gui also), but that is all that happens. It never writes anything to the tape. The tape sits idle in the tape drive and Netbackup just sits there waiting for the tape to be read. 

When using vmoprcmd it shows the tape drives, but does not show that there is a tape in them.

I have upgraded and downgraded the Firmware on the tape drives and library, and it does nothing. Still the same issue. I have checked the drivers and they are all up to date, and none of those have change anyway.

I have removed Microsoft Security patches to see if that will make a change, but that does not change.

It seems that either Netbackup is not reading what is returned from the library/tape drives or the library/tape drives are not sending the load/mount completion back to Netbackup.

Any help would be greatly appreciated.

Thanks

1 ACCEPTED SOLUTION

Accepted Solutions

So we have figured out the issue now.

When the tapes were inserted they never checked what version of Tapes they were putting in. 

We took one out last night and had a look ..... LTO8 tapes. Drives are LTO7. So correct, not a netbackup issue at all, and in the great words of Homer Simpson - "D'OH!!!!!"

So new tapes should be arriving tomorrow and no more issues we hope.

Thanks all for jumping in to help ou in this.

View solution in original post

29 REPLIES 29

Nicolai
Moderator
Moderator
Partner    VIP   

Hi @jpope 

If you have changed the device configuration lately, you may have switched the physicals location around compared to the logical configuration in Netbackup. E.g Tape Drive 1 is a index 2 and Tape Drive 2 is at index 1 but but the Netbackup configuration expect Tape Drive 1 to be a index 1 and Tape Drive 2 at index 2.

If the above is the case I expect you also see downed drives ?

What to look for : SCSI sense key codes, usually to be found in the event viewer.

/Nicolai

I have gone through all of that. I have checked every LUN ID and made sure they match up correctly to the correct serial number of tape drive, including the drive number association. Everything matches up.

We never changed anything with regards to the Tape configuration and library. One day it was working the next it stopped. All we did is add tapes to the library.

prasannad
Level 3
Partner

Hi Jpope,

From the NetBackup end, do you see drives as up. If drives also getting down frequently, check the healthiness of your SAN communication over SAN switches.

Prasanna

StefanosM
Level 6
Partner    VIP    Accredited Certified

also check the cleaning remaining of the cleaning tapes (at netbackup or library, depending who is responsible for the cleaning)

Nicolai
Moderator
Moderator
Partner    VIP   

@jpope 

What sense key codes do you see in the event viewer ?

https://en.wikipedia.org/wiki/Key_Code_Qualifier

The drives show as up all the time. They do not go down at all. Netbackup finds all the devices when doing a "Configure Storage Devices". Also these are directly attached to the server vi SAS Controller/Cables no SAN/FC is involved.

Cleaning is done by the library. No cleaning is initiated by Netbackup. And all the drives are operational in the library

Can you please tell me where specifically I need to look in the Event Viewer to find these ?

In the event viewer (Applications) this is what I see in order:

  • LTID - Sent ROBOTIC request, Type=1, Param2=0
  • TLD(0) MountTape 0074L7 on drive 1, from slot 41
  • TLD(0) Creating Process for MOUNT: "C:\Program Files\Veritas\Volmgr\bin\tldd.exe" -v -child -ro 1 -ltidev 1 -ltix 1 -rn 0 -slot 41 -lf 0 -flgs 0 -rs 65536 -pd 180 -sd 15 -path {0,0,4,0} -vsn 0074L7 -b 000074L7
  • ../tldcd.c.3128, process_request(), received command=1 (ROBOT_MOUNT), from peername=server, version 50
  • Processing MOUNT, TLD(0) drive 1, slot 41, barcode 000074L7 , vsn 0074L7
  • TLD(0) Creating Process for MOUNT: "C:\Program Files\Veritas\Volmgr\bin\tldcd.exe" -v -child -ro 1 -rn 0 -dn 1 -socket 1140 -slot 41 -rht 0 -vsn 0074L7 -b 000074L7
  • starting tldcd
  • TLD(0) [2524] opening robotic path {0,0,3,1} (bus -1, target -1, lun -1)
  • inquiry() function processing library IBM 3573-TL F.11:
  • TLD(0) initiating MOVE_MEDIUM from addr 4136 to addr 256
  • TLD(0) closing/unlocking robotic path
  • inquiry() function processing library IBM 3573-TL F.11:

And that's all. Tape has been loaded into the drive and now just sits there idle and the job does not go any further.

This is all the job says:

27-Jan-2022 13:15:32 - begin Label
27-Jan-2022 13:15:32 - started process bplabel (pid=14032)
27-Jan-2022 13:15:32 - requesting resource  NetBackup:0074L7
27-Jan-2022 13:15:32 - granted resource  0074L7
27-Jan-2022 13:15:32 - granted resource  IBM.ULT3580-HH7.001
27-Jan-2022 13:15:33 - Info bptm (pid=13080) start
27-Jan-2022 13:15:33 - Info bptm (pid=13080) Waiting for mount of media id 0074L7 (copy 1) on server scobkp.uk.bats.com.
27-Jan-2022 13:15:33 - mounting 0074L7
27-Jan-2022 13:15:33 - Info bptm (pid=13080) INF - Waiting for mount of media id 0074L7 on server scobkp.uk.bats.com for reading.

DPeaco
Moderator
Moderator
   VIP   

The tape drives and library ride on top of the O/S drivers. In my past experience, if the O/S (server) works well with the tape library and drives, then NetBackup will work well with it as well. Seems like you've done a lot already to troubleshoot and resolve this issue....all without success at this point.

The issue shouldn't be wrapped around tape drive firmware, after all, the O/S is responsible for the connection to the devices. Have you set vm.conf to VERBOSE and have the debug directories created with increased logging level to 5 to try and see what is happening behind the scenes? For this, I would think you would need as high of detailed logging as you can get or...open a support case to get some assistance that way.

Thanks,
Dennis

Hi @jpope 

On top of the logging suggestion by @DPeaco , another thing to try would be to remove the drives and robot from NetBackup (under Media and Device Management -> Device, then have NetBackup reconfigure the library (Configure Storage Devices wizard). 

It may not help, but certainly woorth a try.

Cheers
David

The logging level is set to 5 across the board at the moment and I currently do have a case open with Veritas, but they seemed stumped by this also, so was just reaching out to the wider community to see if anyone may have had this same issue, or even a bright idea that no-one thought of to try ... 

With regards to the logs, would you know what logs I should look into. Can try tail them while the job is running to extract the relevant information, as there is so much that gets dumped in these logs at level 5.

Thanks

@davidmoline No luck. I had originally just removed the drives and reconfigured, but I went all the way with your suggestion and removed the library and drives and reconfigured them again (and updating the media in the library as well) and it does the same thing.... mounts the tape and sits there waiting 

StefanosM
Level 6
Partner    VIP    Accredited Certified

Do you use the IBM encryption key Manager to encrypt the tapes?
If yes, Had you check if it is working?
I had a similar problem with another software and the problem was the IBM key management software.

else you can download the ITDT software from IBM support and run diagnostic to one drive.
You have to use an EMPTY tape.

mph999
Level 6
Employee Accredited

I suspect this will require the library vendor to see what the drives are doing when NBU tries to use them, as well as a a closer look at AVRD (probably via procmon)

@StefanosM  We do not run encryption on the library itself. Encryption is done via the Netbackup Media SErver

StefanosM
Level 6
Partner    VIP    Accredited Certified

you mean that netbackup is managing the keys?
or you are using Media Server Encryption Option (MSEO) ?
MSEO is a different product with its own installation

I think that you must run ITDT diagnostics. with an EMPTY tape
https://www.ibm.com/support/fixcentral/swg/selectFixes?parent=Tape%20drivers%20and%20software&produc...

@StefanosM 

Netbackup is managing the Keys.

I have run ITDT and it loads the tape but then fails and ejects the tape with "NO TEST TAPE INSERTED":

"MOUNT TAPE": {
"START TIME": "Mon Jan 31 14:16:50.029 2022" ,
"DURATION": 48958 ,
"TEST RESULT": "ABORTED" ,
"RESULT CODE": "NO TEST TAPE INSERTED" ,
"_SENSEDATA": "" ,
"TARGET STATUS": "Check Condition" ,
"HOST STATUS": "Status is not supported" ,
"EC": 1117

All tapes in the library are blank. They had just been newly loaded.

Nicolai
Moderator
Moderator
Partner    VIP   

Hi @jpope 

From LTO.org : LTO drive generations 1-7 are able to read tapes from two generations prior and are able to write to tapes from the prior generation.

Could it be that a series of non compatible LTO7 tapes has sneaked in the scratch pool ?. 

Would it be possible to open the robot, and manual mount a tape in the tape drive, to see if the tape is spat out of the tape drive or stays mounted ?