01-25-2022 08:01 AM
We have an issue where all off a sudden the Media/Master server Netbackup 8.3.0.2 we have will not write to any tapes.
It fires off the job, it says it has selected a tape to mount, I see the tape move in the library into the drive (Can see that on the web gui also), but that is all that happens. It never writes anything to the tape. The tape sits idle in the tape drive and Netbackup just sits there waiting for the tape to be read.
When using vmoprcmd it shows the tape drives, but does not show that there is a tape in them.
I have upgraded and downgraded the Firmware on the tape drives and library, and it does nothing. Still the same issue. I have checked the drivers and they are all up to date, and none of those have change anyway.
I have removed Microsoft Security patches to see if that will make a change, but that does not change.
It seems that either Netbackup is not reading what is returned from the library/tape drives or the library/tape drives are not sending the load/mount completion back to Netbackup.
Any help would be greatly appreciated.
Thanks
Solved! Go to Solution.
02-01-2022 06:12 AM
So we have figured out the issue now.
When the tapes were inserted they never checked what version of Tapes they were putting in.
We took one out last night and had a look ..... LTO8 tapes. Drives are LTO7. So correct, not a netbackup issue at all, and in the great words of Homer Simpson - "D'OH!!!!!"
So new tapes should be arriving tomorrow and no more issues we hope.
Thanks all for jumping in to help ou in this.
01-26-2022 12:15 AM
Hi @jpope
If you have changed the device configuration lately, you may have switched the physicals location around compared to the logical configuration in Netbackup. E.g Tape Drive 1 is a index 2 and Tape Drive 2 is at index 1 but but the Netbackup configuration expect Tape Drive 1 to be a index 1 and Tape Drive 2 at index 2.
If the above is the case I expect you also see downed drives ?
What to look for : SCSI sense key codes, usually to be found in the event viewer.
/Nicolai
01-26-2022 01:30 AM
I have gone through all of that. I have checked every LUN ID and made sure they match up correctly to the correct serial number of tape drive, including the drive number association. Everything matches up.
We never changed anything with regards to the Tape configuration and library. One day it was working the next it stopped. All we did is add tapes to the library.
01-26-2022 01:49 AM
Hi Jpope,
From the NetBackup end, do you see drives as up. If drives also getting down frequently, check the healthiness of your SAN communication over SAN switches.
Prasanna
01-26-2022 01:56 AM
also check the cleaning remaining of the cleaning tapes (at netbackup or library, depending who is responsible for the cleaning)
01-27-2022 04:32 AM
What sense key codes do you see in the event viewer ?
01-27-2022 05:11 AM
The drives show as up all the time. They do not go down at all. Netbackup finds all the devices when doing a "Configure Storage Devices". Also these are directly attached to the server vi SAS Controller/Cables no SAN/FC is involved.
01-27-2022 05:12 AM
Cleaning is done by the library. No cleaning is initiated by Netbackup. And all the drives are operational in the library
01-27-2022 05:13 AM
Can you please tell me where specifically I need to look in the Event Viewer to find these ?
01-27-2022 05:21 AM
In the event viewer (Applications) this is what I see in order:
And that's all. Tape has been loaded into the drive and now just sits there idle and the job does not go any further.
This is all the job says:
27-Jan-2022 13:15:32 - begin Label
27-Jan-2022 13:15:32 - started process bplabel (pid=14032)
27-Jan-2022 13:15:32 - requesting resource NetBackup:0074L7
27-Jan-2022 13:15:32 - granted resource 0074L7
27-Jan-2022 13:15:32 - granted resource IBM.ULT3580-HH7.001
27-Jan-2022 13:15:33 - Info bptm (pid=13080) start
27-Jan-2022 13:15:33 - Info bptm (pid=13080) Waiting for mount of media id 0074L7 (copy 1) on server scobkp.uk.bats.com.
27-Jan-2022 13:15:33 - mounting 0074L7
27-Jan-2022 13:15:33 - Info bptm (pid=13080) INF - Waiting for mount of media id 0074L7 on server scobkp.uk.bats.com for reading.
01-27-2022 11:33 AM
The tape drives and library ride on top of the O/S drivers. In my past experience, if the O/S (server) works well with the tape library and drives, then NetBackup will work well with it as well. Seems like you've done a lot already to troubleshoot and resolve this issue....all without success at this point.
The issue shouldn't be wrapped around tape drive firmware, after all, the O/S is responsible for the connection to the devices. Have you set vm.conf to VERBOSE and have the debug directories created with increased logging level to 5 to try and see what is happening behind the scenes? For this, I would think you would need as high of detailed logging as you can get or...open a support case to get some assistance that way.
01-27-2022 01:11 PM
Hi @jpope
On top of the logging suggestion by @DPeaco , another thing to try would be to remove the drives and robot from NetBackup (under Media and Device Management -> Device, then have NetBackup reconfigure the library (Configure Storage Devices wizard).
It may not help, but certainly woorth a try.
Cheers
David
01-28-2022 02:46 AM
The logging level is set to 5 across the board at the moment and I currently do have a case open with Veritas, but they seemed stumped by this also, so was just reaching out to the wider community to see if anyone may have had this same issue, or even a bright idea that no-one thought of to try ...
With regards to the logs, would you know what logs I should look into. Can try tail them while the job is running to extract the relevant information, as there is so much that gets dumped in these logs at level 5.
Thanks
01-28-2022 02:57 AM
@davidmoline No luck. I had originally just removed the drives and reconfigured, but I went all the way with your suggestion and removed the library and drives and reconfigured them again (and updating the media in the library as well) and it does the same thing.... mounts the tape and sits there waiting
01-28-2022 12:04 PM - edited 01-28-2022 12:05 PM
Do you use the IBM encryption key Manager to encrypt the tapes?
If yes, Had you check if it is working?
I had a similar problem with another software and the problem was the IBM key management software.
else you can download the ITDT software from IBM support and run diagnostic to one drive.
You have to use an EMPTY tape.
01-30-2022 04:40 PM - edited 01-31-2022 12:01 AM
I suspect this will require the library vendor to see what the drives are doing when NBU tries to use them, as well as a a closer look at AVRD (probably via procmon)
01-31-2022 02:19 AM
@StefanosM We do not run encryption on the library itself. Encryption is done via the Netbackup Media SErver
01-31-2022 02:35 AM
you mean that netbackup is managing the keys?
or you are using Media Server Encryption Option (MSEO) ?
MSEO is a different product with its own installation
I think that you must run ITDT diagnostics. with an EMPTY tape
https://www.ibm.com/support/fixcentral/swg/selectFixes?parent=Tape%20drivers%20and%20software&produc...
01-31-2022 06:21 AM
Netbackup is managing the Keys.
I have run ITDT and it loads the tape but then fails and ejects the tape with "NO TEST TAPE INSERTED":
"MOUNT TAPE": {
"START TIME": "Mon Jan 31 14:16:50.029 2022" ,
"DURATION": 48958 ,
"TEST RESULT": "ABORTED" ,
"RESULT CODE": "NO TEST TAPE INSERTED" ,
"_SENSEDATA": "" ,
"TARGET STATUS": "Check Condition" ,
"HOST STATUS": "Status is not supported" ,
"EC": 1117
All tapes in the library are blank. They had just been newly loaded.
01-31-2022 07:08 AM
Hi @jpope
From LTO.org : LTO drive generations 1-7 are able to read tapes from two generations prior and are able to write to tapes from the prior generation.
Could it be that a series of non compatible LTO7 tapes has sneaked in the scratch pool ?.
Would it be possible to open the robot, and manual mount a tape in the tape drive, to see if the tape is spat out of the tape drive or stays mounted ?