Event Viewer Shows Errors TLD(0) io_open: using ndmp and Unload requested, but no unload

[koribilli12]

Hello Everyone,

I got some peculiar issue with my Netbackup and this is getting worsen everyday.

We have Netbackup 7.7.3 with Windows 2008 R2 64bit, using Storagetek SL500 Tape library with LTO4 Drives.

This issue is the tpautoconf -report_disc shows missing device once in a every hour and again that will vanish on its own. The event viewer shows multiple errors like TLD(0) [16804] Unload requested, but no unload, drive x (xxxx 6) and TLD(0) io_open: using ndmp, this is happening for all the drives which are zoned in SAN switch. All the backups write and all the drives suddenly shows as down or down-tld.

We have replaced the faulty drives, robotic arm, HBA card, Mutliple time zoning has been created at SAN switch. Network team confirmed that no issue at Network switch. Cables have been swapped but stil issue remains same. Even the NIC cards are also replaced. OS has no issues except with the Netbackup. KMA device hardware has been replaced and stil the issue remains same.
I opened the case with Veritas however they say that the issue is at Hardware level, but still at what level the hardware is faulty we need to know to replace that part. Whenever we reboot tape library or power cycle the backups will run for a day or two and again the issue occurs.

I am clueless at this point what else could be the issue. Your help will be greatly appreciated.

[quebek]

Hey

Can you shed more light? Do you have any NDMP clients? Why this is NDMP based TLD?? Please provide outputs from

tpconfig -d

tpautoconf -r

tpautoconf -t

vmoprcmd

scan

nbemmcmd -listhosts

bpstulist -U

[mph999]

It looks like the device(s) are disappearing and re-appearing on the SAN.

" tpautoconf -report_disc shows missing device once in a every hour and again that will vanish on its own"

NBU tracks drives via serial number.  If a drive is not found, as you know it appears in tpconfig -report_disc.

I would agree that there is no point looking at NBU. it's not really the fault of NBU if devices disappear off the SAN.

You mention NDMP, where are these drives zoned into, media server, NDMP device (eg netapp or similar).

I'll presume they are attached to the media server, can you see anything in system event logs, in theory, you should see them disappear and reappear in here.  Next place to look is on the switch, if something disappears it should log it.

Depending on the HBA brand, it will have a utiliy, either Sansurfer or HBAnywhere (for qlogix or emulex, though I forget which one goes with which),  I suggest getting the utility installed and then when the issue occurrs, look at what devices can be seen in the utility - I like to explain it as the ability to look straight down the FC cable and seeing what is (or isn't at the end).

[koribilli12]

Hello Quebek

We are using NDMP clients and they are zoned in SAN switch so that NDMP backups can write faster than normal.

We have physical rack server where we have HBA installed and zoned from SAN switch. I thought issue is from HBA and we have two PCI slots for it, we have swapped in both the slots and checked but still getting the errors.

Replaced NIC card, added and removed Teaming but still the issue remains. I am not sure where is the mistake and I still see in the event viewer "TLD(0) [5780] Unload requested, but no unload, drive 2 (device 8)" and "TLD(0) io_open: using ndmp".

I have attached all the outputs in zip folder, please have a look and provide your feedback.

Thank you

Kishore

[koribilli12]

cardHello MPH

Please see my responses:

You mention NDMP, where are these drives zoned into, media server, NDMP device (eg netapp or similar).

They are zoned via SAN switch and EMC Isilon nodes, so that backups from Isilon will be faster when compared to normal tape backups.

I'll presume they are attached to the media server, can you see anything in system event logs, in theory, you should see them disappear and reappear in here.  Next place to look is on the switch, if something disappears it should log it.

We are using master as media and system logs does not show anything major apart from io errors in application level.CISCO switch does not show any errors. I have replaced and swapped the HBA card in PCI slots.

My final option would be replacing the FC cable connected from HBA to Tape library.,

Thank you

Kishore