need help with nbu 6.5.4

[Silvan]

Hey guys,

Hopefully someone can help me out here.  My NBU experience isn't great, I've been in charge of supporting it for close to a year but never really had any major issues that I wasn't able to solve untill now.

I'm running NBU 6.5.4 on Server 2008 attached to a fairly old Sun StorEdge L8.  Last week my backups stopped working, i'm getting several different errors but haven't found a solution yet.

Things I've done so far:  change scsi cables, changed ports on the scsi card, cleaned the drive, uninstall both the Tape storage drivers and media changer drivers, Ran the diagnostic cycles on the actual tape drive with no errors coming up.

in NBU Console the error I receive are:

mounting WUB442
10/6/2010 12:03:37 PM - mounted; mount time: 00:00:51
10/6/2010 12:03:40 PM - positioning WUB442 to file 1
10/6/2010 12:05:23 PM - Error bptm(pid=5524) ioctl (MTWEOF) failed on media id WUB442, drive index 0, Data error (cyclic redundancy check). (23) (bptm.c.9494)
10/6/2010 12:05:30 PM - end writing
media write error(84)

I first thought it was a bad tape, but i've changed tapes several times to ones that have worked in the past so I doubt all my tapes went bad at the same time.

Windows Event Logs I see these:

The device, \Device\Tape0, has a bad block.

(again this makes it seem like a Tape issue, however in device manager under tape drives the Symolic Name is Tape0 so it's refering to the take backup unit not a tape called Tape0)

ALSO this:

The following boot-start or system-start driver(s) failed to load:

halfinchVRTS

(I searched online and it pointed me to a possible driver issue which is why I uninstalled the drivers, rebooted and re-installed them)

also get these once in a while:

TLD(0) drive read_element_status error

TLD(0) going to DOWN state, status: Unable to sense robotic device

TLD(0) key = 0x2, asc = 0x4, ascq = 0x3, LOGICAL UNIT NOT READY, MANUAL INTERVENTION REQUIRED

Request for media ID WUB438 is being rejected because mount requests are disabled (reason = robotic daemon going to DOWN state)

I checked Log files however nothing stood out for me so if you prefer I can post some of those as well.  I appreciate any help you can give me.

Edit:  Also my drive keeps getting Downed so I have to manually Up it to try a test backup again.  I have a backup that goes to Disk first and then Duplicated to Tape and also oens that go directly to Tape, same issues with both.  THe backup to disk works but not the duplication.

[RiaanBadenhorst]

Hi,

I think I can remember the L8, and as far I remember it only has one drive, and its a tape loader not really a robot. Big long loaf of a thing, right?

This means that you cannot switch to another drive in the library to test hardware issue.

Has your driver event error disappeared after the re-installation/reboot? Did you install the netbackup driver package? What type of tape drive is it, vendor, model, etc?

Do you still get unable to sense robotic device errors?

Just trying to figure out what your status is now, before we continue.

[Silvan]

Thanks for the quick reply.  I inherited this thing when I started this job but yes it is more of a tape loader than a robot.  It holds 8 tapes internally.  It's 2U rack mounted so it sounds like you remember it correctly.

I'm doing a reboot now just to make it easier to look at the most recent event logs.

The following boot-start or system-start driver(s) failed to load:

halfinchVRTS

is still happening.  Now I do recall when we did the upgrade to nbu 6.5.4 we had a couple issues getting the right drivers but it did work, and it's been fine since about January.  I had help with a NBU specialist which unfortunatly left the company about a month ago.  We also had some issues with a scsi card I was trying to use at first but ended up going to buy a new one which is in there now.  Unfortunatly I don't have a spare one so it's the only peice that I haven't really been able to swap out to see if it's causing an issue.  I could maybe try to put it in a different slot on the server.

The robotic device error hasn't come back Yet, for about 2 hours now but I only saw it in the event log once so far.  I'm going back a bit further to see if I can find it again. 

Other events I came across from yesterday when I was testing are:

emmlib_UpdateDriveRuntime failed, status=258

TLD(0) Move_medium error

TLD(0) key = 0x5, asc = 0x21, ascq = 0x1, INVALID ELEMENT ADDRESS

TLD(0) slot read_element_status error

[J_H_Is_gone]

It is a drive issue not a tape issue.

Do you have any logs for your loader and drive?

I would put in a repair call on the drive.

[Silvan]

unfortunatly there's no log files I can find for it.  There's no web interface to log into and on the nbu server all that was required were drivers.  The only logs that I have are the netbackup logs (bptm etc)

I'm thinking it's a drive issue as well, it's fairly old so I'm not sure if it's going to be worth putting in a repair call or replacing it.  If I don't have any progress by the end of the day then management will need to make that decision before going too long without a backup.

[RiaanBadenhorst]

If you have the correct drivers and you still get the error then it probably means that the drive is faulty, so drivers dont load, because the device is not working or present on the system, well not present enough to fully work.

What is the status of the tape drive in the device manager?

[Silvan]

Device Manager is fine.  Also on the front of the Tape Drive the green ready light is on.  The error/warning lights are all off and there's a "test drive" feature that I ran for about an hour this morning with no errors.

the last thing I can try to do is try to find another scsi card somewhere and swap it out.

[RiaanBadenhorst]

Hi,

You can try these drivers for 32 bit OS, or if its 64-bit use the vendor drivers. http://www.symantec.com/docs/TECH51096

You might also try and locate a testing util (software) from the specific tape vendor.

Give the SCSI a try but i doubt its the problem. Unfortunately its quite hard to test where the issue is with only 1 drive in the unit.

R

[Silvan]

THanks, i'll give those drivers a try in a few minutes.  I was actually just about to go that route and try and find some different or updated ones.

Just an FYI I went into Device manager and uninstalled the Tape Drivers and Media Changer drivers.  I shut down the server, powered of the Tape drive and turned the server back on.  That weird error below was still in the windows system log.

The following boot-start or system-start driver(s) failed to load:

halfinchVRTS

[RiaanBadenhorst]

That confirms what i thought, the message doesn't mean that the driver has an issue loading, it means it loaded, but could not complete/attach to anything. Or something to that effect :)

[Silvan]

alright, so do you think I should still try that other driver you posted? or should I try to locate a different scsi card first?  I'm not even sure I have one around so may need to buy one.

[RiaanBadenhorst]

I'd try the driver, then maybe swop the card to a different slot. Honestly, SCSI cards are less likely to die than drives. How old is the drive, i must be LTO2 or something like that? Maybe three, could just be on its last legs. The Drive Test util if available from the tape vendor should be a good test.

I wouldn't waste money on a new card. Maybe stick a SCSI disk on the card, and test it like that if you want to.

[Silvan]

well, I installed the new driver, and verified in Device Manager that a newer driver was installed.

I rebooted twice and the weird system driver error posted above has not come back.  However my backup jobs are still failing at the exact same spot.  sigh.  I find it odd that it mounts the media, positions it and then fails.

10/6/2010 2:49:04 PM - started process bpbrm (4788)
10/6/2010 2:49:08 PM - connecting
10/6/2010 2:49:10 PM - connected; connect time: 00:00:02
10/6/2010 2:49:15 PM - mounting WUB442
10/6/2010 2:50:12 PM - mounted; mount time: 00:00:57
10/6/2010 2:50:15 PM - positioning WUB442 to file 1
10/6/2010 2:51:56 PM - Error bptm(pid=1244) ioctl (MTWEOF) failed on media id WUB442, drive index 0, Data error (cyclic redundancy check). (23) (bptm.c.9494)
10/6/2010 2:52:00 PM - end writing
media write error(84)

along with Application event error:

The device, \Device\Tape0, has a bad block.  Source halfinchVRTS

[Will_Restore]

I'd be tempted to say WUB442 is a bad volume but you say it happens with all your tapes?

[Silvan]

yep there's 6 different tapes in there, i've rotated them all out also just to make sure and i'm getting the same issue with many different tapes.

[Silvan]

I'm back at it today.  Any other suggestions or should I be telling management I need a new tape drive?

[RiaanBadenhorst]

StorageTek SL5000 fully populated :p

[Silvan]

well we have an SL500 laying around in our storage room for the past 3 years, but I don't know if it's just the chasis or whats included.  Paid about 10K for it and it's still in a sealed crate.

[RiaanBadenhorst]

LOL, take it out mate, worst case, you have a funky bar fridge :)

[Silvan]

LoL, turns out it's just a chasis so wasn't very helpfull.  I've setup a 1TB cifs share on storage disk that I have however I haven't figured out how to add the disk pool in nbu yet.