Excessive tape unmounts

[mrtba]

Good afternoon,

I recently was having ECC errors on my tape drive.  Those are gone due to new drives.  I noticed that I get an escessive amount of unmounts with my tape drive while waiting for ESX to stream the data to the drive.  That causes the tape to unmount.... 

I'm running windows server 2k8R2 64-bit with a MSL4048 with 2 LTO 5 drives. 

I have mulitplexing turned on to speed up backups already.  This works great on full backups but on daily's when the amount is small, the tape unmounts and mounts many times while proccessing our backup policies.

I'm using Windows-Flashbackup on ESX4.1 and 5.1.  I have media multiplexing turned on to 2, and only daily's do this. 

Anybody know if I can set no rewind in Windows?

I have already tried setting the Media unmount delay to 1800 seconds and it seems to have no effect. 

[Nicolai]

If you use many different volume pools and retentions Netbackup will have to change tape to run the backups.

[mrtba]

Nicolai,

When I'm running daily backups all rentention and pools are the same. 

[Nicolai]

You need to tell me some more. How do you discover that the tapes are unmounting ?

[mrtba]

Nicolai,

I hope these three logs help you.  The first one is of the first daily job that ran last night.  The second, a random middle job that didn't unmount the tape, and a thrid where the backup had to mount the tape and position the tape to the correct file. 

First log showing the tapes mounting and positioning

4/29/2013 8:00:40 PM - Info nbjm(pid=4312) starting backup job (jobid=57679) for client %CLIENTNAME%, policy %POLICYNAME%, schedule Daily 
4/29/2013 8:00:40 PM - estimated 916586 Kbytes needed
4/29/2013 8:00:40 PM - estimated 916586 Kbytes needed
4/29/2013 8:00:40 PM - Info nbjm(pid=4312) started backup job for client %CLIENTNAME%, policy %POLICYNAME%, schedule Daily on storage unit%MASTERSERVER%-hcart2-robot-tld-0
4/29/2013 8:00:41 PM - started process bpbrm (5960)
4/29/2013 8:00:55 PM - mounting EV1009
4/29/2013 8:00:55 PM - connecting
4/29/2013 8:00:57 PM - connected; connect time: 00:00:02
4/29/2013 8:02:03 PM - mounted; mount time: 00:01:08
4/29/2013 8:02:03 PM - mounting EV1518
4/29/2013 8:03:15 PM - mounted; mount time: 00:01:12
4/29/2013 8:03:23 PM - positioning EV1009 to file 727
4/29/2013 8:03:41 PM - positioning EV1518 to file 1
4/29/2013 8:03:44 PM - positioned EV1009; position time: 00:00:21
4/29/2013 8:03:44 PM - positioned EV1518; position time: 00:00:03
4/29/2013 8:03:44 PM - begin writing
4/29/2013 8:04:48 PM - end writing; write time: 00:01:04

Second log showing that the tape didn't have to mount or position

4/29/2013 8:07:51 PM - Info nbjm(pid=4312) starting backup job (jobid=57691) for client VIEWD06v1230, policy SLVS6-Main, schedule Daily 
4/29/2013 8:07:51 PM - estimated 724077 Kbytes needed
4/29/2013 8:07:51 PM - estimated 724077 Kbytes needed
4/29/2013 8:07:51 PM - Info nbjm(pid=4312) started backup job for client VIEWD06v1230, policy SLVS6-Main, schedule Daily on storage unit %MASTERSERVER%-hcart2-robot-tld-0
4/29/2013 8:08:00 PM - connecting
4/29/2013 8:08:02 PM - connected; connect time: 00:00:02
4/29/2013 8:08:02 PM - begin writing
4/29/2013 8:10:01 PM - end writing; write time: 00:01:59
the requested operation was successfully completed(0)

Third log showing the mount and position.

4/29/2013 9:04:05 PM - Info nbjm(pid=4312) starting backup job (jobid=57733) for client %CLIENTNAME%, policy %POLICYNAME%, schedule Daily 
4/29/2013 9:04:05 PM - estimated 632938 Kbytes needed
4/29/2013 9:04:05 PM - estimated 632938 Kbytes needed
4/29/2013 9:04:05 PM - Info nbjm(pid=4312) started backup job for client %CLIENTNAME%, policy %POLICYNAME%, schedule Daily on storage unit %MASTERSERVER-hcart2-robot-tld-0
4/29/2013 9:05:55 PM - mounted
4/29/2013 9:05:55 PM - connecting
4/29/2013 9:06:20 PM - connected; connect time: 00:00:25
4/29/2013 9:06:28 PM - mounted
4/29/2013 9:06:28 PM - positioning EV1009 to file 774
4/29/2013 9:06:28 PM - positioning EV1518 to file 48
4/29/2013 9:06:28 PM - positioned EV1009; position time: 00:00:00
4/29/2013 9:06:28 PM - positioned EV1518; position time: 00:00:00
4/29/2013 9:06:28 PM - begin writing
4/29/2013 9:08:14 PM - end writing; write time: 00:01:46
the requested operation was successfully completed(0)

[Nicolai]

Netbackup will not let a tape be mounted unused for a hour.

First job end 8:04:48PM

Next job start at  8:07:51PM

Last job start at 9:04:05PM

That almost a hour and much longer than media unmount delay of 1800 (½hour). According to this tech note http://www.symantec.com/docs/TECH31867 it's not possible to set media unmount delay larger than ½ hour. I would not care to great about the wear of tape drive and robot upon mounting and dismount tapes - They are built to to excatly to do that. 

[mrtba]

Nicolai,

Thanks for your prompt reply.  There are many jobs inbetween there....  The tape drive is not idle for that long.  Maybe 5 minutes tops.  Setting that media unmount delay has no change in the amount of mounts.

What is happening is this...

Netbackup says "Hey ESX give me a backup", then ESX snapshots the machine and starts streaming the data to it.  That process takes awhile.  The daily backups happen in seconds so the 5 minutes or so it takes ESX to start the stream there is no data being written to the tape drives and there inlies the problem...  Any thoughts on how I can fix this?  I really doubt it's in the netbackup software....  This might be an HP issue.  We have many tape robots and the only one that has failed has been this one that is backing up ESX and the only thing I see is excessive mounts and unmounts...  When I say excessive I mean like 30+ a night.  Also the this tape drive lasted under a year(2 drive failures) where as the other ones have been going for well as far as I have been here which is 4+ years.  

[trv]

Are the drives flagged as 'cleaning required' by any chance ? If yes NetBackup will unmount the tape as soon as possible ignoring media unmount delay.

[mrtba]

Trv,

When you say flagged 'cleaning required' I'm guessing you mean that is happenning on the fly.  I have seen the tape drives say that before on rare occassions.  We keep a cleaning tape in the robot for such occassians but it rarely gets used.  I looked through my logs and I don't see that the drive required any cleaning the last few days.

Any other ideas?

[trv]

If the drive needs cleaning, there should be something like 'needs cleaning' in drive status window. You can check it from command line - tpclean -L (last column will be n/a or needs cleaning). I have seen drives in constant needs cleaning state with library based cleaning, but if you let NetBackup handle the cleaning this shouldn't be an issue. Well I am out of ideas ...

[mrtba]

Thanks trv,

The tape drive is not being cleaned...  I think it may be the hardware from HP.  I could try and contact HP on this....  We have maintance on the tape drive so if it fails we will just replace it again.  I don't have time to contact HP right now.

Thanks for all your help.  Sorry we don't have a resolution.