Restores start fast and then wind down to nothing

[Fischer]

Hi guys,

It's me again, and I would like to say I appreciate all the help on these forums.  I've solved a lot of problems off of all your suggestions.  So, here is my latest problem.

I am running into an issue where a restore job to either a linux or windows box will start out very quickly, I average about 100 gig/hour or more on the restores, but at some point the restore bascially stops.  It doesn't fail and it is still processing, but the kb/sec goes down to anywhere between 100 and 38 KB/sec.  I had a restore today of about 600gig run flawlessly up until it reached about 400gig and then it started processing at 38 KB/sec.  On my linux restores I've seen them start off very quickly and anywhere between 50gig restored and 200 gig restored they start suffering from this slow speed.  I am doing multiplexing of 4 on these backup jobs, but I don't think it's multiplexing or fragment size,  because I am able to get substantial pieces restored before it starts moving at .000001 miles an hour. 

Any help is greatly appreciated.  Thanks

[Fischer]

I think I have tracked the source of the slowness down to one of my two lto-3 drives.  Restores seem to use more than one drive if they can for swapping tapes etc.  I noticed yesterday that my restore slowed after it swapped tapes and started using the second drive.  I tried a restore today with just my second drive and it never really started.  I'm confused though, because that second drive works perfectly fine for backups.

[J_H_Is_gone]

I am thinking old school here ( and showing my age) not sure if it true of new tape drives.

any chance the tapes you are restoring are old(er) or made on the other drive.

Tape drives use to have an issue of 'head drift'.

Meaning the tape drive head is not in effect alignment with the tape it is trying to read and may have to retry a lot to get the data off of it (shoe shine to read).

So if the head that wrote the tape is not at the same position as the one trying to read it, it might have an issue, of course this means that it would write just fine.

Might be off base, but just the first thought that came to my head, was head alignment.

[Darren_Dunham]

---

@J.Hinchcliffe wrote:

Tape drives use to have an issue of 'head drift'.

---

I won't say that can't happen any more, but I don't think it does in the same way.

Older drives had fixed-position heads, or heads that were explicitly positioned on the tape via a calibrated motor.  LTO (and DLT?) tracks are too small to be tracked that way.  Instead the head is dynamically positioned by reading calibration tracks on the media itself.  So while there might be tracking problems with the drive, it wouldn't be a simple drift-over-time issue.

-- 

Darren

[Fischer]

At this point, I'm pretty sure it's the one tape drive.  If I try to cancel a job it takes hours before the drive is finally released and the tape it was trying to restore from  is ejected.  I'm currently working with support to send them a tar file of the restore process.  They are saying that it is potentially hardware, but I can't see how it is a hardware failure if backups work perfectly fine on that drive...

[Fischer]

Ok, I've been working with Support and maybe I'm just an idiot and don't understand.  Can one of you please teach me? I just found out that the drive that is having trouble restoring, can restore a backup job that it backs up.  So, let's say I have drive 1 and drive 2.  If drive 2 backed up 100 gig of data, it can restore it with no problems.  But, if drive 1 backed up 100 gig of data, then drive 2 cannot restore it at anything more than a snail's pace.

Support said this might be due to multiplexing, and I am using multiplexing on both drives.  They are both set to 4 and most of my policies are also utilizing multiplexing.  I have a lot to backup in a short amount of time...

Anyway, I was under the impression that yes multiplexing would slow your restores, but I didn't know that it made your tapes, drive reliant to get your restore done in this century...

Thanks

[Darren_Dunham]

What I'd do would be to replicate this without involving Netbackup.   Is this unix?  Can you access the drives directly?

I'd grab a couple of scratch tapes, use 'dd' to throw enough data on there to get a reasonable restore, then test using 'dd' to pull the data off from each drive.

If you see the same time differences, then you know it's got nothing to do with Netbackup or multiplexing at all.  It must be the drive or system or something at that level

-- 

Darren

[Fischer]

I was able to duplicate the same problem on my old linux system. We just got in a new drive and it fixed the problem. Thanks a lot ofr all of yoru replies.

[Stumpr2]

Thank you for following up on your post. It is always a learning experience for all of us when a solution is presented. now you have educated us and we will know to mke sure the hardware is good before spending a lot of effort in researching other possibilities.

thanks!