cancel
Showing results for 
Search instead for 
Did you mean: 

NBU Encryption issues - Quantum and Symantec

HoldTheLine
Level 4

I could use some advice here - currently have cases open with the vendors but wanted to get some outside perspectives as well.

 

We are using NetBackup for a data migration, nothing fancy: Backup the current data at site A, ship the tapes to site B and restore.

Info for both sites:

site A:  Spectra T120, LTO5 drives, NBU 7.0 (I could not convince them to upgrade and we are doing catalog recoveries, so stuck at 7.0)

site B:  Quantum i500, LTO5 drives, NBU 7.0

 

It's been problematic - since day one the restore have been painfully slow, using LTO5 drives and 2-8gb connections I was seeing restore speeds no better than 5mb/sec.  There was a lot of troubleshooting done, cases opened and closed, optimizing, etc.  Nothing seems to break that barrier of horrible throughput.  We tested outside of NBU, network and disk speeds are just fine, there is no reason those restores should perform so poorly.

Then I had the idea to rule out the incoming tapes, try a local backup and restore with fresh tapes.  Bingo!  Decent speeds, no errors, etc.  Until I tried to use KMS - trying to write encrypted backups fails with these types of errors:

 

 Error bptm(pid=2272) FREEZING media id <Media ID>, Encryption unavailable for an ENCR pool 

Now this is where it gets odd - KMS has been replicated from site A to site B, and from the begining there were never any indications that anything was misconfigured; the tapes coming from site A were always able to be read, and running nbkmsutil on both sites shows identical info.  I followed the simple instructions on exporting/importing keys so no surprises there.

The case I opened with Symantec found errors in bptm that point to the hardware being the issue - Quantum got involved and are confused as well.  Heck, I am confused too - the tapes I am not able to create encrypted backups on are from the same pool we use in our production site, there are no issues with them.  I looked at them personally, they are in fact LTO5 tapes, no damage, and so far six of them give me that Encr error, yet I can run a non encrypted backup and restore with  them with no problems at respectable speeds,

 

While I wait for Symantec and Quantum to review logs I am still poking around trying to find clues, if anybody has seen anything like this please let me know.

 

 

 

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

HoldTheLine
Level 4

Finally have an update -

I was able to upgrade the master from 7.0 to 7.5 and run some tests, encrypted backups are working now.

Go figure...

Maybe I should have suggested upgrading before we started.  Oh wait, I did! :) 

 

View solution in original post

43 REPLIES 43

Will_Restore
Level 6

Found this old technote which describes the bits used in bptm log to verify both the media and the drive support encryption:  http://www.symantec.com/business/support/index?page=content&id=TECH87444

 

mph999
Level 6
Employee Accredited

Fairly sure there have been similar cases - KMS restores fail, down to a drive firmware issue.

Found it ...

http://www.symantec.com/docs/TECH204085

Now this may not match your issue, but it at least shows there have been issues in that area, and there is a possibility you have a slight variation perhaps ?

HoldTheLine
Level 4

Good information, thanks both of you.

 

The firmware question on the drives (and libraries) came up before as we knocked that out a week or so ago - both libraries and drives are now at the latest revs.  That being said, I have not had a chance to test a restore created after the upgrades - so there may be something to that.

HoldTheLine
Level 4

re: http://www.symantec.com/business/support/index?page=content&id=TECH87444

The media can be physically inspected to check the type.

 

I found this technote early on which prompted me to do a personal visit to the data center - I popped all 6 tapes out and looked at them.   Didnt see anything that jumped out at me, they are all LTO5 tapes, brand new.  Even ran quick erases on them.

Will_Restore
Level 6

On the i500, select Reports > System Information from the web client to confirm tape drive encryption settings.  Sorry, don't know about the other library. 

 

HoldTheLine
Level 4

Yea been down that road, looked at every Quantum setting there is. Not only is encryption on the library not enabled, its not even licensed so there is no proprietary encryption getting in the way.

mph999
Level 6
Employee Accredited

Check the basic tuning settings of the OS - eg. nofiles (should be minimum 8192).

We had a multiple cases on Linux for a customer where very very odd thngs were happening.  The one I was involved with, some of the backup headers on the VTL 'tapes' were in fact a bunch of 0's - not good when you try ad restore.

Not the same as you, but you have to agree it's in the same kinda area.

I have absolutley no idea how such a setting could cause such an issue - but once changed all the issues disappeared, 

jim_dalton
Level 6

There seems to be more than one issue at play here, perhaps you can clarify.

FWIW Having the latest firmware isnt relevant, what you need is the latest firmware that works! Been there done that. Backups were clean, couldnt restore any data.

So lets deal with encryted backups : need to check its supported for your device and for this version of NB and what limitations are, check you have it (KMS) configged correctly, check the policy is configged correctly and of course who is doing the encryption - which looks to me to be data at rest T10, like what I use. I would focus on the firmware and netbackup levels.

You need to explain the comment about "from the begining there were never any indications that anything was misconfigured; the tapes coming from site A were always able to be read, and running nbkmsutil on both sites shows identical info. ". So: simply put: whats changed? It used to work , now it doesnt.

We'll get there...

Jim

 

HoldTheLine
Level 4

Check the basic tuning settings of the OS - eg. nofiles (should be minimum 8192).

What/where is this setting?  It's not familiar to me.

 

Thanks

HoldTheLine
Level 4

FWIW Having the latest firmware isnt relevant, what you need is the latest firmware that works! Been there done that. Backups were clean, couldnt restore any data.

I agree - the first time we addressed this, during troubleshooting the Symatnec tech noticed the drive firmware was very old so suggested we upgrade it.  So we did - now it's at the latest but like you say the most recent is not always the best

 

You need to explain the comment about "from the begining there were never any indications that anything was misconfigured; the tapes coming from site A were always able to be read, and running nbkmsutil on both sites shows identical info. ". So: simply put: whats changed? It used to work , now it doesnt.

You are right I do need to explain that - I may have muddled things up here.  It never really did work - at least not very well.  What I meant by "From the begining" was the restores - we have so much data to move that since this environment has been built all it has been doing is restores.  Each client was taking 3-6 days to complete so they were running around the clock.

 

That is what I meant by "from the beginning..." - I may be wrong but my understanding of KMS is that if it was not configured  correctly at both sites, would we be able to read tapes at all?  If I had been asked that question 2 months ago I would have said "Heck no, it's an all or nothing deal - if KMS is not set up you will not be able to read anything at all"

 

At this point I am not so sure - there is literally no other reason we can see that these restores should trickle out at 2-5mb/sec when the network, disk and tape are all capable of much greater performance.

 

So what has changed:

 

LTO5 firmware updates (didnt affect restore speeds)

The backup tests I have done recently were never done before because we were concentrating on troubleshooting the performance of the restores - once I had a lull it occured to me to try local backups so that is a very recent development.

I am just as confused as anybody by all this ...

 

 

jim_dalton
Level 6

I would agree with KMS...if the data coming out of the restore are unencrypted then the decryption is working, whats left is a performance issue.

But that doesnt explain your log snippet ...Encryption unavailable.

Do yourself a favour: on the target master only , find yourself a big fat file, a few Gb , (an iso supplied by Symantec say!), create a policy for it, write it out to tape, restore from tape to a different location and compare with original and observe.

Bugs and misconfigurations aside, this will be rapid both ways and will be encrypted to tape and decrypted on restore. You might want to check this ie report on the image/observe during both backup and restore that encr/decr is happening.

I would add to check the blocking factor on tape: this can slow things: but thats easily checked from the backup viewpoint: what speed backup do you get? ( Its a good idea to up the blocksize for performance to 256k or more - check for LTO5 what you can use). I dont think this is the issue, as you would have logged it as a backup issue not a restore issue. The blocking is (normally) handled automatically on restore since you cant do anything with it at that point. Tape drive issues notwithstanding.

How long have you been running with this setup?  Any other odd issues? I ask since if there are name resolution issue this can be very detrimental to netbackbackup generally.

This sounds like a fun interesting problem, my rates are very reasonable!

 

Jim

HoldTheLine
Level 4

I would agree with KMS...if the data coming out of the restore are unencrypted then the decryption is working, whats left is a performance issue.

But the data coming out of the restore is encrpyted - at least KMS is set up, and it is using an ENCR_ pool.

 

Do yourself a favour: on the target master only , find yourself a big fat file, a few Gb , (an iso supplied by Symantec say!), create a policy for it, write it out to tape, restore from tape to a different location and compare with original and observe.

Interesting idea - assume you mean unencrpyted, correct?  Well, thats my only option since any attempt at a new encrpyted backup fails :)

 

 

How long have you been running with this setup? Any other odd issues? I ask since if there are name resolution issue this can be very detrimental to netbackbackup generally.

 

The DR site has been configured for about 1 1/2 months.  The prod site  pre-dates me, as far as I know since it is still running 7.0 it's been around since - 7.0 came out!  Before we started this project they were using the Spectra proprietary Encryption.

Wonder if we should have tried a backup out there after running a long erase on the media, just in case Spectra left anything wierd on the headers.

 

Of course that doesnt help with the current situation, but something else occurs to me:  If the tapes I am trying to use in this 7.0 environment were once upon a time used in a 7.5 Encrypted environment, might these be some of the symptoms? i.e. can't encrypt because its already encrypted via a method that 7.0 era KMS doesnt know about?

 

jim_dalton
Level 6

Interesting point re versions, that could have mileage - and indeed more work for you!

I am not familiar with the on-tape format: dd would be the tool here if you were unix, hopefully others might chip in with information on the subject under Win.

You say the data coming out of restore is encrypted. How do you know? To clarify my thought: determine its encrypted on write and decrypted on read: once its back on disk you wont be any the wiser.My drives tell me "encrypting" when I write, and of course the reverse on read.The catalog should also tell you. 

For whats its worth, I've done this kind of exercise as part of DR several times and it should work fine.

You are certain the two environments are the same versions/rev/patches?

Jim

mph999
Level 6
Employee Accredited
It depends on the OS - solaris it is viewed in ulimit -a and can be set in /etc/system Could you confirm the OS involved

HoldTheLine
Level 4

It depends on the OS - solaris it is viewed in ulimit -a and can be set in /etc/system
Could you confirm the OS involved

 

Oh that would help, wouldn't it! :)

 

All systems involved (Both masters and one media server) are Windows 2008 Server R2 Enterprise

mph999
Level 6
Employee Accredited
OK, far as I know no concerns with this on windows, I think the max is around 16000 which is fine. This issue is more unix/ linux which has a low default that requires increasing to min of 8192

mph999
Level 6
Employee Accredited
OK, far as I know no concerns with this on windows, I think the max is around 16000 which is fine. This issue is more unix/ linux which has a low default that requires increasing to min of 8192

mph999
Level 6
Employee Accredited
You mentioned earlier, if KMS wasn't configured correctly it wouldn't work at all - I agree, all or nothing. I think it is also 'very unlikely' for the config to give intermittent results (that's my sensitive way of suggesting not a chance ... but with a 0.1% get out clause in case I'm wrong ... ;0) ) I am wondering if this is something to do with the data held on the internal chip in the cartridge - I don't know if this is used in KMS, but it does hold quite a lot of info about the tape, so there is a possibility it is. Can you advise of the tape brand ? This may be different from what is written on the tape (Eg. Oracle branded tapes are actually made by Imation (or at least they used to be)). If you look in the bptm log, you should see the manufacture listed - I guess searching for 'man' might narrow down the search. If they are fuji, we might be in luck as I know someone who works there who might be able to confirm a couple of things. Is there any current Symantec case number as a matter of interest ? Even if not, I can ping an email direct to to a few people to see if we can get some more ideas. To answer Jim's question on 'dd' for win. I think we're stuffed there, I am not aware of any 3rd party equivalent. There is a utility within NBU for windows, but I can't remember what it's called and I think it's only useful for positioning - I'll ask about.

Will_Restore
Level 6

>>since any attempt at a new encrpyted backup fails :)

 

Wait a sec...  Does this mean you can't write a new tape?