Restore Backup w/o catalog

[Doomshammer]

Hi there,

I've got an issue with one of my customers. Their NetBackup Server has been stolen during a burglary. They have no catalog backup (only the one on tape). So I tried to import from the given media, to reproduce the catalog.

In phase 1 I was able to find a full backup spread over 3 tapes- which is good so far... but now comes the bad thing about this. I started phase 2 on this full backup to recreate the catalog, but this phase 2 failed at the last media with a "media read error"- so it seem the 3rd of 3 tapes is broken. The other 2 tapes were read successfully during phase 2 (the log shows all the files that have been backed up), but as the import failed with the last tape, the catalog wasn't written.

Now my question. Is there any possibility for me, to read at least the data that is stored on the first 2 tapes (the data, that showed up in the logs) w/o the catalog. NB-Server is a Windows 2003 box with NB 5.1. The tape robot is an older Overland with a Quantum DLT7000 drive. I already tried to access the tape directly with a Solaris box (with GNU tar and mt) but this wasn't successful. I already received s.th like this:

gtar: Record size = 2 blocks
gtar: This does not look like a tar archive
gtar: Skipping to next header
gtar: /dev/rmt/0n: Cannot read: Not enough space
gtar: /dev/rmt/0n: Cannot read: Not enough space

(I also tried other densities). Any help would be appreciated!

[Stumpr2]

Sorry, but I'm not sure of the scenario. Is this correct?

A NetBackup 5.1 windows master server no longer exists. But you do have a full OS level backup of the missing master server. Now, you are trying to import and restore the missing master's $INSTALL_DIR\Netbackup\db\images from a regular NetBackup OS level backup of that folder onto a new master server that is a Solaris platform?

[Doomshammer]

Hi Stumpr,

sorry for being imprecise. Let me provide some more details...

At one of my customers, there was a bulgary where all servers have been stolen (Backup server as well as file servers and stuff). Now that there is neither an externaly stored catalog backup, nor the backup server is available. We set up a new server as backup server. As the most important thing is to recover the file server, I startet a phase 1 import on the tapes I was given. There I found a (very new) full backup of the file server. So I started the import phase 2 on this image that has been found over 3 tapes. The phase 2 works for tape 1 and 2 (I can see all the files that have been backed up in the extended import log), but when the import phase 2 gets on the 3rd tape, it fails as the tape seems to be damaged. So I wasn't able to recover the catalog, to start an restore of the file server.

So my thinking is... as I can see the files on tape 1 and 2 (from the extd. log), there must be at least a way to restore these files (for my sake even without NetBackup). So I need to figure out, how to access these backups without having a valid catalog, which i could use for the "Backup, Restore" utility.

The old NB server was Windows 2003 Server and the new server is as well. Both were (and are) running NB 5.1. It's very frustrating to see that there are files, but you can't access them. So any hint or help from you would heartily appreciated.

Thanks in advance for your reply!

[Darren_Dunham]

---

Now my question. Is there any possibility for me, to read at least the data that is stored on the first 2 tapes (the data, that showed up in the logs) w/o the catalog. NB-Server is a Windows 2003 box with NB 5.1. The tape robot is an older Overland with a Quantum DLT7000 drive. I already tried to access the tape directly with a Solaris box (with GNU tar and mt) but this wasn't successful. I already received s.th like this:

gtar: Record size = 2 blocks
gtar: This does not look like a tar archive
gtar: Skipping to next header
gtar: /dev/rmt/0n: Cannot read: Not enough space
gtar: /dev/rmt/0n: Cannot read: Not enough space

---

On Solaris, the "Not enough space" error is coming from the tape driver, and indicates that the application reading (tar in this case) allocated buffer space that was smaller than the block size on the tape.  (Not enough space to store the tape block in memory).  Usually the 'b' flag can bump that up. 

Also, besides using 'gtar', I would use the netbackup installed copy of tar (usually /usr/openv/netbackup/bin/tar).

Another method of working the tape blocks is to read the tape via 'dd', then pipe that into tar.  Here I use 64k as a block size.  Increase it if you continue to get "not enough space" (but 64k is usually good).

cd /where/to/restore
dd if=/dev/rmt/0n bs=64k | /usr/openv/netbackup/bin/tar tf -

(tf to read the contents, xf to do the restoration).

Don't worry about the tape density flags.  They don't matter on reads, only on writes.  When you're reading a tape, the density of the written data is always used.

Good luck!

--
Darren

[Doomshammer]

Hi Darren,

thanks very much for this hint. I will try this tomorrow. I was using the Solaris gtar, as I only had a Solaris x86 installation accessable but the NB binaries are SPARC-only. But to avoid this, I just set up a SPARC box with NB to avoid any complications here. Thanks again.

[Colin_North]

If you are trying to restore from 3 tapes and the 3rd one is damaged/corrupt, can you not recall the previous days set of backup tapes from offsite and use those ones instead? All that you'll lose is one days worth of data, better than having none at all!

[Doomshammer]

Hi Colin,

sadly a bunch of tapes has been stolen as well- plus my customer told me that they had some issue with their backup lately. So the 3 tapes I currently have, is the latest backup since weeks :(

Anyhow I made some progress. I was able read a complete dump of one tape to my local solaris disk via "dd": As the tape is very slow, I though this would be easier if one of my tries doesn't work. I did it like this:

dd if=/dev/rmt/0n bs=64k of=tape1.out

This worked w/o problems. Then I moved this file to my old SPARC where NB is installed as well. I tried to extract the dump with tar (solaris tar, gnu tar and nb tar) but it didn't work. Though GNU tar told me about 12 lone blocks, so I tried it this was:

dd if=tape1.out bs=64k | /usr/openv/netbackup/bin/tar -xvif -

It seems as if this worked. I was able to recover 26GiB of data from the file but when I now try to access the files, more then the half of the files are corrupt, as the have some strange binary stuff in the file header. I actually have no idea why this happens.. might it be caused by the blocksize I've choosen? Any other hint to avoid this file corruption would be very appreciated.

Thanks!
Winni

[Doomshammer]

Just another detail which I forgot to mention. During the "tar" I received a lot of these messages:

Unknown file type 'L' for .SeCuRiTy.58, extracted as normal file
/D/xxxxxxxxxxxx/30sec Teaser/_verwendet/file1.flv

Unknown file type 'L' for .SeCuRiTy.59, extracted as normal file
/D/xxxxxxxxxxxx/30sec Teaser/_verwendet/file2.flv

Unknown file type 'L' for .SeCuRiTy.60, extracted as normal file
/D/xxxxxxxxxxxx/30sec Teaser/_verwendet/file3.flv

[...]

Might this be related? Or do I get into issues as the orignal backup was done by Windows NB and is now extracted on Solaris?

[Darren_Dunham]

Yes, this is very likely a windows/unix thing.  I think the files you're seeing are for the ACLs, which do not have a counterpart on a traditional POSIX filesystem.  However, that doesn't really explain the corruption in the other files.  That shouldn't be true even with this cross-platform restore.

You should probably be putting the files on the same type system as the original client, and using the same version of the tools to extract.  (I'm assuming the windows client has a 'tar' to use).

The block size will not be related.  Once it comes off the tape, it's just a bytestream and the tape blocking is no longer part of the file.

--
Darren

[Doomshammer]

Darren,

once again a big thanks for your hint. I already tried to extract the file on the NB server itself with the given tar32.exe, but that wouldn't execute at all. But the idea with the client, could do the job. Thanks!

Winni

[Doomshammer]

Hi again,

I'm a bit further again. Darren was absolutelly correct, the Windows "tar32" handles the backup way different and with it I am able to recover files, that are not corrupt.

So far so good. But I'm still at another problem (which I've already seen with the Solaris box as well)...
I am reading the input from my Solaris box (as this is connected to the tape and has "dd" so that I could dump the tape". I am now using the following commands to extract the files directly on the customers windows box:


ssh.exe root@192.168.182.235 "dd if=/foo1/tape1.tar bs=64k skip=12" | C:\Programme\VERITAS\NetBackup\bin\tar32.exe -x -v -Y -p -P -Q

As said this works so far- at least for the first 90 files. After this a null-byte follows (EOF) and no further data is extracted by tar32.exe. When I did this on Solaris, tar simply stopped but on Windows it doesn't. As workaround on Solaris I could use the paramter "-i" to ignore the zero-bytes, but unfortunatelly the Windows tar32.exe doesn't have this option. And as there is no help or parameter overview for the windows tar32.exe (NetBackup version) and as nothing is sent to STDOUT by this tool- I can't figure out how to get rid of the null-bytes. All STDERR is written into a log files in C:\Programme\Veritas\Netbackup\logs\tar\ but STDOUT goes nowhere. The logfiles tells me that neither "-h" "-help" "--help" "+help" "/?" "/help" nor "/h" is a valid parameter for tar32.exe but of course it doesn't tell me about the valid parameters.

I as well tried adding a "| tr -d \000" behind the "dd" output, but tar32 seem not to like this at all, as not one byte was extracted.

I'm a bit frustrated- I'm sooooo close but as well very very far from the final restore. If anyone has another hint for me it would be very appreciated.


Winni

[Darren_Dunham]

While normally very reliable, I have found that several 'ssh' implentations that have problems with certain data.  Is there any chance that the 'ssh' is choking on the null rather than the tar?

Can you copy a portion of the data from tape (making sure it includes a null) and get it on the windows client.  There you can test the tar and at least verify if the tar is choking on the null (I'm actually hoping it's not), or if the ssh data channel is stalling with the null data?

--
Darren

[Doomshammer]

Hi Darren,

sadly I already tried this w/o success. I copied the complete dump over to the Windows box and ran it through "dd" with same results. Then I even tried to use "type" (just to avoid any imcompatiblity with cygwin) but the result was even small- no file was recovered (as I can't skip bytes like I can with "dd").

Any other advice?


Winni

[Darren_Dunham]

I was looking a bit at the tar format, and I'm presuming that rather than a null byte you mean a null block.  EOF should be indicated by two null blocks.  Null bytes are common in the format and shouldn't cause any problem.

When reading the file on solaris are there two null blocks in the datastream?  that seems odd.

gnutar on unix can be started in the "middle" of the archive.  Can you trim off the portion of the data prior to this null block and hand the rest to tar.exe on windows?

--
Darren