Exchange GRT backups fail suddenly - beremote crashes in virtapi.dll

[rogerroger]

Hi all, we are experiencing sudden problems with GRT backups of Exchange. It worked fine for months and in a sudden the remote agent crashes with errors in module virtapi.dll. Our configuration: - Win2008SBS German Server - Exchange 2007 SP2 - backups directly on the media server - all hotfixes/patches for BE 12.5 installed - server patched Backups without GRT are working well - of course. I went through all the hints in the forum and knowledgebase - no luck. Can anybody help us? I forgot: Beremote crashes since the 6th of April - changes since then: - MS IE patch on all servers - switch from append mode on NAS to overwriting existing media sets (Returning to append mode did not help!)

[Martin_Forster1]

THe service id is for example on your license letter

[rogerroger]

After repair and reboot:
no change - still beremote crashing...

[rogerroger]

...attached the job log:

[Martin_Forster1]

Hallo,
im vssadmin list writers kommen da fehler ?

[rogerroger]

...ok - wo finde ich die denn?
Habe mal ein Backup to Tape gemacht da bekam ich diese Fehler:

Sichern- MPGSERVER
V-79-57344-65233 - AOFO: Initialisierung fehlgeschlagen: "\\MPGSERVER\Microsoft Information Store\F...

Fehler im Snapshot-Provider (0xE000FED1): Beim Abfragen des Writer-Status trat ein Fehler auf.

Ausführliche Angaben finden Sie in der Windows-Ereignisanzeige.

Verfassername: Exchange Server, Verfasserkennung: {76FE1AC4-15F7-4BCD-987E-8E1ACB462FB7}, Letzter ...

Sichern- \\MPGSERVER\Microsoft Information Store\Second Storage Group
V-79-57344-766 - Mit diesen Anmeldedaten ist die Anmeldung bei MAPI nicht möglich. Überprüfen Sie d...


Es nervt jetzt langsam.

[Martin_Forster1]

Hallo 

ja das nervt,
vssadmin ist ein befehl den du in der kommandozeile eingeben kannst.

außer einen Hotline Call kannst du nur noch trial und error machen.

Du kannst/ solltest zb. die ExchangeCDO Mapi noch aktualisieren.
Du kannst noch ein Information store backup ohne GRT testen.

Irgendwann werden wir es schon finden ...

[rogerroger]

Hi,
also:
- vssadmin list writers gibt mir für alle stabil - keine Fehler aus
- die EXCDO mapi ist die v.1.2.1 - scheint mir älter habe ich allerdings frisch runtergeladen und längst installiert gehabt da von Exchange verlangt
- Backup ohne GRT fummuliert einwandfrei, allerdings kann ich dann keine einzelnen Mailboxen mehr zurückholen, was doch eigentlich der Zweck der Übung ist, oder liege ich falsch? Gibts Alternativen?
Alles in allem recht merkwürdig...

[rogerroger]

UPDATE!!!!
GRT backup to a Win Server share works!!!!
So, what does not work is:
- GRT backup to NAS (worked for some time)
- GRT backup to tape

What is that?

[Martin_Forster1]

HI,

GRT to NAS must be some limitating facttor of the nas, most common is file size, but we addressed that before, so ?
GRT to Winodws share, with ntfs as file system does not have such limitations.
GRT to tape, is technically not possible, what happens is a temporary copy on Harddisk which gets deleted.
This copy  is placed in  the temporary GRT Folder.  You probably have a Harddisk space problem there. We also addressed that ?

Regards

PS single mail restore, can also be done with the usage of a recovery storage group, but it is more complicated.
But it is how MS planned it to do Single Item Recovery....

[rogerroger]

Hi again,
did some resetting on the NAS - will try again this evening.
Keep you up to date...
Thx.

[rogerroger]

NAS problems cont.:
Upgraded the NAS to the newest firmware found on the LG sites and did some testing again.
The difference between GRT and non-GRT backup is that BE stores the GRT backup in one database image file and the non-GRT backup stores in 4GB files.
The db image became bigger than 16GB now.
Is there a problem perhaps???
Can't find any 16GB limitation on the LG NAS...
Has anybody an idea?

Thx.

[Martin_Forster1]

HI,
it depends on the file system used by LG. Unfortunately i wasnt able to find a documentation.

But ext3 could have a minimal file size of 16 GB.

[rogerroger]

Hi Martin,
I managed to copy bis files (25GB) via Win to the NAS and back without any problem - so this is not a general limitation.
But something happens to the agent in virtapi.dll - BE succeeds in backing up but short after the agent collapses and the backup is marked as bad. A file bigger than 16GB is on the drive!
I think it could be happening when BE tries to catalogue the backup or something like this and it does not get the right feddback from the NAS?!?
The NAS has a Linux kernel so it should be using SAMBA for Win...

grrr....

Backups to Win NTFS shares work on other servers/Win machines - so buy a new NAS with real NTFS???

PS: Kill that cat!

[Martin_Forster1]

HI,

check if its supports iSCSi, this would be a good workaround for the whole filesharing stuff.
edit: Hey, LG NAS seems to support iSCSI: http://storage.lgblog.de/2010/01/14/lg-nas-zurucksetzen-iscsi-und-die-antwort-auf-andere-fragen/
http://forum.nas-portal.org/showthread.php?9729-n2r1-als-iscsi-target-verwenden

But i think you need invest money in a new box.

Normally i prefer Qnap boxes for such usages. 
Aother good way is openfiler.
And im really intersted in Freenas and netxenta bec. of their ZFS support, which has many extremely good features.
Unfortunately Freenas is still to home user focused, and netxenta looks somehow funny.
For example they seem to be quite unsure if they use opensolaris or a BSD slang.

Regards

[teiva-boy]

Some thoughts...
Run iSCSI if you can on that NAS.  Many can, others you can hack into ;)
Try limiting to 3GB files or 1GB files, so as to not be right at 4GB or right at 2GB.
Have you tried the B2Dtest.exe program yet?  It tests a NAS for basic functionality, and I'm curious if your NAS would pass or fail. Find it on the support site.
Are there any firmware updates for your particular LG NAS?

Lastly, try a different NAS.  Iomega has great success with BackupExec, QNap, Promise, and Buffalo are also all excellent brands that are in the same price range/class as the LG NAS.

[rogerroger]

Hi all,
there is no way to use iSCSI with our NAS - it's an LG N4B1 with firmware 4261 which I think is the newest.
@teiva-boy:
Changing the file size to 3 or 1GBs won't have any effect because with granular backups BE uses the whole Exchange DB in one file which means about 20GBs at the moment.
I don't know if it's going to work with another NAS - has anybody made succesful GRT backups with other NASes?

I think I found it in the B2Dtest log: reparse points and sparse files are not supported - whatever this is!!!

But why did it work until the imsge file reached 16GBs???

Maybe somebody has got an idea...

[rogerroger]

...just read that Symantec BE only supports real Windows NASes!!!!
Somebody knows some of these?

[rogerroger]

...or would it be working with iSCSI?
But how putting iSCSI functionality on a LG N4B1?
 

[teiva-boy]

Not True at all!!!  It supports non-windows NAS devices too.  Look in the HCL for the items like buffalo and Iomega mentioned and a few others.  All running some sort of linux OS.

When I said limit the file size, this is in the B2D folder properties in BackupExec in the device view.  You should be able to break up the file size to 3GB or even 2GB?  

But I do see in your logs, your device doesn't support reparse points and sparse files, as well as not being able to open a large number of connections.  This just goes to show, not all NAS devices are created equally...  You would have to get under the hood of SAMBA in the linux OS to change these settings.  It's possible the connections limit is just due to performance reasons and it's own CPU not being powerful enough.

[rogerroger]

Hi there,
just did a GRT full backup of exchange - and it crahsed again :(((
It seemed to me that the BE agent crashed during the catalogue update process, after having completed the data copy.
Maybe this is the problem with the open file handles BE needs and the NAS is not able to deliver.

Does anybody know how to manipulate the SAMBA settings on the LG NAS? I can't telnet to the device - only webinterface with no samba settings.