cancel
Showing results for 
Search instead for 
Did you mean: 

BackupExec 2012 fails Exchange GRT backup after GRT restore

BaffledBackup
Level 3

SBS 2011 (Server 2008 R2, Exchange 2010, SQL 2008), Backup Exec 2012, all fully patched. RD1000 USB 3.0 backup drive.

Backups of system and Exchange running flawlessly since build three years ago.  Performed random restores of files/Shareport/Exchange (always full), no errors.

A week ago I needed to do a GRT restore of a user's folder and 97 messages.  Worked great, no errors, then suddenly the GRT backup fails with the following:

Capture.PNG

Further, Exchange transaction log files aren't being purged.  Seeing Events 9287 and 2007 in Application log.

Ran Windows Server Backup to test, worked fine, purged transaction logs.

Tried running BEX 2012 full backup of Exchnage only with GRT disabled, same error.  Tried with AOFO disabled, GRT enabled, failed.  Tried with both disabled, worked fine.

To add insult to injury, if I remove the login account for BEX from the Exchange Address Book, no Exchange backup occurs.

Any ideas?

1 ACCEPTED SOLUTION

Accepted Solutions

BaffledBackup
Level 3

Something happened on the way to the regular backup last night.

 

I was notified the regular Full backup had failed around midnight, and the media was offline.  i checked the logs and BEX, and sure enough, the RDX was offline.  Windows showed the drive online and accessible (I saved/deteled a text file to test), and the Dell RDX tool showed the drive working fine.

 

I set the drive status to Online, and attempted to restart the backup.  Within 5 minutes, the drive went offline again.  I checked drive status again, stopped/restarted BEX services, restarted the backup, same problem.

I restarted the server, thinking something may've glitched, media showed online both in Windows & BEX, so I started the backup.  Same thing...within 5 minutes the drive went offline.

In all of these attempts, I could set the drive online again, but it would go offline shortly thereafter.

 

Frustrated and needing a backup, I deleted all jobs, then the media, restarted BEX services.

I re-created the Media pool (Backup-to-Disk-Cartridge), using defaults (as I always do), then created the server (no Exchnage) backup, and ran a test of the backup...it ran fine.  OK, so I ran the backup immediately and waited.

Next morning, backup worked fine, no issues.

To see if this had any effect on Exchange, I created an Exchange backup WITH GRT, and ran a full (during work hours) to another cartridge....ran smooth as silk, no errors.

Can I assume the solution was the cataloging/rights problem, which caused the Meda Pool to corrupt/damage/etc?  Backups to this same pool have been working (except this Exchange GRT issue) for almost 3 years.

One final question...how did the missing rights issue occur?  Was this a Microsoft OS modification?  I perform software installs using vendor defaults.

 

Thanks.

View solution in original post

16 REPLIES 16

CraigV
Moderator
Moderator
Partner    VIP    Accredited

AOFO shouldn't be used when backing up databases.

Have you tried to recreate the job? DId anything change on the media server in terms of patches etc? Or WIndows/Exchange patches?

THanks!

BaffledBackup
Level 3

AOFO shouldn't be used when backing up databases.

I'm aware of that from this week's research.  I've since turned it off.  However, as I stated originally, I've been backing-up with AOFO & GRT from day 1, 3 years ago without issue.

 

Have you tried to recreate the job?

Twice, no change.

 

DId anything change on the media server in terms of patches etc?  Or WIndows/Exchange patches?

No, nothing's changed, other than the latest Windows updates (WSUS).  I tend to wait at least two (2) months to apply Exchange patches...let everyone else work the kinks out.

 

 

VJware
Level 6
Employee Accredited Certified

Quite strange that non-GRT backups aren't successful either since this message usually relates to the "staging" part. What is the staging location set for backup under the backup setting - grt settings ?

Have you tried running a backup to an alternate location without GRT and AOFO ?

BaffledBackup
Level 3

Quite strange that non-GRT backups aren't successful either since this message usually relates to the "staging" part.

You missed my original post...GRT/AOFO disabled, backup worked.

What is the staging location set for backup under the backup setting - grt settings ?  Have you tried running a backup to an alternate location without GRT and AOFO ?

..and that's why I posted, as I would not have thought of looking at the actual BEX settings (outside the job).  I bet that's it...it's using C:\TEMP, but that folder hasn't been updated since the GRT restore.  Trying that right now and will see how it goes.

 

VJware
Level 6
Employee Accredited Certified

I was reading this line of yours "Tried running BEX 2012 full backup of Exchnage only with GRT disabled, same error. "  This is strange.

BaffledBackup
Level 3

Sorry, typical IT week.

Update on this issue...

Created a new temp folder on the second partition, gave the BEX account full rights (owner), ran a GRT...same error.

On the flip side, I removed Exchange from the regular backup, disabled GRT for Exchange, but left it for Sharepoint, and am receiving a similar error (same error codes, but the text under Errors states, "...the folder or directory is not accessible..".

I haven't been able to run a file/system backup due to space issues testing the Exchange backup.

I deleted the original job, and have created two separeate jobs: 1 for Exchange (only), 1 for System/FIles/Sharepoint/etc.

I'll post the results.

VJware
Level 6
Employee Accredited Certified

Is it possible to run GRT enabled backups to a non-RDX device ?

Also, for the existing backup, turn off "Delayed Cataloging" when running GRT backups to the RDX device.

BaffledBackup
Level 3

Sorry for the delay again, I was trying to get the full backups schedule properly, as they're not separate jobs, and had to play with the start times.

I have a 500Gb LaCie drive (USB3) on a USB2 port.  Created a new backup job, with GRT enabled, AOFO disabled...and it's running.  No failure at the onset.  I let it run for 10 minutes, then cancelled (during the work day, really slowed things down).  Worked flawlessly.

I can't find the 'Delayed Cataloging" option.  I found a Symantec TID that referenced such, but the option isn't listed.  Can I assume it's a BEX 2014/2015-only option?

At this point, I'm stumped.  Why would running a GRT with AOFO restore kill backup VSS functionality to an RDX drive?  A Windows update, possibly?  Windows Updates are the only changes that have occured.

VJware
Level 6
Employee Accredited Certified

"Delayed Cataloging" applies to BE 2012 as well.

It's not recommended to explicitly use AOFO with Exchange backups and secondly, the error message which you mentioned at the beginning is related to GRT staging and not VSS functionality.

BaffledBackup
Level 3

I see...I was getting information concerning how VSS was used for the snapshots, and made an illogical leap the issue lay with VSS/media.

So...where do I go from here?  I've two weeks of successful backups, no AOFO or GRT.  This leaves me without the ability to resotre indvidual items/folders.

The sad thkng is that I had a two year support contract when we upgraded to 2012, but never used it (expired last year).

 

VJware
Level 6
Employee Accredited Certified

Try the following:

1) To a RDX device, enable only GRT and disable Delayed Cataloging using the below steps:-

On the BE media server, browse to

HKLM\SOFTWARE\Symantec\Backup Exec For Windows\Backup Exec\Engine\Backup

Create a DWORD named Disable Delayed Cataloging and set the value to 1. Restart the BE services for the changes to take effect. Run GRT backups and observe the results.

2) If the GRT enabled backups still fail, then run a GRT enabled backup to a non RDX storage and check.

BaffledBackup
Level 3

Same result, but let me throw out some more details, one which I didn't mentioned.

Testing:  Added the registry entry as requested, restarted BEX services, created an Exchange only backup, ran it with GRT:on.  From here on AOFO is disabled permanently, so I won't mention it.  Same result.  Tried doing a full format (not a quick) of the catridge thinking that might be the problem.  No change.  Same driver/catridge works fine with non-GRT backups for the last two weeks.

Here may be a clue:  When the backup ran, here's what happened:

0-24 secs: Pre-Processing  --- Drive accessed 12 times

24-30 secs: Queued   ---- No drive activity

30secs -1:47: Snapshot Processing (this is the part I assumed was VSS executing)  --- Drive accessed 6 times

At 2:00 minutes, the cycle repeated, identically, then failed the job.  During this entire process, 0-bytes processed.

Again, bear in mind the non-GRT backups work fine to the same device.

So I ejected the catridge and followed the volume identifier in the registry, and deleted all references (mount points, etc), thinking perhaps it had become corrupted.  Reinserted the cartridge, mount points were re-created in registry, ran test backup. 

Same result.  Since a standard USB hard-drive works fine, BEX starts processing immediately, I though this might be a rights issue.  Default rights to the RDX is Everyone, full rights.  I removed the rights, then explicitly gave Admins, Domain Admins and the BEX user account full rights.  No change.

I went back and stepped through all options for Exchange in GRT, enabling/disabling one at a time (including using Passive/Active copy options) -- no change.

Now, the item I didn't mention.

When I performed the intiial GRT restore, I had to make a change to the BEX account.  Specifically, I had to make the account visible in the Exchange Address Book and send an e-mail from the account.  I'm very security conscious, and tend to not enable options unless I absolutely must.  After performing the restore, I went back and removed the account from the Exchange Address Book.  For the following two days (29th - 30th), I could not get a backup of Exchange in any format.  The fix was to add the BEX account to the Exchange Address Book.

Could this last item be a key?  Everything we do seems to point to some kind of rights issue. 

The BEX account has full, explicit rights to the GRT folder (on the second volume, instead of default C:\TEMP) and to the RDX drive.

I ran the SymHelp tool.  The only issues noted were:

  • Incorrectly identified 2012 SBS as 2014
  • BEX account was missing "Manage audit and security log" settings.  However, account is part of Administrators group which has the right.  That's strange.

Thanks again for your assistance.

 

VJware
Level 6
Employee Accredited Certified

Since a standard USB hard-drive works fine

- Do you mean a GRT- enabled backup to a non-RDX device completes successfully ?

Also, do ensure the BESA has the rights as per the following KB for GRT backups / restores of Exchange - https://support.symantec.com/en_US/article.TECH184449.html

BaffledBackup
Level 3

Yes, all non-RDX backups work fine (GRT/AOFO enabled/disabled...it doesn't matter).  It's strictly the RDX GRT backups that are failing.

What makes this really bizarre is that the BESA mailbox was not in the GAL, had never sent/recievd an e-mail, and had been working fine for 2.5 years.  The GRT restore is what triggered this whole mess.

I'll explicity grant the BESA the missing right and see what happens.

 

CraigV
Moderator
Moderator
Partner    VIP    Accredited

...explicit rights are there for a reason, and something along the way was bound to break.

Duplicating to disk will get you your Information Store if needed, and something like Kernel for Exchange will crack open the *.edb if you don't come right restoring data from the tape in question.

If it doesn, and after making the necessary change to the BESA, I'd recommend trying a restore again to test and see whether or not it is working.

Thanks!

BaffledBackup
Level 3

Something happened on the way to the regular backup last night.

 

I was notified the regular Full backup had failed around midnight, and the media was offline.  i checked the logs and BEX, and sure enough, the RDX was offline.  Windows showed the drive online and accessible (I saved/deteled a text file to test), and the Dell RDX tool showed the drive working fine.

 

I set the drive status to Online, and attempted to restart the backup.  Within 5 minutes, the drive went offline again.  I checked drive status again, stopped/restarted BEX services, restarted the backup, same problem.

I restarted the server, thinking something may've glitched, media showed online both in Windows & BEX, so I started the backup.  Same thing...within 5 minutes the drive went offline.

In all of these attempts, I could set the drive online again, but it would go offline shortly thereafter.

 

Frustrated and needing a backup, I deleted all jobs, then the media, restarted BEX services.

I re-created the Media pool (Backup-to-Disk-Cartridge), using defaults (as I always do), then created the server (no Exchnage) backup, and ran a test of the backup...it ran fine.  OK, so I ran the backup immediately and waited.

Next morning, backup worked fine, no issues.

To see if this had any effect on Exchange, I created an Exchange backup WITH GRT, and ran a full (during work hours) to another cartridge....ran smooth as silk, no errors.

Can I assume the solution was the cataloging/rights problem, which caused the Meda Pool to corrupt/damage/etc?  Backups to this same pool have been working (except this Exchange GRT issue) for almost 3 years.

One final question...how did the missing rights issue occur?  Was this a Microsoft OS modification?  I perform software installs using vendor defaults.

 

Thanks.