ADBO errors without ADBO installed

[wayne_qcc]

A few weeks ago, I started getting errors on a few backup jobs such as:

Backup- SERVERNAME
- AOFO: Initialization failure on: "\\SERVERNAME\L:". Advanced Open File Option used: Microsoft Volu...

V-79-10000-11243 - VSS Snapshot error. No volumes were successfully imported to the media server for...
 

The backup fails.

AOFO is installed (licensed) but not active on this job.

ADBO was purchased (against my recommendation) but the license was not installed since it never worked on previous versions (12.0 and 12.5).

After these errors started appearing, I briefly installed the ADBO license so I could get to the option and make sure it wasn't somehow turned on "in the background" where I couldn't see it without the license. It was not turned on in any of the backup jobs. I removed the license again, and then disabled the "trial" option that was automatically selected after the license key was removed. The errors continue.

The specific server above is a SQL Server critical to operations. I am currently working around this by setting the SQL Server service to stop just before the backup starts (middle of the night), doing a file backup of the database (rather than a SQL-agent-based backup), and then automatically rebooting the SQL server when the backup finishes. Far from an optimum solution, but at least I now have a backup that's not a month old.

At about the same time, backups of our Exchange server (which does use AOFO) suddenly started producing errors:

 - AOFO: Initialization failure on: "\\EXCHANGE\Microsoft Information Store\MainStorageGroup". Advan...

V-79-10000-11231 - VSS Snapshot error. Activity on a volume is preventing all volumes from being sna...

There have been no patches to BE2010, Exchange, SQL Server, or the respective OSes that coincide with the start of these errors. The "one at a time" option has always been selected in AOFO options.

The Exchange cluster has now been patched with MS KB940349, and the active member of the cluster will be rebooted tonight to activate the patch.

Any ideas why BE2010 would be trying to use ADBO and aborting my backups when ADBO is not installed? And why "activity" on the Exchange server is suddenly preventing snapshots? Both SQL and Exchange had been successfully backed up for years with 11d, 12.0, and 12.5 and for several months with 2010 (installed after we replaced our Win 2003 domain controllers with Win 2008 DCs) before the current problems started.

[CraigV]

Hi there,

It can simply be a case of a corrupted installation, or registry key.

First thing I would do in this case would be to run a repair of your installation, and then post the results here...

Craig

[Colin_Weaver]

About one of the problems itself:

VSS is used by Information Store backups and because an error in VSS from the operating system looks like the same error that might be seen if you are using AOFO you get a misleading error message in the logs.

What this is actually saying is VSS has a problem snapping/accessing your information store and you need to troubleshoot this issue to fix your problem.

There is probably a similar reason relating to the L: drive on your SQL server and why off-host seesm to be mentioned. Does this drive hold your SQL Databases as the latest SQL Backup methods probably also use VSS.

[wayne_qcc]

CraigV -- Actually, I had tried the repair option last week, but I ran it again this morning and rebooted the media server afterwards. All steps reported success. I'll have to wait until tomorrow to see if the overnight backups go any better.

Colin Weaver -- The L: drive holds the logs for the databases on that server, both the "live" transaction logs and the (rotating) transaction log backups.

Our configuration does not support transportable snapshots for several reasons, not least of which is that it's not supported in the active version of the SAN operating system (the latest release, planned for installation this winter, does support it).

Whatever happened to cause the problems appears to have occurred on 10/6. All backups that morning (daily differentials run from midnight to about 7 a.m.) were successful. The next morning, the "offhost backup" nonsense started. There were no patches to any servers that day; I was on vacation and nobody else patches servers. We use WSUS for MS patches and all patches must be approved on WSUS and then released at the individual servers for install.

All servers exhibiting this behavior run some kind of database. One is proprietary; another SQL 2005 Express plus a separate proprietary; another SQL 2005 32-bit; the Exchange server; and a SQL 2005 64-bit. There are others, though, that run databases but are unaffected: two with SQL 2005 32-bit; another with a proprietary database; another with a proprietary, a Pervasive SQL, and Access databases; and one with SQL 2008 32-bit.

A possibly-related issue has turned up on our Hyper-V hosts. Backups of those (which run on a less-frequent schedule because of backup window issues) now show "corrupt data" errors and failures when backing up the volumes containing virtual servers. These are backed up with AOFO and, prior to 10/6, had backed up flawlessly.

Thanks for your input. I'll post again once tonight's backups complete.

[wayne_qcc]

Exchange backups now working after disabling AOFO.

SQL Server backups (file-based with SQL services stopped) still not working right; still trying to use AOFO and ADBO and failing because it couldn't mount the snapshot, despite both being disabled for the backup job. Have uninstalled the VSS writer on the SQL box and will see if that helps tonight.

[wayne_qcc]

The SQL Server backups now work as file-based with SQL Server services stopped and no VSS writer on the box.

After workarounds, one box (proprietary database built on Sybase Adaptive Server Anywhere) is still failing. Last night, I installed the ADBO license and activated ADBO in the backup of this server.

Now I get: ADBO: Offhost backup initialization failure on: ... VSS Snapshot error. Activity on a volume is preventing all volumes from being snapped. Try selecting the option 'Process logical volumes for backup, one at a time', and then run the job again." ("Process logical volumes for backup, one at a time" has been selected since Day One.)

At the time this volume is backed up, no-one has been using the database for more than 6 hours.

AOFO backups skip all the live database files and backs up only the backup files that are automatically generated when the server reboots.

For now, I'm going to reboot the server right before the backup starts so I've got a current automatic backup file to send to tape and completely ignore the actual, live, database. When I can find the time, I'll blow away the media server down to bare metal and start from scratch.