Backups on busy system

[Ken_Wallewein]

I've been having trouble getting good backups on a Windows 2003 server. It's starting to look like the boot drive is just too busy. I keep getting errors saying:

AOFO: Initialization failure on: "C:". Advanced Open File Option used: Symantec Volume Snapshot Provider (VSP).
AOFO: Unable to get minimum quiet time for physical volumes. Reduce the activity on these volumes, or wait and run the backup when there is less activity on the volumes.

The system does do a fair amount (but it's a small site, totally idle at night): it's a domain controller, file/print server, Exchange server, SQL server, WSUS server and virtual machine server. I have Symantec Enterprise installed, and it runs SMS for Exchange. The server has two 3.4GHz (single core) Xeons and 6GB of RAM.

I've tried using Sysinternals' Filemon and Regmon to track this down. And I have to agree, it seems fairly rare that the system will go more than 5 seconds without _some_ disk write activity on C:. It's either Exchange, SQL, some log or event log, registry updates, or whatever. SMS for Exchange seems to cause a lot of registry activity. I've tried a command to stop/start WSUS. But does it make sense to shut down Exchange and SQL as well? Besides, if I understand properly, the snapshot provide tells those to pause during the snapshot. There aren't really any third party apps I can shut down.

At this point, I have my backups split 4 ways:
1. C:, shadow copy and info store
2. The other data drives
3. SQL
4. Exchange
Everything works but the first.

One thing I wonder: should I manually, explicitly exclude the SQL and Exchange data areas? I hate doing that, because every manual exclusion is one more place to make a mistake and accidently exclude something important. I've done that, it cost me a lot, and I don't care to repeat it. And BE doesn't exactly provide much for backup content auditing.

Oh, I do have AOFO set for automatic selection and single-volume snapshots, and the quescent timer is already reduced to 4 seconds.

What am I doing wrong?

[Swati_Joshi]

Hi ken,

To backup SQL and Exchange databases, you must uncheck the AOFO option

and use the agents.

You can reduce the quiet time to 2 sec and retry the job.

Have you designed four different backup jobs for these four different elements?

Please do revert back on this.

Thanks,

Note : If we do not receive your reply within two business days, this post would be marked "assumed answered" and would be moved to "answered questions" pool.

[Ken_Wallewein]

Sorry, I don't think I understand your answer.

To backup SQL and Exchange databases, you must uncheck the AOFO option
and use the agents.
That's what I'm already doing with the separate SQL and Exchange backups, and they're working fine. It's the C: drive backup that was giving me problems.

That job just covers all the files on C: (SQL and Exchange do live there), and selects all C: drive files, shadow copy and info store, using AOFO automatic mode. It does NOT select for SQL or Exchange backups. So, again: should I manually, explicitly exclude the SQL and Exchange data areas from the C: drive file backup? I've already explained why I'd prefer not to.

You can reduce the quiet time to 2 sec and retry the job.
Your documentation explicitly warns of the risks of doing that.


Have you designed four different backup jobs for these four different elements?
Yes, I guess I wasn't clear about that. Four different selection sets (five, actually -- SQL daily is different from weekly), and 8 backup job definitions: daily incremental and weekly full for each set.

Please do revert back on this.

The C: backup, using separate jobs, is what I'm having problems with. I went to the separate sets because the combined set was giving me so much trouble. Should I go back to the combined set?

[Ken_Wallewein]

Sorry, I don't think I understand your answer.

>>To backup SQL and Exchange databases, you must uncheck the AOFO option
>>and use the agents.

That's what I'm already doing with the separate SQL and Exchange backups, and they're working fine. It's the C: drive backup that was giving me problems.

That job just covers all the files on C: (SQL and Exchange do live there), and selects all C: drive files, shadow copy and info store, using AOFO automatic mode. It does NOT select for SQL or Exchange backups. So, again: should I manually, explicitly exclude the SQL and Exchange data areas from the C: drive file backup? I've already explained why I'd prefer not to.

>>You can reduce the quiet time to 2 sec and retry the job.

Your documentation explicitly warns of the risks of doing that.


>> Have you designed four different backup jobs for these four different elements?

Yes, I guess I wasn't clear about that. Four different selection sets (five, actually -- SQL daily is different from weekly), and 8 backup job definitions: daily incremental and weekly full for each set.

>>Please do revert back on this.

The C: backup, using separate jobs, is what I'm having problems with. I went to the separate sets because the combined set was giving me so much trouble. Should I go back to the combined set?

[aurora]

Ken,

Although it's not ideal as a long term solution, it may be worth excluding the data directories to see what the outcome is. Once we know this works (hopefully) we at least know where the issue resides and can build up from that, don't you agree?

[Ken_Wallewein]

Sorry, I don't think I understand your answer.

To backup SQL and Exchange databases, you must uncheck the AOFO option
and use the agents.
That's what I'm already doing with the separate SQL and Exchange backups, and they're working fine. It's the C: drive backup that was giving me problems.

That job just covers all the files on C: (SQL and Exchange do live there), and selects all C: drive files, shadow copy and info store, using AOFO automatic mode. It does NOT select for SQL or Exchange backups. So, again: should I manually, explicitly exclude the SQL and Exchange data areas from the C: drive file backup? I've already explained why I'd prefer not to.

You can reduce the quiet time to 2 sec and retry the job.
Your documentation explicitly warns of the risks of doing that.


Have you designed four different backup jobs for these four different elements?
Yes, I guess I wasn't clear about that. Four different selection sets (five, actually -- SQL daily is different from weekly), and 8 backup job definitions: daily incremental and weekly full for each set.

Please do revert back on this.

The C: backup, using separate jobs, is what I'm having problems with. I went to the separate sets because the combined set was giving me so much trouble. Should I go back to the combined set?

[aurora]

Ken,

Although it's not ideal as a long term solution, it may be worth excluding the data directories to see what the outcome is. Once we know this works (hopefully) we at least know where the issue resides and can build up from that, don't you agree?

[Ken_Wallewein]

Sorry, I don't think I understand your answer.

To backup SQL and Exchange databases, you must uncheck the AOFO option
and use the agents.
That's what I'm already doing with the separate SQL and Exchange backups, and they're working fine. It's the C: drive backup that was giving me problems.

That job just covers all the files on C: (SQL and Exchange do live there), and selects all C: drive files, shadow copy and info store, using AOFO automatic mode. It does NOT select for SQL or Exchange backups. So, again: should I manually, explicitly exclude the SQL and Exchange data areas from the C: drive file backup? I've already explained why I'd prefer not to.

You can reduce the quiet time to 2 sec and retry the job.
Your documentation explicitly warns of the risks of doing that.


Have you designed four different backup jobs for these four different elements?
Yes, I guess I wasn't clear about that. Four different selection sets (five, actually -- SQL daily is different from weekly), and 8 backup job definitions: daily incremental and weekly full for each set.

Please do revert back on this.

The C: backup, using separate jobs, is what I'm having problems with. I went to the separate sets because the combined set was giving me so much trouble. Should I go back to the combined set?

[aurora]

Ken,

Although it's not ideal as a long term solution, it may be worth excluding the data directories to see what the outcome is. Once we know this works (hopefully) we at least know where the issue resides and can build up from that, don't you agree?

[aurora]

Ken,

Although it's not ideal as a long term solution, it may be worth excluding the data directories to see what the outcome is. Once we know this works (hopefully) we at least know where the issue resides and can build up from that, don't you agree?

[aurora]

Ken,

Although it's not ideal as a long term solution, it may be worth excluding the data directories to see what the outcome is. Once we know this works (hopefully) we at least know where the issue resides and can build up from that, don't you agree?

[aurora]

Ken,

Although it's not ideal as a long term solution, it may be worth excluding the data directories to see what the outcome is. Once we know this works (hopefully) we at least know where the issue resides and can build up from that, don't you agree?

[Ken_Wallewein]

Wow! FWIW, I got massive java errors trying to reply to your reply. Possibly related to my originally trying to use ">>" to indent your response.

[ray_littlefie1]

You need to exclude the data directories of exchange and Sql- that should provide the quiet time required for OFO.

You have all of your eggs in one basket with this server- you didn't say specifically, but I infer that in addition to this server acting as domain controller, exchanger server, SQL server, file & print server, it is also the backup server of itself- correct? This makes restoration a challange if the server becomes unbootable. Are you using IDR? This becomes a key question as to whether all the OFO files you're trying to capture on the C: drive are worth it. Would you restore them- or would you reload the applications?
If it where me, I would look more to a product like Symantec Live State Recovery to protect your C: drive and application data- and just use Backup Exec for the Exchange & SQL agents- But this depends on how you partitioned things.

[Russ_Perry]

AOFO, specifically VSP, may not ever work on this server. When VSP runs, it snaps at the volume level so even if you exclude data directories for Exchange and SQL, VSP is still trying to cache all changes written to the volume by each of those apps. You have to look at all the activity on the volume when considering whether VSP would be up to the task of caching changes.

Russ.

[aurora]

I had java errors too, now it's posted it 4 times.

Try excluding the directories and let us know how you got on.

[Ken_Wallewein]

> AOFO, specifically VSP, may not ever work on this
> server. When VSP runs, it snaps at the volume level
> so even if you exclude data directories for Exchange
> and SQL...

I wondered about that.

As it happens, it looks like I may have a solution.

I noticed that I had a syntax error in my pre/post command: I was using "net stop/start update services", and forgot the quotation marks around "update services".

Once I fixed that, I seem to be getting successfull backups. I can't imagine why WSUS has to hit the hard drive so often. And I'm still nervous about my chances, given all the other disk activity I see. I am planning to use IDR once this is all stable.

Even though BE job definitions have several options based on whether the pre-backup command succeeds, there is nothing in the log that indicates whether it did. So, unfair warning to anyone else that uses pre/post commands: do your own logging, and record success/failure. Just one more thing that bugs me about BE. The logs don't indicate how long it had to wait to get a snapshot, either. In fact, the whole usage of the term "log" is ifffy: logs are supposed to be chronological, and BE's aren't.

I was interested in the recommendation to use Live State. I'm a bit leery of such products, but haven't yet actually used one. How does it compare to Windows Data Protection Manager, which apparently does something similar? Does it do snapshots like VSS or AOFO? How does it compare to BE, for that matter? How is its logging? Its backup media management?

[priya_khire]

Hello Ken,

It seems that modifying the pre-post command seems to have fixed the issue for you.

Regarding your queries on Live State, we suggest you to contac the appropriate support since this support only deals with queries on Backup exec.

Regards.