Advanced Open File troubles

[Andrew_Lomakin_]

Hello!

My backup server is set up with other servers using dedicated links, using IP configuration as follows:

Ethernet adapter Server1--FS link:

Connection-specific DNS Suffix . :
IP Address. . . . . . . . . . . . : 172.16.1.2
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . :

Ethernet adapter Server2--FS link:

Connection-specific DNS Suffix . :
IP Address. . . . . . . . . . . . : 172.16.2.2
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . :


where FS is the backup server.

The server1 and server2 have 172.16.2.1 and 172.16.1.1 addresses with /24 mask. Servers are connected using cross-over (UTP-5) cables. The link is 100% stable and available at all times. No other application utilizes the link.

Server 1 and server2 run Windows 2000 Server
FS runs windows 2003 Service Pack 1 server with Backup Exec 10 SP1 + all hotfixes.
When i was setting up remote agents with Advanced open file option, i specified that dedicated links' ip addresses should be used.



After i set everything up and tried to run a backup, i got the following at the log:

Job ended: 12 july 2005 at 23:39:00
Completed status: Completed with exceptions



Click an error below to locate it in the job log
Backup- 172.16.2.1AOFO: Initialization failure on: "\\172.16.2.1\D:". Advanced Open File Option used: VERITAS Volume Snapshot Provider (VSP).
Snapshot provider error (0xE000FE30): A communications failure has occurred.
Check the Windows Event Viewer for details.
AOFO: Initialization failure on: "\\172.16.2.1\Microsoft Information Store\Exchange Storage". Advanced Open File Option used: VERITAS Volume Snapshot Provider (VSP).
Snapshot provider error (0xE000FE30): A communications failure has occurred.
Check the Windows Event Viewer for details.

Event viewer states following

Event Type:Warning
Event Source:Backup Exec
Event Category:None
Event ID:57755
Date:12.07.2005
Time:0:05:19
User:N/A
Computer:FS
Description:
Backup Exec Alert: Job Completed with Exceptions
(Server: "FS") (Job: "Server1-FULL&&Server2-FULL&&Exchange-FULL-Media Rotation Policy-Daily Backup") The job completed successfully. However, the following conditions were encountered:

A snapshot operation required by this job was unsuccessful. Check the job log and the Windows Event Viewer for additional information.


For more information, click the following link:
http://eventlookup.veritas.com/eventlookup/EventLookup.jhtml






What is the possible problem? It seems to me that Advanced open file is working correctly on one of the servers, but is not on another. I tried reinstalling remote agent. No luck :\ What else can be done?

Thank you,

Andrew V Lomakin

[Andrew_Hunter]

Hi Andrew,

Feel free to browse my own thread on the subject:

http://forums.veritas.com/discussions/thread.jspa?threadID=50520&tstart=0

No answer yet from Veritas.

[Andrew_Lomakin_]

Here's a little followup fot the process that's going on. Andrew (Hunter), do you have similar log messages in your job log?


AOFO: Started for resource: "\\172.16.1.1\D:". Advanced Open File Option used: VERITAS Volume Snapshot Provider (VSP).
AOFO: VERITAS Snapshot Provider (VSP) snapshot cache file settings for volume "D:" : Initial: 0 MB. Maximum: 2169 MB. File: "\\?\Volume{519f8e8f-63e6-11d8-ba7f-806d6172696f}\Backup Exec AOFO Store\_BEVspCacheFile_21.VSP".
AOFO: VERITAS Snapshot Provider (VSP) snapshot settings used: Quiet Time: 5 seconds. Time Out: 2000 seconds.
Note: Resource: "\\172.16.1.1\System?State" is not snappable. Attempting to back up directly from the source volume.
Network control connection is established between 172.16.1.2:4219 <--> 172.16.1.1:10000
Network data connection is established between 172.16.1.2:4224 <--> 172.16.1.1:2702

[Shyam_Sundar]

Hello,

Refer to the following document to perform single volume snap at a time.

http://support.veritas.com/docs/264464
http://support.veritas.com/docs/269591

Backup Exec - Tools - Options - AOFO - under 'automatically chosen backup technology', use Veritas Snapshot Provider

Also check whether the default drive where the AOFO Cache file is stored (C drive) has sufficient space. Else reconfigure so that the Cache file is stored in an alternate drive which has ample space. This can be done in the AOFO wizard - Cache File Settings.

Also check if the port 10000 is open on both the servers. Run telnet sessions on both the servers to check.

Additional Information :
For information on the recent VERITAS Backup Exec

security vulnerabilities, including links to the downloads

for the necessary hotfixes, please refer to the following

document:
Patch summary for Security Advisories VX05-001,

VX05-002, VX05-003, VX05-005, VX05-006, VX05-007

http://seer.support.veritas.com/docs/277429.htm

Note : If we do not receive your reply within two
business days, this post would be marked ‘assumed
answered’ and would be moved to ‘answered questions’
pool.

Thanks.

[Andrew_Lomakin_]

Hello Shyam,

Thank you for a very informative reply. I have set up the options you recommended and verified i have connectivity on port 10000. What gave me idea for thoughts was cache file location. When running advanced open file wizard, should i specify the cache file location on the remote server (server which runs backup exec remote agent with open file) or on the server that runs backup exec? How much space does it typically take for a cache file?



Thank you,

Andrew

[Renuka_-]

Hello,

Please refer to the foll for the cache file size:
http://support.veritas.com/docs/233099
The cache file is by default created on the server being snapped.


Additional Information :
For information on the recent VERITAS Backup Exec security vulnerabilities, including links to the downloads for the necessary hotfixes, please refer to the following document:
Patch summary for Security Advisories VX05-001, VX05-002, VX05-003, VX05-005, VX05-006, VX05-007

http://seer.support.veritas.com/docs/277429.htm

NOTE : If we do not receive your reply within two business days, this post would be marked assumed answeredand would be moved toanswered questions pool.

[Andrew_Hunter]

The link refers to an article on backup exec for netware, are you sure it applies to windows renuk a?

[Andrew_Lomakin_]

Hi Andrew, good to know you're following my thread.

Dear vertitas support. I have checked all issues you told me to check, including the cache file and single instance volume backups. The problem still persists.

The error message is as follows:

Backup- 172.16.2.1AOFO: Initialization failure on: "\\172.16.2.1\D:". Advanced Open File Option used: VERITAS Volume Snapshot Provider (VSP).
Snapshot provider error (0xE000FE30): A communications failure has occurred.
Check the Windows Event Viewer for details.
AOFO: Initialization failure on: "\\172.16.2.1\Microsoft Information Store\xxx Storage". Advanced Open File Option used: VERITAS Volume Snapshot Provider (VSP).
Snapshot provider error (0xE000FE30): A communications failure has occurred.
Check the Windows Event Viewer for details.




The error occurs on server1 which has a couple gigs free on hard drive. i COULD have believed that this is because there is not enough cache file space on drives, if not the following issue: Server2 has VERY few space on all HDD's (comparatively to server1), and all backups on it are performing very well. What additional issues could you suggest, or is there any way to turn extensive logging in backup exec to analyze the problem i am experiencing more thoroughly.

Thank you,

Andrew V Lomakin

[Andrew_Hunter]

Hi Andrew,

We did some major work this weekend on our servers including rebuilding server 1 and upgrading to windows 2003 server. We pushed out all the remote agents and AOFO AGAIN to server 1 and server 2. After rebooting server 2 it now backs up succesfully. Go figure, why this didn't work the first time we tried it I don't know. We are doing a full backup Tuesday night so I will update you on Wednesday if the backup went through fine on server 1. So far I am guessing that something isn't upgrading correctly when you run the upgrade path from 9.1 to 10.

[Andrew_Lomakin_]

Hi Andrew,

My servers are running Win2k and i have no plans to upgrade them to 2k3 :\ My Veritas Backup Exec configuration was not deployed as an upgrade. What i did (a couple of times already) was removing the old version and installing the new one....then removing it...removing agents...installing again....and again...hotfixes...rebootsrebootsreboots...and my Server1 takes 35 minutes to reboot...i had enough of long nights. Unfortunately server rebuild won't be possible in my scenario, since that server hosts Exchange, AD (PDC master, schema master, etc etc etc) Symantec Antivirus Сorporate server and many many more (yeah, we're low on servers)....i guess that server reinstall will cost me all my weekend and i don't think i want that :)

Would be nice to know if your backup goes fine after a rebuild. Do you use dedicated inter-server links to perform backup jobs?

Does ANYONE know how to turn advanced logging in Advanced open file agent?


Thanks

[Andrew_Hunter]

hehehe

Yep same scenario as you. Took the whole weekend, but this was planned ages ago and had nothing to do with the backup problem. Installed new harddrives and rebuilt the whole server from scratch. Performs great now we have cleared all the junk off.

The funny thing is that server 2 which we did not rebuild, just reinstalled AOFO and remote agent for the umpteenth time, is all of a sudden behaving itself. No more errors. Can't figure it out.

We are not currently running dedicated interserver links for the backups. Just our normal network and running the backups at night.

[Renuka_-]

Hello Andrew Lomakin,

Even in case of backup exec for windows servers the cache file is created on the server being snapped.

1. please make sure that the cache file is clearing after each backup.
2. There is no antivirus scanning the cache file location.
3. that there is enough space to store the cache file on the destination server.
- In tools>wizards>AOFO wizard you may change the default location of the cache file.
4. If you have not specified a location for the cache file, Backup Exec will attempt to locate the cache file on volumes other than the source volumes. These volumes must meet the following requirements. If any of these criteria are not met, the cache file is created on the source volume.
5. For your question on cache file size:
If the cache file is located on the volume that is being snapped,
Backup Exec uses the following calculations:
For the initial size, Backup Exec calculates 10 percent of the used space and 50 percent of the free space, and then uses the smaller number. For the maximum size, Backup Exec calculates 30 percent of the used space and 90 percent of the free space, and then uses the smaller number.

If the cache file is located on a different volume than the volume that is being snapped, Backup Exec uses the following calculations:
The initial size is zero. For the maximum size, Backup Exec
calculates 30 percent of the free space on the target volume and
uses that figure.
The maximum size cannot be smaller than the initial size. If the
maximum size is calculated to be smaller than the initial size,
Backup Exec sets the maximum size to match the initial size. Both the initial size and the maximum size must be at least 50 MB.
----------------------------------------------------------------------------

Additional Information :
For information on the recent VERITAS Backup Exec security vulnerabilities, including links to the downloads for the necessary hotfixes, please refer to the following document:
Patch summary for Security Advisories VX05-001, VX05-002, VX05-003, VX05-005, VX05-006, VX05-007

http://seer.support.veritas.com/docs/277429.htm

NOTE : If we do not receive your reply within two business days, this post would be marked assumed answeredand would be moved toanswered questions pool.

[Andrew_Lomakin_]

Hello Renuk!

Considering steps you advised me to take
1. please make sure that the cache file is clearing after each backup.
1:: moved cache file to non default location and verified the directory specified for cache is clean and tidy

2. There is no antivirus scanning the cache file location.
2:: Enabled extensions in symantec antivirus to skip the specified directory.
3. that there is enough space to store the cache file on the destination server.
3:: Plenty (30gig)


Still failures. Reinstallation of remote agent does not help.

Any other advices?

Thank you

P.S. is there a checkbox or something to enable ADVANCED logging in advanced open file remote agent so that the problem can be pinned down and solved

Andrew Lomakin

[tejashree_Bhate]

Hello,

Are you also Backing up Exchnage using AOFO? In that case case create 2 Separate Backup jobs for the Exchange Data (without AOFO) and another job with AOFO.
This has been mentioned in the thread posted by Mr Hunter,however please verify once again.

Note : If we do not receive your reply within two business days, this post would be marked ‘assumed answered’ and would be moved to ‘answered questions’ pool.


Thanks.

[Andrew_Lomakin_]

Hello tejashree,

I admit i haven't been reading Andrew Hunter's thread too carefully (shame on me actually), but i will try the suggested solution. In fact this smells badly of a `workaround` not of a `solution`, so i should ask if Vertias is going to do something about this in the future, since now i got to recreate all my backup policy (and that's what i don't feel too enthusiastic about).


Will try later today and report the results.

[Deepali_Badave]

Hello,

Have you cretaed the two separate backup jobs?Is the problem persists?
Please update us on this issue.

NOTE : If we do not receive your reply within two business days, this post would be marked assumed answered and would be moved to answered questions pool.

[Andrew_Lomakin_]

Hello Deepali,

Thank you for your inquiry about my current status of Backup Exec. (Actually first time i see that support really cares) :)


I have created two separate jobs for file backup and exchange backup (took me a great deal of policy re-planning though). Everything is working fine now. For over a two weeks there have been no failures in backup jobs, but still i consider this as a `workaround`, not a `fix`, and will be happy to see the problem solved by some patch. For now it's all cool with me, problem solved, thanks to everyone form Veritas Support, and specially to Andrew Hunter.