Jobs being Canceled ??

[colin_marr]

I have Backup Exec 9.1 on a windows server.

I have two jobs being canceled.

There is no reason given for why the jobs are being canceled.

One based on the size of the backup byte count "seems" to be backing everything up. They are just files. When I open the Job log it gives and error but opens listing the files etc that were backed up. eg.

======================================================================
Job server: BACKUP
Job name: Public Images [14A] Monthly
Job started: May 1, 2010 at 9:00:01 PM
Job type: Backup
Job Log: BEX18016.xml
======================================================================

Drive and media information from media mount:
Drive Name: AppsE_Public_M-T_[12A] + Images Monthly [14A]
Media Label: B2D005289
Media GUID: {5885C0D8-720B-469A-8495-27AE80B28F9A}
Overwrite Protected Until: 02/05/2010 1:14:00 AM
Appendable Until: 31/12/9999 12:00:00 AM
Targeted Media Set Name: Monthly Public Folder Backup
======================================================================
Media operation - append.
Software compression enabled.
======================================================================
NTSERVEROFO: Started for device: "\\NTSERVER\D:". Advanced Open File Option used: Veritas Volume Snapshot Provider (VSP).
Network control connection is established between 10.1.0.3:2227 <--> 10.1.0.19:10000
Network data connection is established between  10.1.0.3:2230 <--> 10.1.0.19:1895
\\NTSERVER\D: Data
Family Name: ">BACKUP >Public Images [14A] Monthly >12/4/2006 >10:38:38 AM"
Backup of "\\NTSERVER\D: Data"
Backup set #41 on storage media #136
Backup set description: "Public Images [14A] Monthly"
Backup Type: FULL - Back Up Files - Reset Archive Bit
Backup started on 01/05/2010 at 9:00:56 PM.
Directory \
Directory \Public
Directory \Public\IMAGES
READ THIS ON WHERE TO PUT PICTURES.txt
                                          2082  02/12/2009     2:43 PM
Shortcut to Opening.wmv.lnk
                                           730  18/12/2007     2:57 PM
The opening.wmv                63405786  18/12/2007     1:17 PM
Thumbs.db                 <HS>           13312  13/07/2009     2:08 PM
Directory \Public\IMAGES\100th Anniversary
IMGP3109.JPG                            239754  09/10/2008     4:53 PM
IMGP3110.JPG                            240243  09/10/2008     4:53 PM
IMGP3111.JPG                            236953  09/10/2008     4:53 PM

... etc



The second is an Exchange Info Store Backup.

It "Says" in the Job Summary Information that 0 files directories etc where backed up.  But then in the Job log, which displays an error when I open it , eventually says
======================================================================
Job server: BACKUP
Job name: Exchange [7A] Backup -Full Storage Group
Job started: May 1, 2010 at 10:37:12 PM
Job type: Backup
Job Log: BEX18019.xml
======================================================================

Drive and media information from media mount:
Drive Name: Weekly Bank Backup Drive
Media Label: B2D003911
Media GUID: {2C15A677-22FF-44CB-B6DA-1FB7CC6D7E56}
Overwrite Protected Until: 23/05/2010 2:41:55 AM
Appendable Until: 09/05/2010 2:41:55 AM
Targeted Media Set Name: 3_Weeks_Daily
======================================================================
Media operation - overwrite.
Software compression enabled.
======================================================================
EXCHANGENetwork control connection is established between 10.1.0.3:2806 <--> 10.1.0.2:10000
Network data connection is established between  10.1.0.3:2807 <--> 10.1.0.2:58300
\\EXCHANGE\Microsoft Information Store\First Storage Group
Family Name: ">BACKUP >Exchange [7A] Backup -Full Storage Group >01/05/2010 >10:37:12 PM"
Backup of "\\EXCHANGE\Microsoft Information Store\First Storage Group "
Backup set #1 on storage media #1
Backup set description: "Exchange Backup "First Storage Group" - Full"
Backup Type: FULL - Database & Logs (flush committed logs)
Microsoft Exchange Server Agent: Started
Backup started on 01/05/2010 at 10:41:56 PM.
Database Mailbox Store (EXCHANGE)
    C:\Program Files\Exchsrvr\mdbdata\priv1.edb

Drive and media information from media mount:
Drive Name: Weekly Bank Backup Drive
Media Label: B2D003846
Media GUID: {DAD7E253-CA44-463A-A2A4-4B4D60C90EAF}
Overwrite Protected Until: 23/05/2010 2:45:51 AM
Appendable Until: 09/05/2010 2:45:51 AM
Targeted Media Set Name: 3_Weeks_Daily
Backup set #1 on storage media #2

Drive and media information from media mount:
Drive Name: Weekly Bank Backup Drive
Media Label: B2D003847
Media GUID: {67FE990B-0892-42FA-9875-7792568ECD71}
Overwrite Protected Until: 23/05/2010 2:49:15 AM
Appendable Until: 09/05/2010 2:49:15 AM
Targeted Media Set Name: 3_Weeks_Daily
Backup set #1 on storage media #3

.... etc

The log almost seems to show that the exchange was backed up, but not sure.

Anyway how can I deduce why these jobs are being canceled. I DO NOT have auto Cancel enabled.

thanks.

[Dev_T]

Hello,

Open the job properties and uncheck AOFO option. Its not recomended to backup any database using snapshot technology.

[Symc_Techie]

Check whether the Backup Exec job engine service is terminating. It may have terminated because of some faulty dll or exe. Check Application log for any errors related to Backup Exec. Also, check whether you are latest on the Service packs/hotfixes for Backup Exec 9.1

[colin_marr]

Thank you for your suggestion,

However the first job that fails are just images, eg. pictures. There is no db. So I think I want the open file option on? Correct me if Im wrong. These files are just ordinary files eg .jpgs mostly.

The second backup that fails is an exchange information store and the Open file option is indeed off.

Regards

[colin_marr]

This is Possible but if the backup exec Job engine did terminate. Wouldnt that mean that all subsequent jobs would have the same error, or at least fail? However other jobs did not fail? eg here is the list of jobs. At least one was successfull 1 1/2 hours later.
---------------------------------
Starting with the latest

2:00 AM   Failed

1:00 AM    Successful

11:50 PM Failed

10:37 PM Canceled

9:00 PM   Sucessful

9:00 PM   Canceled

9:00 PM  Successfull
--------------------------------

Please advise.

[Symc_Techie]

It could be possible that it terminated but the service is set to restart as a recovery action and it started and completed the subsequent jobs successfully. We will know about this from the event viewer.

Check if there are any Exchange related errors/warnings in the Event log on the Exchange Server. Also, if possible, try rebooting the exchange server during your off-peak hours and run the backup again. Maybe that will help to eliminate Exchange problems.

[colin_marr]

Your comment on checking the event log tweaked me to look at the event log of the backup server. This is good advice, to anyone who is looking for errors that Veritas may not be reporting.
It seems a little more telling . Despite the fact that the Veritas software shows very little errors the event log is more telling. I'm not sure how much to tell or who might be interested but here are some things I discovered.
1. I looked at the logs produced by both my jobs then compared them with the files that it says it created. I think despite the cancelled job, both actually completed. The files seem to exist. It would be nice to have the bu software tell me this but anyway I think the backup was don.

2. I looked at the event log of the backup server (2003). They show things like this see below.
(I have reduced unimportant lines.) 

The be engine seems to have died I think and it did restart. I checked all the service settings for the bu software and indeed they are set to restart.

While some of the failers seems to indicate that the drives were off line, Im not sure if they were or if it was the be engine that was not working and thus i got those errors.

One important error that is interesting is the backup 11A which seems to have failed completely is not even listed in the failed jobs of the BU software. A little scary that it is not reporting that that job didnt even run.

Any thoughts?

Ps I cant really restart the exchange server. But I dont think that is the problem.

Time:  11:50:21 PM
User:  N/A
Computer: BACKUP
Description:
Faulting application bengine.exe, version 9.1.4691.54, faulting module bengine.exe, version 9.1.4691.54, fault address 0x000a8707.

Time:  11:51:23 PM
User:  N/A
Computer: BACKUP
Description:
Catalog server (C:\Program Files\VERITAS\Backup Exec\NT\bengine.exe) started.
   Catalog directory is C:\Program Files\VERITAS\Backup Exec\NT\Catalogs\.
   Remote catalog server is BACKUP.

Time:  11:58:06 PM
User:  N/A
Computer: BACKUP
Description:
Backup Exec Alert: Device Error
(Server: "BACKUP") (Job: "Public Images [14A] Monthly") The drive hardware is offline!

Please confirm that the drive hardware is powered on and properly cabled.

Time:  11:58:08 PM
User:  N/A
Computer: BACKUP
Description:
Backup Exec Alert: Job Cancellation
(Server: "BACKUP") (Job: "Public Images [14A] Monthly") The job was canceled by user SMI\Administrator.

Time:  11:59:16 PM
User:  N/A
Computer: BACKUP
Description:
Backup Exec Alert: Job Cancellation
(Server: "BACKUP") (Job: "Exchange [7A] Backup -Full Storage Group") The job was canceled by user SMI\Administrator.

Date:  01/05/2010
Time:  11:59:16 PM
User:  N/A
Computer: BACKUP
Description:
Backup Exec Alert: Device Error
(Server: "BACKUP") (Job: "Exchange [7A] Backup -Full Storage Group") The drive hardware is offline!

Time:  12:00:23 AM
User:  N/A
Computer: BACKUP
Description:
Backup Exec Alert: Job Failed
(Server: "BACKUP") (Job: "Exchange [11A] Mailbox Interim Backup M-T") Exchange [11A] Mailbox Interim Backup M-T -- The job failed with the following error: The Backup Exec job engine system service is not responding.

Time:  12:00:24 AM
User:  N/A
Computer: BACKUP
Description:
Backup Exec Alert: Job Failed
(Server: "BACKUP") (Job: "---V------------ Saturday 11:59pm ------------V---") ---V------------ Saturday 11:59pm ------------V--- -- The job failed with the following error: The Backup Exec job engine system service is not responding.
 
 

[Symc_Techie]

From the events, the hardware (Backup to disk folder in this case) seems to throwing the errors. You can try the following:

1) Run a disk defragment operation and the Windows Error Checking Utility (CHKDSK /F) on the disk where the Backup to disk is residing.
2) Confirm disk compression is not enabled on the hard disk/s.
3) Check if there are any Disk Controller related errors in the System Log.
4) If possible, copy the data to some other disk, format the disk in question and recreate the Backup to Disk folder on that. Then try the backup.
5) Try creating a new B2D on a separate disk and check the result.

About the error logging/reporting, the job log is ideally just an indicator of the problem. It would not give you a detailed report. Also, in your case, because of the job engine service terminating, it could not completely write the log. However, the events in the Application log are posted by Backup Exec components itself and they are posted as and when they occur and do not wait for the job to complete. Thus, you may find the events but no error in the job log. This is how you can effectively use the job log, Alerts tab and Event logs for troubleshooting an issue with Backup Exec. Next step would be debugging which should be ideally handled by a Symantec Support engineer.

Let me know if the above steps do not help or if you have any queries.

[colin_marr]

I have been working through the steps, but each seems to take time. The defrag alone on a 1 TB drive takes a looong time.

I will get back with results. As well the jobs only run once a week. And I am loath to run them additional time cause they I use up precious space on the bu drives.
thanks