Backup Exec 12.5 tapes not filling - end marker unreadable

[MattSmith]

Hello all,

I've been having a problem with backups for a while, here's a perfect example: a backup ran that totaled 2.1tb took EIGHT 800/1600 LTO4 tapes instead of the expected 3-4 even with zero compression.

All the jobs are set to "append, but overwrite if nothing is appendable"

The message under the Appendable Until column in Devices is "Not Appendable (End marker unreadable)".  I've researched this error a bit and followed a suggestions such as update all firmware/drivers (done for smart array controller, nics, tape autoloaders and tape drives) and update the backup exec software using liveupdate (done).  

I've read that it could indicate bad tapes or drives, but this is happening with both autoloaders (HP 1/8 G2), one is a few months old and one is 2-3yrs old and with brand new tapes (< 1tb written) as well as older tapes.  Both drives have been cleaned multiple times.

Any other advice/tips/tricks!?  Running out of tapes fast...

One other piece of info: these are mostly vmware backups

thanks,
Matt

[Kiran_Bandi]

Are you facing this problem with media used for a specific backup job?

Check out the following TN....

http://www.symantec.com/docs/TECH37091

[MattSmith]

Thanks for the link Kiran.  I have gone over everything in the how to prevent section except for:

"5. Make sure the media server is not rebooting and/or the Backup Exec Services are not stopping during a backup"

I did not mention (i thought they were unrelated because the jobs do complete fully despite the error) that I am often seeing a network error as follows: Final error: 0xe00084f8 - The network connection to the Backup Exec Remote Agent has been lost. Check for network errors.

Could this be causing the end marker unreadable per item #5 above? 

In attempt to resolve it I updated all remote agents - the backups are running again tonight so we'll see if that fixes it tomorrow!

[CraigV]

Hi Matt,

Is your OPP/append periods correct on that media set?

[MattSmith]

Hi Craig,

yeah, 1 week/3 days is how the Media Set is configured.  I run tapes on a 5 week rotation and quick erase/inventory each tape before use so they shouldn't have been overwrite protected at all.

[pkh]

When your network connection is lost, your backup job failed and it caused the end marker to be unreadable because BE never has a chance to close the tape.  You should solve your network connection lost problem first, check that the remote agent service on all your servers did not abend for whatever reason by looking at the event log.

You don't have to do anything about the unreadable tape marker.  When you next overwrite the tape and the backup completes successfully with any errors whatsoever, then you will not have the problem.

[MattSmith]

Yes pkh this does make sense!  However how to resolve the network connection lost error...

You mentioned checking logs, could you be more specific as to which logs?  I have checked system/application event logs on both the server and on the servers being backed up when the "network connection lost" error occurs and I have not been able to discover anything at all.

This last weekend as an example:

Server A according to the backup exec Job History experienced "error e000fe30 - a communications error has occurred".  The job started at 11:35pm and finished at 12:56am.

In the Job Log tab is as follows:

VMware vcbMounter job started to export virtual machine 'xxxx'. Friday, November 12, 2010 at 11:35:00 PM
VMware vcbMounter job to export virtual machine 'xxxx' ended successfully. Saturday, November 13, 2010 at 12:16:14 AM

The connection to target system has been lost. Backup set canceled

 The logs on the backup server really don't show me much aside from "VDS Basic Provider 1.0" errors, which according to this http://support.microsoft.com/kb/912063 shouldn't be an issue because we have Server 2003SP2 installed.  There's also http://support.microsoft.com/kb/932446 which also indicates "not an issue".

So based on the times above how can I determine at exactly what time the error occurred so as to uncover any corresponding entries in the logs?

Hope this made sense and thank you for the help,

Matt

[pkh]

I would check the system log of the remote server to see the reason why the BE remote agent service was interrupted.  You can go to this service and set it to restart automatically after the first failure.  Sometimes this will be sufficient.  However, if the BE remote agent service keeps failing, then you would have to use Windebug to analyse the minidump that is produced by the failure to determine the cause and fix it.

[malladine]

pkh - I tried setting the services to auto restart, but that didn't help.  I have since tried the following in addition to everything above:

http://www.symantec.com/business/support/index?page=content&id=TECH56772

I followed both solutions linked from there - the tape reconfiguration was necessary as I had two of the four set, but removing these for the problem jobs did not help.  The priority has also been changed to below normal and then to lowest on two seperate days, but this did not help.

I opened a case with Symantec tech support.  After 3 hrs with two different technicians, most of it on hold, the only thing that was attempted by a level 2 technician was to have me install Symantec tape drivers (http://www.symantec.com/business/support/index?page=content&id=TECH128716), but this resulted in windows being unable to initialize the drivers and he reverted to the HP drivers (latest) I was previously using and then asked if he could put me on hold again for a higher lvl technician.  I said I couldn't afford the time to be on hold for that but please call the following morning.  Very disappointed that after 3 days I have received no call from Symantec.

But anyway, there's my update, still burning through tapes.  Anyone else seeing this in a VM environment and actually resolve it??

EDIT: I should add that I am entirely unfamiliar with obtaining and analyzing the minidumps as you suggested and have not researched how to do that yet.

[MattSmith · Accepted solution]

Just an update to say this has been resolved.  As it turned out the two problem jobs (each backing up several VMs) were conflicting somehow.  I changed the schedules to avoid most of the overlap and they are both now completing fully without network connection failure and therefore without the end marker unreachable errors on the tapes.  Which means i'm using 2-3 a day instead of 6-7!

Hope this helps someone else

Matt