Job to duplicate B2D sets to removable drive scatters data all over the place and kills Agent

[xilog]

Hello.

I hope you good people will be able to help me, I'm at my wits end.

Apologies for a lengthy post, the devil is probably in the detail with this problem

I'm attempting to create a backup server to enable a two-stage backup, first stage being a nightly backup from servers to the local drive array and then duplicating that backup to a removable drive.  Then at the end of the week the removable media is swapped and the local backup deleted, leaving the removable device as the "live" backup and the local array clean for another week's backups.  Full backup runs on Friday night with diffs every Sat-Thu night.

I have created what I think are suitable devices, media sets and jobs and indeed everything works fine until the duplicate jobs start.

There are three devices, a B2D folder called "Servers" for daily backups, a B2D folder called "backupsvr" which contains a backup of the backup server immediately after Backup Exec installation and a removable B2D device called "Disk_1" (set up as drive F:) which is the destination for duplication.  All these use the default settings of 1GB files and 100 backup sets.

There are just two media sets, "Daily Backups" and "Daily Duplicates".  Daily Backups has an OPP of 2 weeks and indefinite append, Daily Duplicates has an OPP of just 1 day since its disks will be ejected at the end of the week and not reinserted for at least 5 weeks and also has infinite append.

There are two main backup jobs, Friday Full and Daily Diff, both targeted at the Servers device and putting its media into the Daily Backups pool.  Each of those has an associated Duplicate task targeted at Disk_1, the idea being that after the main backup is complete, it's then duplicated on the removable device.

To my way of thinking, thios should all work, yes?

Well what actually happens is this:

The "Full Backup" runs properly, backing up to the Servers B2D device and putting the media in the Daily Backups pool.
The "Duplicate full backup" task starts, initially putting files onto Disk_1 as expected.  When it's put 20GB of data onto the disk (it's a 1TB drive) it goes very, very slowly and then kills the BE agent, finally it gets round to verifying this 20GB and when it starts writing again it starts putting the B2D files in the Servers device and the backupsvr device (seemingly at randon) and putting the media in any pool it seems to feel like; it's put some in Daily Backups, some in Daily Duplicates, some in Imported media.  Some even ends up in Scratch Media.

What the blazes can be going on?  It's driving me nuts trying to figure out what's gone wrong.

Any advice you can offer?

Thanks,

Kevin.

[xilog]

Deatheye, I've tried a couple of tests and it's definitely GRT that's causing the problem. If it's on, the replicate fails and sprays data all over the shop.  If it's off the replicate works.  Reproducible 100% here.

What seems to happen is that the duplicate job reaches a point in the duplication of the information store, borks and then seems to get confused over which device pool it's using, changing from the defined device pool for the job to the "All Devices (SERVER)" pool.

GRT is deactivated by unchecking "Enable the restore of individual mail messages and folders from information store backups" on the Exchange options tab of the backup job that the duplicate job is linked to.

Good luck, and if you get a solution that works without disabling GRT, please let us know :)

[Deatheye]

hmm. that job which has this problems doesn't even backup exchange.. well I try an ddeactivate every GRT Option I find. There are other places where you can activate GRT, like Sharepoint. For you only the Exchange GRT Option caused this problem?

I just let a copy job run inside a testenviroment. The data was copied to places where it souldn't, now I'll try and see what happens with GRT deactivated.

EDIT: deactivated every GRT Option I found, and copy job looks fine. Did you trace anything with sgmon?
Does this only happen if you actually backup something that has a GRT setting?
It's still kinda wierd cause the copy jobs we have this problem with (actuall all media servers we got have this problem), don't even backup an exchange store. At least not all of them. And one server has an backup job that only makes a backup onf an information store, duplicate job there worked fine when I tested it once. But on the same media server an other duplication job has this missbevhaviour and doesn't backup the information store.
So I kinda wonder if this only happens when GRT is activated, you backup objects which support GRT and objects that don't, and you duplicate the backup.
Some jobs we had even worked fine with GRT activated. But I think they don't backup any data where GRT is used.

So right now it looks like there is no problem with GRT activated, and only GRT data, GRT activated and only non GRT Data, but if a job contains GRT data and non GRT data and you duplicate that it happens.

I'd like to analyse this further and make some more test jobs to be sure the last job not only did work out by accident.
Is there anything else besides Exchange, Sharepoint and active Directory that got a GRT Option?

I just noticed that this would mean about 36 or more different backup jobs with a copy job to define which GRT Optiojn excatly causes this, and if the assumption is really correct. O_o
I think I just make again 2 jobs one with GRT one without to be sure that the behavior really isn't by accident and is repeatable.
This really did cost a lot of time allready (severell days... didn't rellay count them)... and without the hint from xilog we propably still would wander blind together with the support...

[xilog]

No, I didn't use sgmon yet, that's usually a last resort :)

I'd like to hear from a few others using GRT and B2D duplication to see if they suffer the same issues.

I've not yet tried an information store only backup so that there's only a GRT-using element to the backup and given that I can restore to a restore group in exchange 2003 anyway, I think I'm just going to settle for not using GRT.  It's a nice feature but far from critical for us.

You're right, there are a large number of combinations to get this right, for me there's only Exchange to worry about but if you're using Exchange, sharepoint, AD and SQL then that's a hell of a lot of potential failure points in an enterprise's strategy.

[Deatheye]

We once had an Information Store only backup with duplication job to B2D. Which worked fine but since other had the error, we also did stop that one, in case it fucks up too.
We got Exchange, Sharepoint, SQL and AD ... xD

Well right now it's going to the internal symantec bug team. Waiting for a responce from there.

[Deatheye]

Nice I'm right now looking into an other error:
http://seer.entsupport.symantec.com/docs/328487.htm
Why do you think we installed that update...? Yes, excatly cause of the duplicate job. I really need vacation...

EDIT: And just found this:
http://seer.entsupport.symantec.com/docs/305109.htm

specially nice is the note at the bottom... not inteded to be fixed...
Seriously... we got envirements where we solved this creating a second backup job that does backup to an USB-Device.
That works for small envirements but allready makes it more complex then needed.
Our internal backup takes over 30h to finish. There is just no way to add another backup job cause of time issues, not even talking about the complexity added in our enviroment.

[xilog]

Jeez, I wish I'd found that article before.  And they're not going to fis what is a quite serious bug?  Thanks Symantec, thanks a bunch.  It's not as if your software is cheap, y'know?  If it was shareware, then hey, no problem but having spent over £4k on software for backups, I kinda expect it to work, and if there's a serious bug, that it be fixed.

Harumph.  That's put me in a sour mood for the weekend.

[Deatheye]

Just got the confirmation that this aplies to 12.5 as well. They update the articel since it's only listening 12.0 as affected.
So the only solution at the moment is to seperate GRT-Date from other data...
I'l try to find out what excatly this means. Can I just put all the GRT-data into one job? What about system state where active directory data is stored? I can't select AD-data alone...