BESR 7.01 / 7.00 / 6.52 fails to run and crashes server

[Tripz]

We were running Livestate System Recovery 6.02 and took advantage of the upgrade to 7.00. Since then we have not had success running 7.0, 7.01 or 6.52 - we have not attempted 6.02 due to our window of downtime, but intend to go back to 6.02 shortly.

I have run the Symantec Utilities (batch files) for uninstalling 7.01 / 7.00 and the 6.x series and then performed reinstalls but to no avail.

Symptoms are:

1.) Manual backup begins, system responsiveness becomes incredibly slow - almost a hang, then as the livestate is actually created the system responsiveness returns.

2.) Manual backup beings, system responsiveness becomes incredibly slow and does not return. The behaviour of the hang is very rare, there is no ability to open new program, or access start menu, but certain items off the desktop open etc - and a nightly backup ran without a hitch, yet all other services and 4 VM's which ran on the server were unresponsive.

----------------------------------

The server is a Win2k3 R2 SP2 box with all the latest MS Updates, running on an HP ML350 dual 3.2Ghz Xeon with 4 Gigs of memory and 15rpm disks - Mirrored OS array and Raid 5 array.

It runs VMWARE 1.03 and 4 VM's within it.

System recovery is installed on the mirrored OS and creates a livestate image of the Mirrored OS / Raid 5 Array and dumps to a 500GB IDE drive.

The Raid 5 Array runs 4 Virtual Machines.

System changes which have occurred about the time Livestate failed to run:

1.) OS taken to SP2

2.) Latest HP 7.90 Firmware and 7.90 Driver set

3.) VMWARE upgraded to 1.03

4.) Symantec Livestate 7.00 loaded

If the VMWare are not running, System Recovery appears to run ok, but as soon I boot the VMWARE's up and run System Recovery with the VMWare's running the job fails and the server becomes unresponsive.

I have identical hardware, OS and VMWare's running on two other machines and do not experience the same issue - although on one, after upgrading from 6.02 to 7.00 the initial System Recovery failed, but none others since. I thought it may have benefited from uninstalling and on reinstall saying "no" to importing old configs and histories / jobs etc, but alas no.

We have successfully backed up the same setup using 6.02 and 3.0 without any issue.

I have a technical support issue open with Symantec, but am hoping someone might have some goodies to try in the interim.

Cheers for reading this!

Message Edited by Tripz on 09-17-2007 03:02 AM

[Bill_Felt]

Hello,

It appears you have narrowed down the problem to being some kind of conflict when you have your VM's running.  Logically I would think that the disk I/O is just being overloaded with four VM's running and having BESR also try to use its fair share of disk I/O.

However, if you were able to have the four VM's up with LSR 6.02 and not see any issues, then something must be happening that needs further investigation.

Any updates from Symantec support?

Thanks.

[Tripz]

Hi there,

Disk I/O is very low across the host and all VM's before the job begins and as the job runs. This time I monitored the job as it begun and ran perfmon on each VM.

I just went back to 6.02 and managed to get a successful backup going - however the initial delay is extremely troublesome.

The server in semi unresponsive for up to 15 minutes with very HIGH disk i/o.

But the high disk i/o only begins after the job starts - when the job actually gets to the point that it writes the V2I file to a separate drive it speeds up dramatically.

Whatever Livestate / System Recovery is doing initially is sometimes hanging, sometimes not - but in all instances painfull slow. If it gets to the point of actually writing the file the job is successful.

I am going back to Symantec with concerns that 6.02 still presents issues.

I have a similar setup, same hardware, server, OS, VM's which are more intensive and run without issue.

This should work - it did for 12 months and the physical host beside it is running without issue with the same setup.

[sgladiadis]

Hi,

Just to let you know that we are seeing this exact problem on BESR Server 7.0.

I ran the liveupdate over the weekend which didnt help.

Shutting down the VM's allows for the imaging to take place without any hitchs. This is of course far from ideal.

Hopefully someone at Symantec is listening...

I managed to get in to the Task Manager and saw that there was little I/O expect from VPRO... something and LSASS... Managed to try to attempt a shutdown from ALT-CTL-DEL  which FAILED. I had to HARD RESET to get out.

We have ABANDONED running the BESR job until this is fixed.

S.

[Tripz]

Hi there, yeah, very frustrating.

BESR does cause high I/O (Obviously), but up until recently it never caused an issue with the VMWARE Host.

I have had success with running a command file and using the VMRUN Suspend command to suspend those VM's which are non-essential.

It means the BESR job completes quicker and the less VM's running the better it is (well so far). I leave one essential VM running while BESR runs to keep it up.

I then VMRUN Start them when the BESR job has completed.

To minimise downtime, I do a Monday early morning FULL Recovery point, then do incrementals each evening.

The benefit of the incrementals is that downtime during the weekdays is kept to 10-15 minutes, the base takes 2 hours.

I run the BESR job as std compression and the resultant file is 89 gig, with 3.5 gig inc each night.

This is NOT ideal and NOT a solution.

I've been working with Symantec trying everything they have suggested, but it is not as easy as trying one thing here and there - there are small windows of downtime and HIGH risk when testing on servers that are LIVE.

I hope that Symantec can resolve the issues.