cancel
Showing results for 
Search instead for 
Did you mean: 

Blue Screen Caused by virtfile.sys

rewritten
Level 2
Partner Accredited

This post is FYI as this weekend after our LiveUpdate installed a couple of hitfixes onto our Backup Exec server it started to Blue Screen while executing our Backup-To-Disk Jobs for File System, Exchange, Active Directory and VMware, with the following Crash Dump Information.

 

----- Start Crash Analysis

BugCheck 3B, {c0000005, fffffade381a651c, fffffade30fa1a10, 0}

 

Unable to load image \SystemRoot\system32\DRIVERS\VirtFile.sys, Win32 error 0n2

*** WARNING: Unable to verify timestamp for VirtFile.sys

*** ERROR: Module load completed but symbols could not be loaded for VirtFile.sys

Probably caused by : VirtFile.sys ( VirtFile+b51c )

 

Followup: MachineOwner

---------

 

0: kd> !analyze -v

*******************************************************************************

*                                                                             *

*                        Bugcheck Analysis                                    *

*                                                                             *

*******************************************************************************

 

SYSTEM_SERVICE_EXCEPTION (3b)

An exception happened while executing a system service routine.

Arguments:

Arg1: 00000000c0000005, Exception code that caused the bugcheck

Arg2: fffffade381a651c, Address of the exception record for the exception that caused the bugcheck

Arg3: fffffade30fa1a10, Address of the context record for the exception that caused the bugcheck

Arg4: 0000000000000000, zero.

 

Debugging Details:

------------------

 

 

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The memory could not be "%s".

 

FAULTING_IP:

VirtFile+b51c

fffffade`381a651c ??              ???

 

CONTEXT:  fffffade30fa1a10 -- (.cxr 0xfffffade30fa1a10)

rax=000000006e66744e rbx=fffffade6d5e3c08 rcx=0000000008000000

rdx=fffffade377b3c20 rsi=fffffade6d5e3ac0 rdi=fffffade6dcb6ea8

rip=fffffade381a651c rsp=fffffade30fa2220 rbp=fffffade6dcb6c30

 r8=fffffa80001e5800  r9=fffffade30fa2200 r10=fffffa80001e0007

r11=fffffade6d070000 r12=000000001000000c r13=fffffade6d5e3b68

r14=0000000000000000 r15=fffffade6d657010

iopl=0         nv up ei pl nz na po nc

cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00010206

VirtFile+0xb51c:

fffffade`381a651c ??              ???

Resetting default scope

 

CUSTOMER_CRASH_COUNT:  2

 

DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP

 

BUGCHECK_STR:  0x3B

 

PROCESS_NAME:  REMNTD~1.EXE

 

CURRENT_IRQL:  0

 

LAST_CONTROL_TRANSFER:  from fffffa80001e5800 to fffffade381a651c

 

STACK_TEXT: 

fffffade`30fa2220 fffffa80`001e5800 : fffffade`30fa2270 fffffa80`001e5800 fffffade`30fa2200 fffffade`6dff7100 : VirtFile+0xb51c

fffffade`30fa2228 fffffade`30fa2270 : fffffa80`001e5800 fffffade`30fa2200 fffffade`6dff7100 00000000`00000000 : 0xfffffa80`001e5800

fffffade`30fa2230 fffffa80`001e5800 : fffffade`30fa2200 fffffade`6dff7100 00000000`00000000 fffffade`6ce7b010 : 0xfffffade`30fa2270

fffffade`30fa2238 fffffade`30fa2200 : fffffade`6dff7100 00000000`00000000 fffffade`6ce7b010 00000000`00000000 : 0xfffffa80`001e5800

fffffade`30fa2240 fffffade`6dff7100 : 00000000`00000000 fffffade`6ce7b010 00000000`00000000 fffffade`00000000 : 0xfffffade`30fa2200

fffffade`30fa2248 00000000`00000000 : fffffade`6ce7b010 00000000`00000000 fffffade`00000000 00000000`00000000 : 0xfffffade`6dff7100

fffffade`30fa2250 fffffade`6ce7b010 : 00000000`00000000 fffffade`00000000 00000000`00000000 fffffade`6d832350 : 0x0

fffffade`30fa2258 00000000`00000000 : fffffade`00000000 00000000`00000000 fffffade`6d832350 fffff800`c00000bb : 0xfffffade`6ce7b010

fffffade`30fa2260 fffffade`00000000 : 00000000`00000000 fffffade`6d832350 fffff800`c00000bb 00000000`00000000 : 0x0

fffffade`30fa2268 00000000`00000000 : fffffade`6d832350 fffff800`c00000bb 00000000`00000000 fffffa80`001e5800 : 0xfffffade`00000000

fffffade`30fa2270 fffffade`6d832350 : fffff800`c00000bb 00000000`00000000 fffffa80`001e5800 fffffade`6d832380 : 0x0

fffffade`30fa2278 fffff800`c00000bb : 00000000`00000000 fffffa80`001e5800 fffffade`6d832380 00000000`00000000 : 0xfffffade`6d832350

fffffade`30fa2280 00000000`00000000 : fffffa80`001e5800 fffffade`6d832380 00000000`00000000 00000000`00000000 : 0xfffff800`c00000bb

fffffade`30fa2288 fffffa80`001e5800 : fffffade`6d832380 00000000`00000000 00000000`00000000 fffffade`37783ee2 : 0x0

fffffade`30fa2290 fffffade`6d832380 : 00000000`00000000 00000000`00000000 fffffade`37783ee2 fffffade`6d5e3b68 : 0xfffffa80`001e5800

fffffade`30fa2298 00000000`00000000 : 00000000`00000000 fffffade`37783ee2 fffffade`6d5e3b68 fffffade`30fa2340 : 0xfffffade`6d832380

fffffade`30fa22a0 00000000`00000000 : fffffade`37783ee2 fffffade`6d5e3b68 fffffade`30fa2340 fffffade`30fa2308 : 0x0

fffffade`30fa22a8 fffffade`37783ee2 : fffffade`6d5e3b68 fffffade`30fa2340 fffffade`30fa2308 00000000`00000000 : 0x0

fffffade`30fa22b0 fffffade`37787160 : fffffade`30fa2408 fffffade`6d657010 fffffade`6d658720 00000000`00000000 : fltmgr!FltpPerformPreCallbacks+0x3e2

fffffade`30fa23a0 fffffade`377a3e07 : fffffade`6d658720 fffffade`6ce7b010 fffffade`6ce7b010 00000000`00000000 : fltmgr!FltpPassThroughInternal+0x40

fffffade`30fa23d0 fffff800`012827c9 : fffffade`6fbcba60 fffff800`01282170 fffffade`30fa2950 fffffade`6d74fcb0 : fltmgr!FltpCreate+0x3a7

fffffade`30fa2460 fffff800`01280164 : fffffade`6fbcba60 fffffade`6fbcba10 fffffade`30fa2790 00000000`00000001 : nt!IopParseDevice+0x1088

fffffade`30fa2610 fffff800`01284887 : 00000000`00000000 fffffade`30fa2780 00000000`00000040 00000000`00000000 : nt!ObpLookupObjectName+0x931

fffffade`30fa2720 fffff800`01291e4d : 00000000`00000000 00000000`00000000 00000000`00000b81 00000000`00000000 : nt!ObOpenObjectByName+0x180

fffffade`30fa2910 fffff800`01298ef7 : fffff700`00081080 fffff700`00081000 fffff680`0000e510 fffffade`41d11c40 : nt!IopCreateFile+0x630

fffffade`30fa2aa0 fffff800`01298d39 : fffff6fb`7da00000 fffffade`30fa2c70 fffffade`6d14d040 fffff800`01052c0f : nt!IoCreateFile+0x12f

fffffade`30fa2b80 fffff800`0102e33d : fffff6fb`7dbed000 00000000`00000000 fffffade`6d14d3b0 00000000`00000000 : nt!NtCreateFile+0x69

fffffade`30fa2c00 00000000`77ef0f3a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x3

00000000`0012e448 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x77ef0f3a

 

 

FOLLOWUP_IP:

VirtFile+b51c

fffffade`381a651c ??              ???

 

SYMBOL_STACK_INDEX:  0

 

SYMBOL_NAME:  VirtFile+b51c

 

FOLLOWUP_NAME:  MachineOwner

 

MODULE_NAME: VirtFile

 

IMAGE_NAME:  VirtFile.sys

 

DEBUG_FLR_IMAGE_TIMESTAMP:  48a30145

 

STACK_COMMAND:  .cxr 0xfffffade30fa1a10 ; kb

 

FAILURE_BUCKET_ID:  X64_0x3B_VirtFile+b51c

 

BUCKET_ID:  X64_0x3B_VirtFile+b51c

 

Followup: MachineOwner

---------

 

0: kd> lmvm VirtFile

start             end                 module name

fffffade`3819b000 fffffade`381ab000   VirtFile T (no symbols)          

  Loaded symbol image file: VirtFile.sys

   Image path: \SystemRoot\system32\DRIVERS\VirtFile.sys

    Image name: VirtFile.sys

    Timestamp:        Thu Aug 14 01:44:05 2008 (48A30145)

    CheckSum:         0001673B

    ImageSize:        00010000

Translations:     0000.04b0 0000.04e0 0409.04b0 0409.04e0

 

 

----End Crash Analysis

 

We Run Backup Exec 12.5 with SP1 on an IBM x3550 with 16GB RAM, with Windows Server 2003 SP2 x64 as the OS.

 

The version of the virfile.sys that was causing us an issue is "1.0.114.0" and was updated as part of one of the hotfixes, but unfortunately we have been unable to ascertain which hotfix at this stage.

 

The blue screens were prevented by rolling back to the version of virtfile.sys that is available in "C:\Program Files\Symantec\Backup Exec". The version of this file is "1.0.114.125".

 

Once this version of the virtfile.sys was replaced in safe mode all jobs proceeded successfully.

 

Hopefully this information will be of assistance to someone... :)

 

Cya..

19 REPLIES 19

Andrea_Rizzo
Level 3

Hi, i have same problem on bews 12.5 w/SP1 after installing hotfix 317412, 317920, 317966, 310648, 318828 On liveUpdate 15.02.2009...! I have blue screen while running Backup-to-Disk jobs for Exchange 2007 w/SP1 from passive node of CCR cluster on Windows 2008 x64 nodes. But i dont know wich one is it causing the problem and i have at this time no solution. I ran BEWS 12.5 on Dell PE2950 with 16GB RAM

 

I try to replace the virtfile.sys

Mighty_Mouse_-_
Level 2
Partner
Wish I saw this before I updated, but I too can confirm this update has blown up my server as well!!  I have only tested the backups on Exchange 2007 using Backup Exec 12 going to virtual disks.  I have disabled all my backups until a solution comes along since it is causing the BSOD.

Andrea_Rizzo
Level 3

After i put back virtfile.sys V 1.0.114.125 in C:\Windows\System32\drivers...the system doesnt crash anymore, but i have still big problem during Storage group backup with GRT. Jobs failed because file VirtApi.dll is missing. But i check and the file is present in 2 folders:

1. D:\Program Files\Symantec\Backup Exec\VirtApi.dll (111 KB, V1.0.114.125)

2. D:\Program Files\Symantec\Backup Exec\x86\VirtApi.dll (91.8 KB, V1.0.114.125)

 

I can also not browse the content of mailboxes in restore job wizrad. I got same error message (VirtApi.dll is missing...) If anyone as an idee because my backup are not running because i have put this 5 hotfix.....!? help... help...

Mighty_Mouse_-_
Level 2
Partner

I believe you have the numbers backwards; the issue is VirtFile.sys version 1.0.114.125Remove Hotfix 310648 (http://support.veritas.com/docs/310648 ) and the issue should be resolved. 

 

- More Information -

Hotfix 310648 replaces the virtfile.sys on the system and obviously breaks backup exec 12.5 and Server 2008 x64.  Follow the uninstall of the hotfix (http://support.veritas.com/docs/300796 ) and backup of Exchange works again! 

 

I am assuming all others will work; I have taken the jobs off hold.  On Server 2008 x64 you do NOT need to reboot after uninstalling Hotfix 310648, which would be better called BreakServer 310648....back to the drawing board and TESTING TESTING TESTING Symantec!!!!!

 

Thank you "rewritten" for the start of the solution!

Andrea_Rizzo
Level 3

Hi Mighty Mouse and thank you. But unfortunatly i try to uninstall hotfix 310648 but it doesn't work. I got:

"Error 1325.Live Update is not a valid short file name"

 

Drinkingbird
Level 3

Same issue here, uninstall above mentioned update resolved it (so far), thanks to those who posted their resolution.  Been testing this out on one server, this won't bode well for management's decision.  How an update with such a massive flaw (blue screening server 08 is no small feat) gets released is baffling.  Yikes.

 

System is server 2008 enterprise x64, BEWS 12.5, Exchange 2007 agent, backing up to disk.  Advanced disk based is installed but not in use that I know of, advanced open file is installed with default settings.  Both the system state and exchange backups blue screened, I was able to halt it before it got too far into the regular file backups so not sure if that one would have gone or not.

Tanel
Level 3

We too have ran in to this problem. Currently we're evaluating BEWS 12.5 and it's missfortunes like these that will make us really consider other solutions.

 

Edit: I guess I should mention we too are running 2008 x64, Exchange 2007 if that is of any help locating the problem.

Message Edited by Tanel on 02-17-2009 03:29 AM

Mighty_Mouse_-_
Level 2
Partner

Something must be wrong with your live update.  If you went into Program and showed updates and removed the hotfix all issues should go away.  I have not had to reboot my server since, I actually for the first time got a full backup without any stupid meaningless errors on shortcuts and no more BSOD.  Seems to be an issue with anyone running Server 2008 x64 and Exchange 2007, like all other posters I am running the same.

Andrea_Rizzo
Level 3

Thank Mihty Mouse, uninstall of this hotfix is OK now. Uninstall doesn't work if CD BEWS12.5 is not in the box. I have put the install CD and uninstal of this hotfix works fine.

And now i can run my backup jobs ok with GRT enable on Storage Group on Disk media. Thank you for all. But i'm little desapointed about Symantec Guys because i receive no help from Symantec people...

Mighty_Mouse_-_
Level 2
Partner

Andrea - I understand your frustrations with Symantec, We are a Symantec partner and get NO support on the products we run from Symantec (actually less than no support).  Since we get NFR versions to test out they do not offer support, so when they break our servers we are up the creek. 

 

I can tell you without a doubt do NOT install Mail Security on a 2008 Server x64 running Exchange 2007 if you like your server.  That was the worst mistake we ever tested, since the only way to recover your server is safe mode and removing the software, it breaks Exchange in a devastating way.  I would like to tell you that there is a solution, but we get no support so as a partner I tell people buy another product that may actually work.  I can also tell you do not install the management parts of Endpoint on a 2008 x64 server, installing it as a Stand alone works great.

 

When it comes to Backup Exec, depending on how many years you have been in the business you may remember when Veritas bought Backup Exec from Seagate or was it Arcadia, in either case there was at least a year of pain and suffering.  I know many posting here are looking or trying backup softwares, again I am a Symantec partner (mainly because of Backup Exec and Corporate versions of Antivirus). That being said I will tell you it is the lesser of all evils! Microsoft broke a wonderful NTBACKUP in Server 2008, you can't pay me enough to install CA's product lines and the choices go down hill from there.  While I am less than thrilled that Symantec failed to seriously test their hotfix before releasing it, I also point out to the newcomers in IT (us old guys will remember) there was a time that none of us would install any patch released by Microsoft for weeks or months because there was surely a hotfix or an "a" version of that coming out.  Novell had always been rock solid in their service packs until NetWare 5 when they started breaking things on release of their service packs/hotfixs. 

 

The overall point is there is no better backup product for the price than Backup Exec, Symantec owns it and surely screwed up in this hotfix release.  I have been running the product on my production server since Nov 2008 without incident until now and running Backup Exec for over 16 years.  It does a great job of Brick Level (GRT), I have tested the restores over and over again.  It is the best bang for the buck out there (again lesser of all evils).  As a consultant for over 16 years, having seen just about every backup solution over those years, Backup Exec still comes out as number 1 unless you are in a multi million dollar network and then EMC stuff is going to be better.  But again, having seen them all, EMC's stuff is no better when it comes to HUGE HUGE mistakes! 

 

This does not let Symantec off the hook for this screw up....means take caution folks....if they continue to release poor hotfixes....then it is time to start bailing on the product and going to the next best thing....not yet!  They need to start listening to us partners, asking us to help test the products before mass release, which of course would mean they need to give us SUPPORT!!!  I hope Symantec is listening!!

Tanel
Level 3
It would be nice to get some official recognition of this problem.

The_Zone
Level 2

Hi everyone,

 

Ran into this same issue on two client networks. Discovered that BE 12.5 has an express liveupdate mode that automatically installs patches... I suggest turning this off so it doesn't reinstall itself!

 

BE 12.5 has a Express Mode for live update that installs patches automatically. Be sure to set to "interactive mode" to prevent it from Re-downloading and installing. In 2008, its in the control panel only.

 

Cheers,

 

Rob

wits
Level 3
We are duplicating this issue

Will_Salen
Level 3
Same issue here: System is server 2003 R2 Standard x64, BEWS 12.5, Exchange 2007 agent, backing up to disk.  I would say this issue is critical...
Message Edited by Will Salen on 02-19-2009 07:25 AM

a7k
Level 2
I have the same problem here. System: Win 2008 x64 Enterprise, BEWS 12.5, Exchange 2007 agent, B2D. Uninstall of Hotfix 310648 resolved the problem for the moment.

Joshua_Kane
Level 6
Employee

Hello:

 

Symantec is aware of the current Blue Screen’s that have been occurring with Hotfix 310648 for Backup Exec 12.5 and Hotfix 313901 for Backup Exec 12.0.  These Hotfixes have been pulled from our website, and from our LiveUpdate server.

 

The issue has been investigated, and the root cause has been identified.  We have subsiquently released Hotfix 319699 for Backup Exec 12.5 and Hotfix 319698 for Backup Exec 12.0.  These Hotfixes can be found both on hour website and via the LiveUpdate server.  

 

Hotfix 319699 (12.5)

http://seer.entsupport.symantec.com/docs/319699.htm

 

Hotfix 319698 (12.0)

http://seer.entsupport.symantec.com/docs/319698.htm

bktheking
Not applicable

Hotfix 319699 is supposed to fix the 32 mailbox display limit. I applied this patch, ran a backup last night, select restore today and I still see only 32 mailboxes. Is this ever going to be fixed?

 

Thanks

jtatum
Level 4
Employee Certified

Hi King,

 

After applying the patch, you must push the remote agent to the remote servers you are backing up. This can be accomplished in the Backup Exec user interface by selecting Tools | Install Remote Agents. If the remote agent is not updated, you will continue to see the same symptom.

Admin_Dennis
Not applicable
Same problem with Backup Exec 12.5 Service Pack 3 !!!

The dump file indicates that the problem come from the virtfile.sys.

Do you have any idea to resolve this one ?

Thanks