paul_fahy
20 years agoLevel 2
EVFSF.SYS :- intermittent server hangs
We have been experiencing intermittent server hangs on two files servers, and have just received the following from Microsoft advising that this is due to the evfsf.sys, we are running EV6 sp1 on w2k3 sp1 see below for MS advise. I was wondering if anyone else had experienced this issue. We are logging a call with Symantec and are waiting on them.
Thanks
There are 3 threads that have been waiting for about 20 minutes. All of these threads are eventually waiting on the MiMappedPageWriter thread to release a certain NTFS ERESOURCE. Typically, the MiMappedPageWriter would not hold on to such a resource. I checked the current outstanding IRPs for the MiMappedPageWriter thread, and all of them (29!) are pending in \FileSystem\evfsf on an IRP_MJ_WRITE. This is a filter driver which MPS Reports lists as written by "KVS Inc". It looks like to me like an IRP fired by the MiMappedPageWriter thread obtained the NTFS ERESOURCE in question, and then because the IRP got stuck in evfsf, the resource was never released. The customer needs to engage the vendor of evfsf.sys and find out why all these IRPs are pending in their driver.
- Matthew
Debug notes
// locks summary
No. Type Lock Addr Owner Thread Owner Wait Time Owner Waiting On Owner Function
1 8bd3ec2c 89c91020 20m:53.343s No. 3 Ntfs!NtfsAcquirePagingResourceExclusive+0x20 (srv!WorkerThread)
2 86bff018 8afaa020 19m:42.609s No. 1 Ntfs!NtfsAcquireSharedVcb+0x23 (srv!WorkerThread)
3 8acb5428 8cf47288 20m:49.015s ??? nt!MiMappedPageWriter+0x4d
4 8ac96290 881aadb0 20m:26.062s No. 1 Ntfs!NtfsAcquireSharedVcb+0x23 (srv!WorkerThread)
// All 3 threads are eventually waiting on this thread.
// And this thread owns resource 8acb5428, which is a PFCB->PagingIoResource
0: kd> !thread 8cf47288
THREAD 8cf47288 Cid 0004.0064 Teb: 00000000 Win32Thread: 00000000 WAIT: (WrVirtualMemory) KernelMode Non-Alertable
808aafc0 NotificationEvent
Not impersonating
DeviceMap d6600908
Owning Process 8cf63440 Image: System
Wait Start TickCount 21441911 Ticks: 79937 (0:00:20:49.015)
Context Switch Count 227850
UserTime 00:00:00.0000
KernelTime 00:00:16.0828
Start Address nt!MiMappedPageWriter (0x80847a04)
Stack Init f793f000 Current f793ece8 Base f793f000 Limit f793c000 Call 0
Priority 17 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr Args to Child
f793ed00 80832f7a 8cf47300 8cf47288 8cf47330 nt!KiSwapContext+0x26 (FPO: )
f793ed2c 8082927a 00000002 808aafd0 00000000 nt!KiSwapThread+0x284 (FPO: ) (CONV: fastcall)
f793ed74 80847a51 808aafc0 00000012 00000000 nt!KeWaitForSingleObject+0x346 (FPO: ) (CONV: stdcall)
f793edac 80948bb2 00000000 00000000 00000000 nt!MiMappedPageWriter+0x4d (FPO: ) (CONV: stdcall)
f793eddc 8088d4d2 80847a04 00000000 00000000 nt!PspSystemThreadStartup+0x2e (FPO: ) (CONV: stdcall)
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16
// Outstanding IRPs for thread 8cf47288
0: kd> !irpfind 0 0 thread 8cf47288
Looking for IRPs with thread == 8cf47288
Scanning large pool allocation table for Tag: Irp? (86153000 : 861d3000)
Searching NonPaged pool (85001000 : 8d000000) for Tag: Irp?
Irp irpStack: (Mj,Mn) DevObj MDL Process
878cf9d8 irpStack: ( 4, 0) 8bd88e90 0x00000000
87a27620 irpStack: ( 4, 0) 8bd88e90 0x00000000
87b96860 irpStack: ( 4, 0) 8bd88e90 0x00000000
87cf0af0 irpStack: ( 4, 0) 8bd88e90 0x00000000
891c1008 irpStack: ( 4, 0) 8bd88e90 0x00000000
89326a40 irpStack: ( 4, 0) 8bd88e90 0x00000000
894ff3c0 irpStack: ( 4, 0) 8bd88e90 0x00000000
8950a008 irpStack: ( 4, 0) 8bd88e90 0x00000000
89974d50 irpStack: ( 4, 0) 8bd88e90 0x00000000
89a0ad50 irpStack: ( 4, 0) 8bd88e90 0x00000000
89a19d50 irpStack: ( 4, 0) 8bd88e90 0x00000000
89a865e8 irpStack: ( 4, 0) 8bd88e90 0x00000000
89ce1820 irpStack: ( 4, 0) 8bd88e90 0x00000000
89d2a7f0 irpStack: ( 4, 0) 8bd88e90 0x00000000
89ecea60 irpStack: ( 4, 0) 8bd88e90 0x00000000
8a17e268 irpStack: ( 4, 0) 8bd88e90 0x00000000
8a29ec20 irpStack: ( 4, 0) 8bd88e90 0x00000000
8a42e4d0 irpStack: ( 4, 0) 8bd88e90 0x00000000
8a442ab0 irpStack: ( 4, 0) 8bd88e90 0x00000000
8a44a6b8 irpStack: ( 4, 0) 8bd88e90 0x00000000
8a630548 irpStack: ( 4, 0) 8bd88e90 0x00000000
8a6705d0 irpStack: ( 4, 0) 8bd88e90 0x00000000
8a730008 irpStack: ( 4, 0) 8bd88e90 0x00000000
8a866108 irpStack: ( 4, 0) 8bd88e90 0x00000000
8a998008 irpStack: ( 4, 0) 8bd88e90 0x00000000
8aa512d0 irpStack: ( 4, 0) 8bd88e90 0x00000000
8aac8490 irpStack: ( 4, 0) 8bd88e90 0x00000000
8acd4450 irpStack: ( 4, 0) 8bd88e90 0x00000000
8afa12a8 irpStack: ( 4, 0) 8bd88e90 0x00000000
Module
Company Name: KVS Inc.
File Description: Enterprise Vault File System Filter
Product Version: (1.0:0.0)
File Version: (2.0:0.0)
File Size (bytes): 68480
File Date: Mon Jul 04 00:15:36 2005
Module TimeDateStamp = 0x42934e79 - Tue May 24 16:55:37 2005
Module Checksum = 0x000161f6
Module SizeOfImage = 0x00010b80
Module Pointer to PDB =
Module PDB Guid = {3D76A8B9-6A9F-42B7-8A4A-B563157617E8}
Module PDB Age = 0x1
Thanks
There are 3 threads that have been waiting for about 20 minutes. All of these threads are eventually waiting on the MiMappedPageWriter thread to release a certain NTFS ERESOURCE. Typically, the MiMappedPageWriter would not hold on to such a resource. I checked the current outstanding IRPs for the MiMappedPageWriter thread, and all of them (29!) are pending in \FileSystem\evfsf on an IRP_MJ_WRITE. This is a filter driver which MPS Reports lists as written by "KVS Inc". It looks like to me like an IRP fired by the MiMappedPageWriter thread obtained the NTFS ERESOURCE in question, and then because the IRP got stuck in evfsf, the resource was never released. The customer needs to engage the vendor of evfsf.sys and find out why all these IRPs are pending in their driver.
- Matthew
Debug notes
// locks summary
No. Type Lock Addr Owner Thread Owner Wait Time Owner Waiting On Owner Function
1 8bd3ec2c 89c91020 20m:53.343s No. 3 Ntfs!NtfsAcquirePagingResourceExclusive+0x20 (srv!WorkerThread)
2 86bff018 8afaa020 19m:42.609s No. 1 Ntfs!NtfsAcquireSharedVcb+0x23 (srv!WorkerThread)
3 8acb5428 8cf47288 20m:49.015s ??? nt!MiMappedPageWriter+0x4d
4 8ac96290 881aadb0 20m:26.062s No. 1 Ntfs!NtfsAcquireSharedVcb+0x23 (srv!WorkerThread)
// All 3 threads are eventually waiting on this thread.
// And this thread owns resource 8acb5428, which is a PFCB->PagingIoResource
0: kd> !thread 8cf47288
THREAD 8cf47288 Cid 0004.0064 Teb: 00000000 Win32Thread: 00000000 WAIT: (WrVirtualMemory) KernelMode Non-Alertable
808aafc0 NotificationEvent
Not impersonating
DeviceMap d6600908
Owning Process 8cf63440 Image: System
Wait Start TickCount 21441911 Ticks: 79937 (0:00:20:49.015)
Context Switch Count 227850
UserTime 00:00:00.0000
KernelTime 00:00:16.0828
Start Address nt!MiMappedPageWriter (0x80847a04)
Stack Init f793f000 Current f793ece8 Base f793f000 Limit f793c000 Call 0
Priority 17 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr Args to Child
f793ed00 80832f7a 8cf47300 8cf47288 8cf47330 nt!KiSwapContext+0x26 (FPO: )
f793ed2c 8082927a 00000002 808aafd0 00000000 nt!KiSwapThread+0x284 (FPO: ) (CONV: fastcall)
f793ed74 80847a51 808aafc0 00000012 00000000 nt!KeWaitForSingleObject+0x346 (FPO: ) (CONV: stdcall)
f793edac 80948bb2 00000000 00000000 00000000 nt!MiMappedPageWriter+0x4d (FPO: ) (CONV: stdcall)
f793eddc 8088d4d2 80847a04 00000000 00000000 nt!PspSystemThreadStartup+0x2e (FPO: ) (CONV: stdcall)
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16
// Outstanding IRPs for thread 8cf47288
0: kd> !irpfind 0 0 thread 8cf47288
Looking for IRPs with thread == 8cf47288
Scanning large pool allocation table for Tag: Irp? (86153000 : 861d3000)
Searching NonPaged pool (85001000 : 8d000000) for Tag: Irp?
Irp irpStack: (Mj,Mn) DevObj MDL Process
878cf9d8 irpStack: ( 4, 0) 8bd88e90 0x00000000
87a27620 irpStack: ( 4, 0) 8bd88e90 0x00000000
87b96860 irpStack: ( 4, 0) 8bd88e90 0x00000000
87cf0af0 irpStack: ( 4, 0) 8bd88e90 0x00000000
891c1008 irpStack: ( 4, 0) 8bd88e90 0x00000000
89326a40 irpStack: ( 4, 0) 8bd88e90 0x00000000
894ff3c0 irpStack: ( 4, 0) 8bd88e90 0x00000000
8950a008 irpStack: ( 4, 0) 8bd88e90 0x00000000
89974d50 irpStack: ( 4, 0) 8bd88e90 0x00000000
89a0ad50 irpStack: ( 4, 0) 8bd88e90 0x00000000
89a19d50 irpStack: ( 4, 0) 8bd88e90 0x00000000
89a865e8 irpStack: ( 4, 0) 8bd88e90 0x00000000
89ce1820 irpStack: ( 4, 0) 8bd88e90 0x00000000
89d2a7f0 irpStack: ( 4, 0) 8bd88e90 0x00000000
89ecea60 irpStack: ( 4, 0) 8bd88e90 0x00000000
8a17e268 irpStack: ( 4, 0) 8bd88e90 0x00000000
8a29ec20 irpStack: ( 4, 0) 8bd88e90 0x00000000
8a42e4d0 irpStack: ( 4, 0) 8bd88e90 0x00000000
8a442ab0 irpStack: ( 4, 0) 8bd88e90 0x00000000
8a44a6b8 irpStack: ( 4, 0) 8bd88e90 0x00000000
8a630548 irpStack: ( 4, 0) 8bd88e90 0x00000000
8a6705d0 irpStack: ( 4, 0) 8bd88e90 0x00000000
8a730008 irpStack: ( 4, 0) 8bd88e90 0x00000000
8a866108 irpStack: ( 4, 0) 8bd88e90 0x00000000
8a998008 irpStack: ( 4, 0) 8bd88e90 0x00000000
8aa512d0 irpStack: ( 4, 0) 8bd88e90 0x00000000
8aac8490 irpStack: ( 4, 0) 8bd88e90 0x00000000
8acd4450 irpStack: ( 4, 0) 8bd88e90 0x00000000
8afa12a8 irpStack: ( 4, 0) 8bd88e90 0x00000000
Module
Company Name: KVS Inc.
File Description: Enterprise Vault File System Filter
Product Version: (1.0:0.0)
File Version: (2.0:0.0)
File Size (bytes): 68480
File Date: Mon Jul 04 00:15:36 2005
Module TimeDateStamp = 0x42934e79 - Tue May 24 16:55:37 2005
Module Checksum = 0x000161f6
Module SizeOfImage = 0x00010b80
Module Pointer to PDB =
Module PDB Guid = {3D76A8B9-6A9F-42B7-8A4A-B563157617E8}
Module PDB Age = 0x1
- Module
Company Name: KVS Inc.
File Description: Enterprise Vault File System Filter
Product Version: (1.0:0.0)
File Version: (2.0:0.0)
Make sure support is aware of the file version, as I do not think this is the latest, there may be a hotfix that applies to your situation.