EV 8.0.4 FSA Problem retrieving WAV Files only

[ia01]

Hi,

We have a EV 8.0.4 FSA Archiving file server.

When we try to copy a folder with lots of files with lots of archived files ... all files are copied fine apart from the WAV files.

An unexpected error is keepinh you from copying the file. If you continue to receive this error, you can use the error code to search for help with this problem.

Error: 0x800710FE: This file is currently not available for use on this computer.

Any idea please?

Many Thanks

[ia01]

DTRACE of placeholder service

68 11:05:13.447 [5820] (EvPlaceholderService) <2480> EV:H BC 1

69 11:05:13.447 [5820] (EvPlaceholderService) <2480> EV:H dwcode 200

70 11:05:13.447 [5820] (EvPlaceholderService) <2480> EV:H BT 1

71 11:05:13.447 [5820] (EvPlaceholderService) <2480> EV:L EV-Content-Length not present in header

72 11:05:13.447 [5820] (EvPlaceholderService) <2480> EV:H Exception occured in CDownloadFile::ThreadMain(), error 0x80004005

73 11:05:13.447 [5820] (EvPlaceholderService) <2480> EV:H CDownloadFile::ThreadMain: exit with failure, error: 0x80004005

74 11:05:13.447 [5820] (EvPlaceholderService) <2480> EV:L CDownloadFile::ThreadMain (Exit) |Unspecified error  [0x80004005] |

75 11:05:13.447 [5820] (EvPlaceholderService) <820> EV:L Download status for file: \\?\P:\folder\file1.wav is 80004005

76 11:05:13.447 [5820] (EvPlaceholderService) <820> EV:L RequestArchivedFile::DownLoadFile (Exit) |Exception occurred.  [0x80020009] |

77 11:05:13.447 [5820] (EvPlaceholderService) <820> EV~E Event ID: 20491 Error downloading file: \\?\P:\folder\file1.wav, Url: http://vault.domain.local/EnterpriseVault/download.asp?VaultID=128D2CB150C560C4A8D7D26E1071549321110000mailvault&SavesetId=201301319599178~200307140837290000~Z~10E15420B85FD76900D0FE9CF2B3D981&Request=NativeItem&EVHeaders=1&HashName=SHA256 |Error Unspecified error  [0x80004005] |

78 11:05:13.447 [5820] (EvPlaceholderService) <820> EV:L RequestArchivedFile::RespondToFSADriver (Entry) |

79 11:05:13.447 [5820] (EvPlaceholderService) <820> EV:L Attempting to respond to FSA driver with downloaded file: \\?\P:\folder\file1.wav

80 11:05:13.447 [5820] (EvPlaceholderService) <820> EV:L RequestArchivedFile::RespondToFSADriver (Exit) |Success  [0] |

81 11:05:13.447 [5820] (EvPlaceholderService) <820> EV:L RequestArchivedFile::Process (Exit) |Success  [0] |

82 11:05:13.447 [5820] (EvPlaceholderService) <820> EV:L CQueue::DeQueue (Entry) |

83 11:05:13.447 [5820] (EvPlaceholderService) <5364> EV:M Filter message event signaled [WAIT_OBJECT_0 + 1]

84 11:05:13.447 [5820] (EvPlaceholderService) <5364> EV:M [EvRequestArchivedFile] Queueing placeholder request for file: P:\folder\file1.wav

85 11:05:13.447 [5820] (EvPlaceholderService) <5364> EV:L RequestArchivedFile::RequestArchivedFile (Entry) |

86 11:05:13.447 [5820] (EvPlaceholderService) <5364> EV:M WorkItem::GetExeName: Trying to get the .exe name for pid: 4

87 11:05:13.447 [5820] (EvPlaceholderService) <5364> EV:M WorkItem::GetExeNameUsingPHHelper: entry - PID:4

88 11:05:13.463 [5820] (EvPlaceholderService) <5364> EV:M WorkItem::GetExeNameUsingPHHelper: Failed to get .exe name for pid [4] using IPlaceholderSvcHelper, error: 0x80070005

89 11:05:13.463 [5820] (EvPlaceholderService) <5364> EV:M WorkItem::GetExeNameUsingPHHelper: exit - PID:4, exe name:

90 11:05:13.463 [5820] (EvPlaceholderService) <5364> EV:M WorkItem::GetExeNameUsingEnumProcessModules: The .exe name for for pid: 4 is

91 11:05:13.463 [5820] (EvPlaceholderService) <5364> EV:M WorkItem::GetExeName: The .exe name for for pid: 4 is

92 11:05:13.463 [5820] (EvPlaceholderService) <5364> EV:L RequestArchivedFile::RequestArchivedFile (Exit) |Success  [0] |

93 11:05:13.463 [5820] (EvPlaceholderService) <5364> EV:L CQueue::EnQueue (Entry) |

94 11:05:13.463 [5820] (EvPlaceholderService) <5364> EV:M CQueue::EnQueue item enQueued

95 11:05:13.463 [5820] (EvPlaceholderService) <5700> EV:M CQueue::DeQueue item deQueued

96 11:05:13.463 [5820] (EvPlaceholderService) <5364> EV:L CQueue::EnQueue (Exit) |Success  [0] |

97 11:05:13.463 [5820] (EvPlaceholderService) <5700> EV:L RequestArchivedFile::Process (Entry) |

98 11:05:13.463 [5820] (EvPlaceholderService) <5364> EV:M FilterGetMessage() returned [800703e5] : Overlapped I/O operation is in progress.

99 11:05:13.463 [5820] (EvPlaceholderService) <5700> EV:M Processing a placeholder restore request for file: \\?\P:\folder\file1.wav Url: http://vault.domain.local/EnterpriseVault/download.asp?VaultID=128D2CB150C560C4A8D7D26E1071549321110000mailvault&SavesetId=201301319599178~200307140837290000~Z~10E15420B85FD76900D0FE9CF2B3D981&Request=NativeItem&EVHeaders=1&HashName=SHA256, placeholder version: 1

100 11:05:13.463 [5820] (EvPlaceholderService) <5364> EV:M Wait for multiple objects...

101 11:05:13.463 [5820] (EvPlaceholderService) <5700> EV:L RequestArchivedFile::DownLoadFile (Entry) |

102 11:05:13.463 [5820] (EvPlaceholderService) <5700> EV:L Attempting to download file: \\?\P:\folder\file1.wav at Url: http://vault.domain.local/EnterpriseVault/download.asp?VaultID=128D2CB150C560C4A8D7D26E1071549321110000mailvault&SavesetId=201301319599178~200307140837290000~Z~10E15420B85FD76900D0FE9CF2B3D981&Request=NativeItem&EVHeaders=1&HashName=SHA256

103 11:05:13.463 [5820] (EvPlaceholderService) <4976> EV:L CDownloadFile::ThreadMain (Entry) |

104 11:05:13.463 [5820] (EvPlaceholderService) <4976> EV:L CDownloadFile::UpdateIntranetZoneSites (Entry) |

105 11:05:13.463 [5820] (EvPlaceholderService) <4976> EV:H |[CDownloadFile] Inside UpdateIntranetZoneSites

106 11:05:13.463 [5820] (EvPlaceholderService) <4976> EV:L CDownloadFile::UpdateIntranetZoneSites (Exit) |Success  [0] |

107 11:05:13.463 [5820] (EvPlaceholderService) <4976> EV:L WinHttpConnect handle = 0x6681eb8 for URL=http://vault.domain.local/EnterpriseVault/download.asp?VaultID=128D2CB150C560C4A8D7D26E1071549321110000mailvault&SavesetId=201301319599178~200307140837290000~Z~10E15420B85FD76900D0FE9CF2B3D981&Request=NativeItem&EVHeaders=1&HashName=SHA256

108 11:05:13.463 [5820] (EvPlaceholderService) <4976> EV:H ConnectForSegment:/EnterpriseVault/download.asp?VaultID=128D2CB150C560C4A8D7D26E1071549321110000mailvault&SavesetId=201301319599178~200307140837290000~Z~10E15420B85FD76900D0FE9CF2B3D981&Request=NativeItem&EVHeaders=1&HashName=SHA256&Segment=0

109 11:05:13.681 [5820] (EvPlaceholderService) <4976> EV:H BC 1

110 11:05:13.681 [5820] (EvPlaceholderService) <4976> EV:H dwcode 200

111 11:05:13.681 [5820] (EvPlaceholderService) <4976> EV:H BT 1

[Arjun_Shelke]

Is this related to large files? Do you get this event 6287 unable to fetch item, if yes then try following:

EV Storage code checks the existence of the IResponse interface registry key as part of normal processing for large file requests.
The IResponse interface is created during the installation of Internet Information Services with ASP enabled. The log analysis reveals the root-cause being with the IResponse interface for which the corresponding Registry entry was not present.

Placeholder Service Dtrace => EV-Content-Length not present in header
EV Server Dtrace => Passed IUnknown is not IIS IResponse

Try this:

Re-registering asp.dll with regsvr32.exe recreates the missing IResponse interface registry key. This file is located under: \%windir%\system32\inetsrv
 

[ia01]

Hi 

Yes that happens for large files.

Where is IResponse interface registry key ?

I have done re-registering asp.dll with regsvr32.exe but still no luck.

Does this reg key needs to be changed to something? 

Many Thanks

[Arjun_Shelke]

This interface key can be found under multiple locations....can you please confirm if the Dtrace of W3wp, StorageOnlineOPNS shows this error: Passed IUnknown is not IIS IResponse

[ia01]

No Don't get that error. Dtrace error below

424 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CVaultStorePartitionCache::ReadEntry Id:1770E32936872194CB35A03B7D46C5AC91210000mailvault hr:0

425 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CVaultStorePartitionCache::ReadEntry Id:1770E32936872194CB35A03B7D46C5AC91210000mailvault hr:0

426 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CMigratorHelper::AccessFile : Checking for primary copy of file : Q:\Enterprise Vault Stores\FSAStore4 Ptn15\2013\01-31\1\0D1\10D19514DF1D8F7CBD452A7B70F13271~2C~4F612794~00~1.DVFSP

427 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CMigratorHelper::DelayFileDeletion : PartitionEntryId = 17509FA9E80A42441B6D40D3B5E52A9511q10000mailvault : PartitionRootPath = Q:\Enterprise Vault Stores\FSAStore4 Ptn15 : WriteOnceMedia = 0 : File = Q:\Enterprise Vault Stores\FSAStore4 Ptn15\2013\01-31\1\0D1\10D19514DF1D8F7CBD452A7B70F13271~2C~4F612794~00~1.DVFSP

428 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CCabFileMutex::wait[LOCK] :  Attempting to take a lock on file Q:\Enterprise Vault Stores\FSAStore4 Ptn15\2013\01-31\1\0D1\10D19514DF1D8F7CBD452A7B70F13271~2C~4F612794~00~1.DVFSP.lock

429 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CCabFileMutex::wait :  handle: ffffffff

430 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CCabFileMutex::wait :  invalid file handle, going to sleep and wait to be unlocked

431 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CCabFileMutex::wait : total sleep 600000ms

432 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CCabFileMutex::wait :  final handle: ffffffff

433 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:L CCabFileMutex::wait : Could not lock file, sleep value: 128, slept for 0, last error: 0x80070070, returning: 0x80070020

434 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CCabFileMutex::wait[LOCK] :  Lock failed on file Q:\Enterprise Vault Stores\FSAStore4 Ptn15\2013\01-31\1\0D1\10D19514DF1D8F7CBD452A7B70F13271~2C~4F612794~00~1.DVFSP.lock

435 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CCabFileMutex::wait[LOCK] :  Cleaning up lock file Q:\Enterprise Vault Stores\FSAStore4 Ptn15\2013\01-31\1\0D1\10D19514DF1D8F7CBD452A7B70F13271~2C~4F612794~00~1.DVFSP.lock

436 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CCabFileMutex::wait[LOCK] :  Lock released on file Q:\Enterprise Vault Stores\FSAStore4 Ptn15\2013\01-31\1\0D1\10D19514DF1D8F7CBD452A7B70F13271~2C~4F612794~00~1.DVFSP.lock

437 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:H CNTFSStorageDeviceReader::AccessPotentiallyMigratedFile _com_error exception. hr=The process cannot access the file because it is being used by another process.  [0x80070020]

438 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CNTFSStorageDeviceReader::AccessPotentiallyMigratedFile (Exit). hr=The process cannot access the file because it is being used by another process.  [0x80070020]

439 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:H CNTFSStorageDeviceReader::LoadLargeMigratedFile _com_error exception. hr=The process cannot access the file because it is being used by another process.  [0x80070020]

440 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CNTFSStorageDeviceReader::LoadLargeMigratedFile (Exit). hr=The process cannot access the file because it is being used by another process.  [0x80070020]

441 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:H CNTFSStorageDeviceReader::GetFileStream _com_error exception. hr=The process cannot access the file because it is being used by another process.  [0x80070020]

442 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CNTFSStorageDeviceReader::GetFileStream (Exit). hr=The process cannot access the file because it is being used by another process.  [0x80070020]

443 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CNTFSStorageDeviceReader::GetSisPartStream (Exit). hr=Success  [0]

444 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:H CStorageSisPartReader<class CVaultStoreEMCCentera>::GetSisPart _com_error exception. hr=The process cannot access the file because it is being used by another process.  [0x80070020]

445 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CStorageSisPartReader<class CVaultStoreEMCCentera>::GetSisPart (Exit). hr=The process cannot access the file because it is being used by another process.  [0x80070020]

446 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:H CStorageSavesetReader<class CVaultStoreEMCCentera>::GetSisPart _com_error exception. hr=The process cannot access the file because it is being used by another process.  [0x80070020]

447 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CStorageSavesetReader<class CVaultStoreEMCCentera>::GetSisPart (Exit). hr=The process cannot access the file because it is being used by another process.  [0x80070020]

448 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:H CSisPartConstructorBase::LoadSisPart _com_error exception. hr=The process cannot access the file because it is being used by another process.  [0x80070020]

449 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CSisPartConstructorBase::AddSisPartToBlacklist (Entry)

450 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CStorageSavesetReader<class CVaultStoreEMCCentera>::GetSisPartIdentifier (Entry)

451 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CSavesetLocator::GetSisPartLocation (Entry) AttachmentId = [0]

452 08:53:45.730 [8508] (StorageOnlineOpns) <7596> EV:M CSavesetLocator::GetSisPartLocation (Exit). hr=Success  [0]

[Arjun_Shelke]

This could be related to storage, do you have enough space on vault store partition? Storage is NTFS or Centera? Do you migrate archived files to secondary device?

StorageOnlineOPNS spawns multiple threads while retrieving archived items, if collection has been enabled and request made to retrieve multiple files from same cab file, and 1st thread takes too long to access the archived file then this behaviour can be seen.

[ia01]

No Collection is not enabled.

When we try to view the same file via search we get ERROR 500 

500 - Internal server error.

There is a problem with the resource you are looking for, and it cannot be displayed.

[Arjun_Shelke]

Ok then you might want to try if at all you can access this item from your storage using dumpsaveset, if its too much for you then its better to contact Symantec Support and log a case.

[ia01 · Accepted solution]

I think we have fugured out the issue. One of disk volume containing vault store partition has 0 byte free space. Therefore EV cannot create a lock file for that corresponding file on vault store. We going to extend the disk and hopefully the issue should be resolved.

[Arjun_Shelke]

Yes I had asked you this in earlier comment...

"This could be related to storage, do you have enough space on vault store partition?"

Anyways...glad to know that issue will be resolved.

[ia01]

We have extended the disk and it works fine now.

Anyway thanks for your help Arjun

[plaudone]

Space is most likely the issue and hopefully that will resolve the problem.  If the issue continues then it would be suggested to upgrade to the latest version of EV 10 as there were changes made in how EV handles creating the lock on CAB files and large files using the CCabFileMutex method. 

[Arjun_Shelke]

Cheers!!