cancel
Showing results for 
Search instead for 
Did you mean: 

Error while peforming Searches in CA

Cdee
Level 6

Hi,

Whil Peforming a search in CA for a specified Period of time i am getting error in search as below.

Search request failed. Reason: %1 Index Id:%2 Requestor:%3 Search Query: %4 Search Argument: %5 [0xc0041c67]

I have checked the Event Logs and i get the Below event at the sametime.
 

Event Type: Error
Event Source: Enterprise Vault
Event Category: Index Server
Event ID: 7264
Date:  6/8/2011
Time:  5:31:34 PM
User:  N/A
Computer: JournalServer
Description:Abnormal error occurred 
Error:     A low level indexing operation has failed   Error:    Index Id:       [0xc0041c43]
Reference: CAltaVistaSearch::ChunkedSearch
Index:     1C8A2E6A8DBEC064D9A1B2F03A7D5286D1110000journalingJrn/Volume:328 (Vault Site name)
Info:      EM/ce

Event Type: Error
Event Source: Enterprise Vault
Event Category: Index Server
Event ID: 7264
Date:  6/8/2011
Time:  5:31:34 PM
User:  N/A
Computer: JournalServer
Description:Abnormal error occurred 
Error:     A low level indexing operation has failed   Error:    Index Id:       [0xc0041c43]
Reference: CAltaVistaSearch::ProcessChunkResults
Index:     1C8A2E6A8DBEC064D9A1B2F03A7D5286D1110000journalingJrn/Volume:328 (Vault Site name)
Info:      EM/h
           
        
Event Type: Error
Event Source: Enterprise Vault
Event Category: Index Server
Event ID: 7235
Date:  6/8/2011
Time:  5:31:34 PM
User:  N/A
Computer: JournalServer
Description:
A low level indexing operation has failed
Error: document data too long [AV:20]
Index Id: 1C8A2E6A8DBEC064D9A1B2F03A7D5286D1110000journalingJrn/Volume:328 (Vault Site name)

I have tried to search on this errors but could not find much info.
I also have checked the index and they all show me as normal and none are failed.

          

1 ACCEPTED SOLUTION

Accepted Solutions

Kenneth_Adams
Level 6
Employee Accredited Certified

In addition to Paul's note about TCP Chimney, check the network cards on the EV, CA and SQL Servers for any Offload or Receive Side Scaling and disable these features.  Access the properties of the network card, then click on the Advanced button to access the tabs that have these features.

The error you received is also indicative of possible issues between the index volume's physical size and the complexity of the CA search.  First, check the physical size of the index volumes being searched.  The error is associated with a particular archive, so you can do a quick check of all of the index volumes for that archive (unless there are over 10 or so).  Any index volume that has its physical folder size over 3 GB can cause this error when the search criteria is even slightly complex.

When CA performs a search, it sends a request to the search engine to load into memory the index volumes for the archives selected to be searched.  Each IndexServer process that is used to search the index volumes is allowed a maximum of 2 GB of memory (physical and virtual).  If the physical size of the index volume is larger than 2 - 3 GB, then the entire index volume cannot be loaded into memory and must be swapped out.  This swapping can cause issues with the search, which can throw the error you received.

If you have more than 10 or so index volumes for the archive, then you can use the volume number referenced to find the actual physical index volume folder.

Look in the CA customer database for the table tblIndexVolumeSet.  Look in the right-most column of this table for the volume number (328).  Look in the row that contains that volume number for the column named FirstIndexSequenceNumber.  This is the number at the beginning of the numbers shown for each index volume in the archive's properties.  Note the physical path for that volume and use Windows Explorer to access that path.  Then take that FirstIndexSequenceNumber and add it with an underscore to the end of the ArchiveID to find the physical folder in that location.

For example, you have ArchiveID 1C8A2E6A8DBEC064D9A1B2F03A7D5286D1110000journalingJrn  shown in your posts.  Let's presume the FirstIndexSequenceNumber of Volume 328 is 153982.  Putting these together would give you the folder name of 1C8A2E6A8DBEC064D9A1B2F03A7D5286D1110000journalingJrn_153982.

One of the things we've found to help releave the error you received is to perform a major compaction of the index volume.  To do that, create an empty file named Compact.task in the physical folder, then run the 'Update Index Volume' action through the Vault Admin Console (right click on the index volume and selec this option).  Note that no search can be run against the archive by CA while the major compaction is running.  The EV Event Log should provide informational events to let you know the update has started and completed.  Once the completion event is thrown, the archive can be searched by CA.

Now, the complexity of the search terms may need to be broken into multiple searches.  If the search terms have lots of wild cards, then the complexity can easily exceed the available resources for the index volume in memory.  Remember that the search criteria must also fit into the 2 GB memory limitation of the IndexServer process, so the more complex the search criteria, the less memory available to the actual index data at any given time.

The ultimate 'fix' to the error would be to rebuild the index volume with the AVSMaxLoc registry setting configured with the value 500000000 (500 million decimal).  This value will keep the physical index volume folder size to between 2 and 3 GB.  We don't recommend rebuilding an index volume unless it's the last thing to do to resolve an issue as this operation can take a long time, especially if you're rebuilding multiple index volumes for the same archive.

I hope this helps you resolve the issue.

View solution in original post

10 REPLIES 10

TonySterling
Moderator
Moderator
Partner    VIP    Accredited Certified

What version of EV and CA?

Can you locate the AVtrace.log file in the folder for this index? 1C8A2E6A8DBEC064D9A1B2F03A7D5286D1110000journalingJrn/Volume:328

Cdee
Level 6

EV and CA version is 9.0.1 and in the log i can see this.

Thu Jan 20 13:13:44.577 2011 (thread:9356) backtrace: backtrace starts:
Thu Jan 20 13:13:44.577 2011 (thread:9356) backtrace: ..\indexlib\emalloc.c:12: (errno:0:No error) (result:0x00000800) malloc
Thu Jan 20 13:13:44.577 2011 (thread:9356) backtrace: ..\indexlib\xstdio.c:518: (errno:0:No error) (result:0x00000800)
Thu Jan 20 13:13:44.577 2011 (thread:9356) backtrace: ..\indexlib\xstdio.c:489: (errno:0:No error) (result:0x00000800)
Thu Jan 20 13:13:44.577 2011 (thread:9356) backtrace: ..\indexlib\xstdio.c:508: (errno:0:No error) (result:0x00000800) D:\Index Data\index2\1C8A2E6A8DBEC064D9A1B2F03A7D5286D1110000journalingMUM/ind_Cb.1
Thu Jan 20 13:13:44.577 2011 (thread:9356) backtrace: ..\indexlib\xstdio.c:437: (errno:0:No error) (result:0x00000800)
Thu Jan 20 13:13:44.593 2011 (thread:9356) backtrace: ..\indexlib\index.c:4238: (errno:0:No error) (result:0x00000800)
Thu Jan 20 13:13:44.593 2011 (thread:9356) backtrace: ..\indexlib\index.c:4376: (errno:0:No error) (result:0x00000800)
Thu Jan 20 13:13:44.593 2011 (thread:9356) backtrace: ..\indexlib\index.c:4497: (errno:0:No error) (result:0x00000800)
Thu Jan 20 13:13:44.593 2011 (thread:9356) backtrace: ..\indexlib\index.c:4583: (errno:0:No error) (result:0x00000800)
Thu Jan 20 13:13:44.593 2011 (thread:9356) backtrace: ..\querylib\qtree_exec.c:1240: (errno:0:No error) (result:0x00000800)
Thu Jan 20 13:13:44.593 2011 (thread:9356) backtrace: ..\querylib\qtree_exec.c:1292: (errno:0:No error) (result:0x00000800)
Thu Jan 20 13:13:44.593 2011 (thread:9356) backtrace: ..\querylib\qtree_exec.c:251: (errno:0:No error) (result:0x00000800)
Thu Jan 20 13:13:44.593 2011 (thread:9356) backtrace: ..\querylib\qtree_exec.c:283: (errno:0:No error) (result:0x00000800)
Thu Jan 20 13:13:44.593 2011 (thread:9356) backtrace: ..\querylib\qtree_exec.c:392: (errno:0:No error) (result:0x00000800)
Thu Jan 20 13:13:44.593 2011 (thread:9356) backtrace: ..\querylib\qtree_exec.c:366: (errno:0:No error) (result:0x00000800)
Thu Jan 20 13:13:44.593 2011 (thread:9356) backtrace: ..\querylib\qtree_exec.c:354: (errno:0:No error) (result:0x00000800)
Thu Jan 20 13:13:44.593 2011 (thread:9356) backtrace: ..\querylib\qtree_exec.c:432: (errno:0:No error) (result:0x00000800)
Thu Jan 20 13:13:44.593 2011 (thread:9356) backtrace: ..\querylib\qtree_exec.c:201: (errno:0:No error) (result:0x00000800)
Thu Jan 20 13:13:44.593 2011 (thread:9356) backtrace: ..\querylib\qtree_exec.c:232: (errno:0:No error) (result:0x00000800)
Thu Jan 20 13:13:44.593 2011 (thread:9356) backtrace: ..\qsdklib\ni2query.c:3694: (errno:0:No error) (result:0x00000800)
Thu Jan 20 13:13:44.593 2011 (thread:9356) backtrace: ..\api\avs_search.c:737: (errno:0:No error) (result:0x00000800) starting catch block
Thu Jan 20 13:13:44.593 2011 (thread:9356) backtrace: ..\api\avs_search.c:753: (errno:0:No error) (result:0x00000800) no specific catch clause caught
Thu Jan 20 13:13:44.593 2011 (thread:9356) backtrace: backtrace completed.
Thu Jan 20 13:13:44.593 2011 (thread:9356) avs_search: error: can't allocate memory
Thu Jan 20 13:13:55.749 2011 (thread:996) avs_close: locked out
Thu Jan 20 13:13:55.749 2011 (thread:996) avs_release_valtypes: locked out

TonySterling
Moderator
Moderator
Partner    VIP    Accredited Certified

Look at the properties of that archive on the Index tab.  Highlight that index volume, do you have the repair option?  If yes, please click it.

Don't choose Rebuild just yet, that can take some time and may not be necessary.

Cdee
Level 6

But i dont see any index as marked failed to repair in the index tab.

Also how can i identify which index to repair..

TonySterling
Moderator
Moderator
Partner    VIP    Accredited Certified

This is your index: 1C8A2E6A8DBEC064D9A1B2F03A7D5286D1110000journalingJrn and the specific volume is Volume:328

Cdee
Level 6

Thats my problem that i cant find 1C8A2E6A8DBEC064D9A1B2F03A7D5286D1110000journalingJrn in Index volume tab it only shows me the index location , range , status

TonySterling
Moderator
Moderator
Partner    VIP    Accredited Certified

oops!  Sorry about that. blush

How many volumes do you have?

I think you can right click, do a select all, then choose copy.  Paste that in notepad and I think that one of the ID's displayed will match the 328.  That should be the problematic index.

Paul_C_
Level 3

Also , make sure TCP Chimney and / or TCP Offload Engine (TOE) features are disabled

 

There is a tech note

 

http://www.symantec.com/business/support/index?page=content&id=TECH62307&actp=search&viewlocale=en_U...

hope it helps,

 

Rgds,

Paul

Cdee
Level 6

Hi,

Thanks for pointing this out just noticed that TCP chimney is enabled on Journal / CA and archiving server.
will disable the same and then check how it looks.

Kenneth_Adams
Level 6
Employee Accredited Certified

In addition to Paul's note about TCP Chimney, check the network cards on the EV, CA and SQL Servers for any Offload or Receive Side Scaling and disable these features.  Access the properties of the network card, then click on the Advanced button to access the tabs that have these features.

The error you received is also indicative of possible issues between the index volume's physical size and the complexity of the CA search.  First, check the physical size of the index volumes being searched.  The error is associated with a particular archive, so you can do a quick check of all of the index volumes for that archive (unless there are over 10 or so).  Any index volume that has its physical folder size over 3 GB can cause this error when the search criteria is even slightly complex.

When CA performs a search, it sends a request to the search engine to load into memory the index volumes for the archives selected to be searched.  Each IndexServer process that is used to search the index volumes is allowed a maximum of 2 GB of memory (physical and virtual).  If the physical size of the index volume is larger than 2 - 3 GB, then the entire index volume cannot be loaded into memory and must be swapped out.  This swapping can cause issues with the search, which can throw the error you received.

If you have more than 10 or so index volumes for the archive, then you can use the volume number referenced to find the actual physical index volume folder.

Look in the CA customer database for the table tblIndexVolumeSet.  Look in the right-most column of this table for the volume number (328).  Look in the row that contains that volume number for the column named FirstIndexSequenceNumber.  This is the number at the beginning of the numbers shown for each index volume in the archive's properties.  Note the physical path for that volume and use Windows Explorer to access that path.  Then take that FirstIndexSequenceNumber and add it with an underscore to the end of the ArchiveID to find the physical folder in that location.

For example, you have ArchiveID 1C8A2E6A8DBEC064D9A1B2F03A7D5286D1110000journalingJrn  shown in your posts.  Let's presume the FirstIndexSequenceNumber of Volume 328 is 153982.  Putting these together would give you the folder name of 1C8A2E6A8DBEC064D9A1B2F03A7D5286D1110000journalingJrn_153982.

One of the things we've found to help releave the error you received is to perform a major compaction of the index volume.  To do that, create an empty file named Compact.task in the physical folder, then run the 'Update Index Volume' action through the Vault Admin Console (right click on the index volume and selec this option).  Note that no search can be run against the archive by CA while the major compaction is running.  The EV Event Log should provide informational events to let you know the update has started and completed.  Once the completion event is thrown, the archive can be searched by CA.

Now, the complexity of the search terms may need to be broken into multiple searches.  If the search terms have lots of wild cards, then the complexity can easily exceed the available resources for the index volume in memory.  Remember that the search criteria must also fit into the 2 GB memory limitation of the IndexServer process, so the more complex the search criteria, the less memory available to the actual index data at any given time.

The ultimate 'fix' to the error would be to rebuild the index volume with the AVSMaxLoc registry setting configured with the value 500000000 (500 million decimal).  This value will keep the physical index volume folder size to between 2 and 3 GB.  We don't recommend rebuilding an index volume unless it's the last thing to do to resolve an issue as this operation can take a long time, especially if you're rebuilding multiple index volumes for the same archive.

I hope this helps you resolve the issue.