cancel
Showing results for 
Search instead for 
Did you mean: 

Not enough storage is available to complete this operation. [0x8007000e]

Hermann2009
Level 4
Dear all,

we run Enterprise Vault 8 SP3 and SQL 2005 on the same Windows 2008 X64 Server.
Serverhardware: DL380 G5, 2 CPU, 4GB RAM & enough diskspace ( 2T )

Because of troubles with access of the Virtual Vault (no items are visible in the virtual vault & archive explorer) i found the problem in the index process.
The Vault Server gives me 2 error's (see below)

On the support page i found a solution:
http://seer.entsupport.symantec.com/docs/293961.htm

I increase the virtual memory up to 6196MB, server reboot, after 10 Minutes rebuilding of index, the same error.

Is there really the (low) physical memory (4GB) the problem?
Any other ideas to index the mailbox successful (4943MB & 33242 objects) without hardware increase?

the event logs errors:

################
Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          14.12.2009 10:00:28
Event ID:      7180
Task Category: Index Server
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Servername
Description:
Could not access indexable item: Not enough storage is available to complete this operation.  [0x8007000e] 
Internal reference: PFS/r

#################
Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          14.12.2009 10:00:28
Event ID:      7284
Task Category: Index Server
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Servername
Description:
Failed to add item to the index due to failure loading the item's index metadata.
Reason: Not enough storage is available to complete this operation.  [0x8007000e]
Index Id: 12BB1CF84C973A343A465F21A59FD64721110000vie-eu-archiv/Volume:30 (my colleague)
Saveset Id: 200911269244393~200809121611480000~Z~90882F5F9D65E7B734B5CA6F6C4122E1
Internal reference: II load

br

1 ACCEPTED SOLUTION

Accepted Solutions

Wayne_Humphrey
Level 6
Partner Accredited Certified
Well I normally recommend 4GB RAM for Enterprise Vault, I never recommend using SQL on the same box. The OS itself consumes about 512MB RAM, and saying that normally most people have min of 8GB RAM for SQL.  In most of my cases 16GB RAM for SQL is the norm.

So you can see SQL will be hogging all the RAM from Enterprise Vault.

View solution in original post

16 REPLIES 16

Wayne_Humphrey
Level 6
Partner Accredited Certified
Hermann2009,

Use PerfMon to see monitor your Memory usage. You might also wanna add page file. See this:
http://www.evdiscuss.net/blogs/viewpost/87

Once you have some more info please attach it to your first post ;)

BTW the link you provided does not work.

Hermann2009
Level 4
Dear Wayne,
thx for the link, i will check the memory usage with perfmon.
Additional i fixed the link problem...
have you any experience in minimum hardware requirement in real live systems?
are 4GB Ram for Vault & SQL on the same hardware for indexing of 5GB mailboxes enough?
br

Wayne_Humphrey
Level 6
Partner Accredited Certified
Well I normally recommend 4GB RAM for Enterprise Vault, I never recommend using SQL on the same box. The OS itself consumes about 512MB RAM, and saying that normally most people have min of 8GB RAM for SQL.  In most of my cases 16GB RAM for SQL is the norm.

So you can see SQL will be hogging all the RAM from Enterprise Vault.

Hermann2009
Level 4
i will upgrade the RAM from 4GB to 12GB, thx for the support!

Michael_Bilsbor
Level 6
Accredited
Hi,

I am not at all sure that physical memory is the issue.  It sounds like it's trying to open up the indexable item (II) from the archived item. So it's probably very large one item

Are you saying that this error is definately leading to problems with VV, because I find that surprising.

Wayne_Humphrey
Level 6
Partner Accredited Certified
Mike,

Sorta agree on that statement but I think the key element here is windows 2008 hovers around 800MB RAM doing nothing, then adding SQL 2005, that can peak, lets just say it is consuming 3GB that leaves EV with 200MB RAM see where its going so yes a big item, but what happens if its arching at the same time?

Hermann2009
Level 4
Hi EV Director,
"Are you saying that this error is definately leading to problems with VV, because I find that surprising."

I start to archiv old emails for some users, everything works fine (more or less, the outlook addin is worst),
only 2 users are unable to sync the virtual vault cache with the Enteprise Vault Server.

So I'am starting to search the problem with the result, the index of 2 users was failed.
Update the Index takes 1 day with high CPU & all available memory usage and 1000+1 error in the eventlog...
So I'am not sure if the index is the problem, because i could not finish the rebuild.

Next step is to increase the Hardware ( additional 8GB RAM -> 12GB)
Following we try again to rebuild the index ...
If it fails, i have to start to work with DTrace -> troubleshooting troubleshooting troubleshooting, .... really nice product!

On this point i have to say: Enterprise Vault isn't really stable, it takes 2 month to go online (lot of toubles with the configuration)
now we have 35 users on the Vault and the System breaks down...  -> NO ADDITIONAL COMMENT :(

Michael_Bilsbor
Level 6
Accredited
Hi,

You are correct in that VV will not work correctly if the Index has been marked as failed.  However that even you mention on it's own should not lead to an index being marked as failed.  Are there other events from the Index Server as those might be the ones to troubleshoot.

Mike

Hermann2009
Level 4
Yes, if the index is failed the VV doesn't show any archived emails.
I have no idea why, because if i check the local cache it looks ok.

Back to the Server, i start to rebuild the index, the server cpu is very bussy and memory is nearly all available empty.
In the eventlog it starts with information 7314:

Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          15.12.2009 15:08:18
Event ID:      7314
Task Category: Index Server
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      Servername
Description:
Index volume update in progress
Index Volume: 1AE898FE00470684CB2EF5EFBCC7DB7851110000vie-eu-archiv/Volume:36 (Username)
Percentage completed: 168%
Estimated time of completion: 15.12.2009 15:29
Index Volume Path: E:\Index\index7\1AE898FE00470684CB2EF5EFBCC7DB7851110000vie-eu-archiv
Job Id: Vol_36
Job Author: Serviveusername
Job Description: Rebuild index volume
Job elapsed time: 01:32:56 (hours:minutes:seconds)
Processing time: 01:31:41 (hours:minutes:seconds)
Number of items added: 53056
Number of items additions abandoned: 0
Number of items deleted: 0
Number of items updated: 0

Following from error 7235:

Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          15.12.2009 15:25:56
Event ID:      7235
Task Category: Index Server
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Servername
Description:
A low level indexing operation has failed
Error: can't allocate memory [AV:1]
Index Id: 1AE898FE00470684CB2EF5EFBCC7DB7851110000vie-eu-archiv/Volume:36 (Username)

Next Error 7313:

Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          15.12.2009 15:25:56
Event ID:      7313
Task Category: Index Server
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Servername
Description:
Failed to commit Index updates to disk.
Reason: Not enough storage is available to complete this operation.  [0x8007000e]
Index Id: 1AE898FE00470684CB2EF5EFBCC7DB7851110000vie-eu-archiv/Volume:36 (Username)
Index Path: E:\Index\index7\1AE898FE00470684CB2EF5EFBCC7DB7851110000vie-eu-archiv
Available Disk Space: 115338444 KB(115338444 KB)
Number of Items added: 14352
Number of Items deleted: 0
Number of Items updated: 0
Number of new words: 2000136
Additional Info: Makestable
                 Vers:-1 Mem Load:1 (Max:1310720) KB
Last Error 7187:
Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          15.12.2009 15:25:56
Event ID:      7187
Task Category: Index Server
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Servername
Description:
Could not add Indexable Item and Per-user data to Index: 1AE898FE00470684CB2EF5EFBCC7DB7851110000vie-eu-archiv/Volume:36 (Username) 
Internal reference: CWC
Additional Info: a severe error occurred; cannot continue [AV:63]


The error 7235 & 7313 repeats 4x times, following from the Information, every 20 Minutes

Hermann2009
Level 4
Restart the Server, memory is in use, nearly 4GB Ram usage

Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          15.12.2009 16:13:33
Event ID:      7180
Task Category: Index Server
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Servername
Description:
Could not access indexable item: Not enough storage is available to complete this operation.  [0x8007000e] 
Internal reference: PFS/ce


??? Windows 2008 X64, Enteprise Vault 32 bit ??? -> maximum 4GB Ram?

Michael_Bilsbor
Level 6
Accredited
Hi,

ok so it's the make stable that is failing and this is causing the index to goto failed (make stable is the operation of writing the index to disk)
Memory certainly seems to be an issue, though it could be a per item problem as well.  Some of the events you are seeing should mention a saveset id or transaction id.  if so do you see the same saveset id being repeated each time?


Hermann2009
Level 4

YES IT IS, always the same Saveset ID:

Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          15.12.2009 17:12:04
Event ID:      7288
Task Category: Index Server
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      vie-eu-arc-02.global.domain
Description:
Addition of item to index abandoned.
Index: 1AE898FE00470684CB2EF5EFBCC7DB7851110000vie-eu-archiv/Volume:36 (Hasler Birgit)
Saveset Id: 200912110527112~200809221012000000~Z~6147BF0799F27DAE59A2A82511633E21
Sequence Number: 28788
All previous attempts to add the item to the index have failed. To prevent further failures the item will not be indexed.

Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          15.12.2009 17:12:04
Event ID:      7284
Task Category: Index Server
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      vie-eu-arc-02.global.domain
Description:
Failed to add item to the index due to failure loading the item's index metadata.
Reason: Could not access indexable item:     Internal reference:    [0xc0041c0c]
Index Id: 1AE898FE00470684CB2EF5EFBCC7DB7851110000vie-eu-archiv/Volume:36 (Hasler Birgit)
Saveset Id: 200912110527112~200809221012000000~Z~6147BF0799F27DAE59A2A82511633E21
Internal reference: II load
For more information, see Help and Support Center at http://evevent.symantec.com/rosetta/showevent.asp


Michael_Bilsbor
Level 6
Accredited
Hi,

ok just to get it going you could temporarily rename the file assocaited with that 'poisionous item' and see if it's able to move beyond that then.

Hermann2009
Level 4
the index rebuild over the nicht "works" with 1000+1 errors in the eventvwr.
Now the index is ok, virtual vault cache sync is ok, and the archive is available.
Why we have these troubles is unknown, because we upgrade the hardware from 4GB RAM up to 12GB RAM, the same error messages.

GertjanA
Moderator
Moderator
Partner    VIP    Accredited Certified
Hello Hermann,

On your comment:

On this point i have to say: Enterprise Vault isn't really stable, it takes 2 month to go online (lot of toubles with the configuration)
now we have 35 users on the Vault and the System breaks down...  -> NO ADDITIONAL COMMENT :(


I've done a few installations of EV, of which 3 totally new (80sp2 and sp3), and 3 upgrading (as of V6.0 to 7.0sp3 to 7.5sp2 to 8.0sp2 / 80sp3)

When making triplesure that all pre-reqs are met, and thouroughly thinking out your archiving setup (provisioning, vaultcache, OS, mailclient, mailserver, storage etc) I have not run into any issues. As for your info, I am running all EV-servers (28 now) as Virtual Machines, where OS en MSMQ are vmdk's, and indexes, vaultcache folder, and storage (dvs etc) are on a SAN. We're using W2003SP2. SQL200864bitSP1 on dedicated machine for EV only.


RAM on the VM's is 1.5 GB (yes, one and a half GB) works fine. I have 89900active archives. On a new environment, I have since early this week 700 archives, rising with about 200 weekly (will be 4500). Works fine.

You might have the ' murphy'  scenario, but I disagree when you say EV is unstable. Running EV and SQL on the same server is recommended for test and ' show'  usability, not live environment.

Having said that, I hope to be able to pinch in some help if I can. Wayne and Mike are some of the best you get on the forum to help out!

Keep posting your issues, who knows we might get them solved, and get you a set of happy users.

Good luck,






Regards. Gertjan

Liam_Finn1
Level 6
Employee Accredited Certified
Hermann,

What I recommend is that you have a Symantec partner work with you to help you get this working.

Setting it up is not that hard. It does require steps to be completed in order and seeing as you are having issues maybe having professional services help you set it up correctly and in the process show you how to do it correctly.

I have 16 servers serving my EV environment with over 30TB of archived data.

EV is a great product but it has allot of complexity in the background and having professional services assist you with the setup will show you the possibilities this software can offer you and your company
The forum members are always here to assist you when ever we can and when we are unable to help we will recommend you go to support. In many cases the experienced EV engineers that are in the forums will help you solve your issues.

Don't walk away from the produce because of issues setting up your test environment. Let the support forums or get professional services help you get it going then after you see how it is done and what it can do then make your decision as to continue working with the product or not