Event ID 6883and 6760

Hi Guys,

I currently get a lot of errors in the EV logs for Storage Delete. I am running Ev 9.0 and have just completed a Move archive. That went petty smoothly, until I try an delete the old archive. after that i recieve the errors every 5 - 10 mins. We are using centera for our storage and compliance mode is off.

Event 6760

Error from EMC Centera FPLibrary

Function call: NetworkPacket.checkControl(17)<HPPPurgeBlobTransaction::run()<Cluster:Smiley Very HappyeleteBlob(BGCLVS6LB9315eD6GF95514H3GVG4151FPEAEC03AG2AVHH2G5H6J,,0)<ClusterCloud:Smiley Very HappyeleteBlob(BGCLVS6LB9315eD6GF95514H3GVG4151FPEAEC03AG2AVHH2G5H6J,,0) with 6 retries<FPClip:Smiley Very Happyelete(pool,BGCLVS6LB9315eD6GF95514H3GVG4151FPEAEC03AG2AVHH2G5H6J,,0)<_FPClip_Delete(pool,BGCLVS6LB9315eD6GF95514H3GVG4151FPEAEC03AG2AVHH2G5H6J,NULL,0)<FPClip_Delete(-,BGCLVS6LB9315eD6GF95514H3GVG4151FPEAEC03AG2AVHH2G5H6J)

Status: FP_TRANSACTION_FAILED_ERR (-10156)

Reason: Transaction on the server failed (transid='EvServerName/86022/DELETE_CLIP'

 

Event 6760=

Unable to complete deletion request

 

Reason: Error from EMC Centera FPLibrary Function call: #1 Status: #2 (#3) Reason: #4 Pool Address: #5 [0xc0041a68]

Vault: AbdulAziz M. Al - Haider [2008-01-31 - 2011-01-11]

Vault Id: 100F4F30FAF2E4C4C83A90335A58C88D01110000evsite_mobily

Item Id: C1523CE86ECEC6F23567BF1DD0AFE001

Reference: DIFCS/cd

Supplementary Info: BGCLVS6LB9315eD6GF95514H3GVG4151FPEAEC03AG2AVHH2G5H6J

 

 Any Ideas

 

1 Solution

Accepted Solutions
Accepted Solution!

So what the above means,

So what the above means, either the replica centera is down and can't be connected to , or the CLIPs cannot be found.

You can use either JCAS Scripts or EVSVR to attempt to get the clip

To use JCAS to extract the blob
http://www.symantec.com/business/support/index?page=content&id=TECH65517&key=50996&actp=LIST


To use EVSVR to get the item
http://www.symantec.com/business/support/index?page=content&id=HOWTO37697&cat=USING&key=50996&actp=L...


Really though what i would do is the following

1. RDP to your EV Server
2. Stop the Enterprise Vault Storage Service
3. Open a command prompt and CD \Program Files\Enterprise Vault
4. Type "Dtrace" and press Enter"
5. Type "set StorageDelete v" and press Enter
6. Type "log C:\Temp\EVDeletion.log" and press Enter
7. Minimize the command prompt
8. Go to Control Panel -> System -> Advanced (tab)
9. Press the "Environment Variables" button
10. Under "System Variables" press the New button
11. For Variable Name enter "FPLIBRARY_LOG_FILE_DEBUG"
12. For Variable Value enter "C:\Temp\CenteraSDK.log"
13. Press OK and then Ok again
14. Start the Enterprise Vault Storage Service
15. Await the same errors to occur again
16. Stop the Enterprise Vault Storage Service again
17. Remove the Environment Variable we just added (otherwise it will consume all disk space)
18. Go back to your dtrace and type "Exit"

Now I would suggest calling EMC and passing them the SDK logs and they can help you through finding why the item can't be deleted (i.e. is it because a replica is offline? is it because the items are missing?)

Some other things could be that the centera is in compliance mode and you are attempting to delete an item that hasn't expired on the Centera yet, though this is doubtful.

Also try C:\Program Files\Enterprise Vault\CenteraVerify.exe, ensure that the program can connect to each of the centeras, including replicas, that it can read, write delete etc especially with the PEA file.

If this fails then its definitely an EMC issue, CenteraVerify.exe is a tool written by EMC and not Symantec.

It could also be that the PEA file doesn't have permissions to perform the deletion or doesn't have access, sometimes you have to add an environment variable to make sure the PEA File is passed to the SDK properly.

The Environment variable is CENTERA_PEA_LOCATION and you give it the value of the name and location of the PEA file (i.e C:\Centera\MyPEA.pea), once that change has been made you will have to restart the storage services
The only reason I don't think its a permission issue though is you typically will see very specific Access Denied type messages as opposed to what you are seeing.

Your best bet though is really working with EMC to make sure that the clips exist and it can connect to the Replicas and what not. If the CLIPs are missing then you should work with EMC to try and find out why they are missing etc, you may have to work with EV Support too to do any clean ups and what not and seeing if any other data is missing

 

 


 

https://www.linkedin.com/in/alex-allen-turl-07370146

View solution in original post

4 Replies

From the API Reference guide

From the API Reference guide for Centera

http://pages.cs.wisc.edu/~dewitt/includes/queryopt/API_Reference_Guide_069001185_A03.pdf
 

Page 100

C-Clips will only be deleted when all the parity fragments or mirror copies of the C-Clip have been found and successfully removed. When some nodes are down or when network problems occur, the delete or purge will fail and error -10156, FP_TRANSACTION_FAILED_ERR, will be returned. Try again later

https://www.linkedin.com/in/alex-allen-turl-07370146
Accepted Solution!

So what the above means,

So what the above means, either the replica centera is down and can't be connected to , or the CLIPs cannot be found.

You can use either JCAS Scripts or EVSVR to attempt to get the clip

To use JCAS to extract the blob
http://www.symantec.com/business/support/index?page=content&id=TECH65517&key=50996&actp=LIST


To use EVSVR to get the item
http://www.symantec.com/business/support/index?page=content&id=HOWTO37697&cat=USING&key=50996&actp=L...


Really though what i would do is the following

1. RDP to your EV Server
2. Stop the Enterprise Vault Storage Service
3. Open a command prompt and CD \Program Files\Enterprise Vault
4. Type "Dtrace" and press Enter"
5. Type "set StorageDelete v" and press Enter
6. Type "log C:\Temp\EVDeletion.log" and press Enter
7. Minimize the command prompt
8. Go to Control Panel -> System -> Advanced (tab)
9. Press the "Environment Variables" button
10. Under "System Variables" press the New button
11. For Variable Name enter "FPLIBRARY_LOG_FILE_DEBUG"
12. For Variable Value enter "C:\Temp\CenteraSDK.log"
13. Press OK and then Ok again
14. Start the Enterprise Vault Storage Service
15. Await the same errors to occur again
16. Stop the Enterprise Vault Storage Service again
17. Remove the Environment Variable we just added (otherwise it will consume all disk space)
18. Go back to your dtrace and type "Exit"

Now I would suggest calling EMC and passing them the SDK logs and they can help you through finding why the item can't be deleted (i.e. is it because a replica is offline? is it because the items are missing?)

Some other things could be that the centera is in compliance mode and you are attempting to delete an item that hasn't expired on the Centera yet, though this is doubtful.

Also try C:\Program Files\Enterprise Vault\CenteraVerify.exe, ensure that the program can connect to each of the centeras, including replicas, that it can read, write delete etc especially with the PEA file.

If this fails then its definitely an EMC issue, CenteraVerify.exe is a tool written by EMC and not Symantec.

It could also be that the PEA file doesn't have permissions to perform the deletion or doesn't have access, sometimes you have to add an environment variable to make sure the PEA File is passed to the SDK properly.

The Environment variable is CENTERA_PEA_LOCATION and you give it the value of the name and location of the PEA file (i.e C:\Centera\MyPEA.pea), once that change has been made you will have to restart the storage services
The only reason I don't think its a permission issue though is you typically will see very specific Access Denied type messages as opposed to what you are seeing.

Your best bet though is really working with EMC to make sure that the clips exist and it can connect to the Replicas and what not. If the CLIPs are missing then you should work with EMC to try and find out why they are missing etc, you may have to work with EV Support too to do any clean ups and what not and seeing if any other data is missing

 

 


 

https://www.linkedin.com/in/alex-allen-turl-07370146

View solution in original post

JesusWept2, this is

JesusWept2, this is awesome.

I will try what you suggested and let you know the out come.

Hey JesusWept2,   I have

Hey JesusWept2,

 

I have tried this but the results between the Dtrace and SDK Log seem to be skewed, somehow there seems to be a delay of 5 hours between the 2.

 

Good thing to know is that Symantec Support reference this site as well, i logged a case with them and they sent me my own artical back... Cool Hey Smiley Happy