Highlighted

Storage Queue files disappearing

Had some strange situations in the past with some customers. 

They all had problems with the VSA not longer beeing able to access storage queue files or the storage queue location until they did a restart of the server. Problem was that we were never able to check the situation before the customer restarted the ev-server.

After the restart alle storage queue files in question were missing. 

 

So now I had a customer where I was able to see storage queue files disappearing. We were checking the relevant storagequeuebatch sql-table for some IDs. At this time some of the IDs were in this table and also als the file was available in the file location. After a few minutes we saw event 29060,29010 and 29059 occur stating that ev is not able to access the storage queue files and that we have to repair them with the help of evsvr. 

After that the entry for this storage queue file in the sql-table as well as the file in the storage queue location was missing. 

Customer uses Trendmicro Antivirus. compatibility is pending. deactivating does not change the behavior. exclusions are set.

Its strange - Is EV deleting this files by itself? 

Could there be anything letting EV rollback and delete the storage queue files? 

 

Event 29060 and 29010 threw some specific transaction IDs. I took one of these IDs and checked some tables for it. There is no entry in the saveset table but an entry in the journalarchive table.

Details:

idPartition = -1 (Never ingested to storage)

Indexcommited and backupcomplete = 0

RecordCreationDate = this morning

 

Environment: 

EV 11.0.1 CHF4. EMC Centera as partition. Safetycopy set to "yes, in the original location"

Would appreciate any idea or comment on this. 

 

 

PMCS GmbH & Co. KG - A Serviceware Company
www.serviceware.de
6 Replies

Re: Storage Queue files disappearing

You need to make sure the anti virus software is not scanning that location. It needs to be excluded asap

Re: Storage Queue files disappearing

as stated above exclusions are set based on https://www.veritas.com/support/en_US/article.000032085

PMCS GmbH & Co. KG - A Serviceware Company
www.serviceware.de

Re: Storage Queue files disappearing

May want to try putting an Eicar test file in that directory and see if it stays http://www.eicar.org/85-0-Download.html, just to make sure exclusions are working.

Re: Storage Queue files disappearing

we deactivated the antivirus services last week but storage queue files are still disappearing. I think this may be unrelated to the antivirus solution - feels more like ev is rolling back. 

Probably going to open a Veritas case. 

 

 

PMCS GmbH & Co. KG - A Serviceware Company
www.serviceware.de

Error messages stopped

Error messages stopped occuring a few days after my last update to this post. 

Together with Veritas we verified that EV tried to roll back these items. 

 

The initial cause of the problem is still a bit unclear. 

Further analyzing the eventlogs showed us that EV reached the storage queue threshold a few days before these errors occured and refused to accept new items. At this time the customer set the shortcuttimeout to 0 as he wanted to reset the pending items. Not sure but the rollback of EV may be related to this. 

 

If it is occuring again i'll try to analyze this and update the post. 

PMCS GmbH & Co. KG - A Serviceware Company
www.serviceware.de

Re: Error messages stopped

Hi Marc,

I ran into this article. It is specifically for EV 11.x and seems to be fixed in EV12.

could this be the issue you were seeing?

http://www.veritas.com/docs/000115512

Regards. Gertjan