10-19-2012 04:38 AM
Yesterday we had EV server queue pile up issue for all EV servers, As we checked the server and confirmed the below information.
1. Confirmed that the MSMQ A5 queues are piled up.
2. Verified all the EV services are running fine.
3. Checked the server event log and confirmed that there was SQL database instance connectivity issue. Please find below event logs
Log Name: Symantec Enterprise Vault
Source: Enterprise Vault
Date: 10/19/2012 5:03:51 PM
Event ID: 13397
Task Category: Storage Online
Level: Warning
Keywords: Classic
User: N/A
Computer: ev03.stf.nus.edu.sg
Description:
The connection 'Provider=SQLOLEDB;Server=evdb01;Database=EVStudentVaultStore1;Trusted_Connection=Yes' was lost and the system is waiting to reconnect (Thread Id: 11336)
For more information, see Help and Support Center at http://evevent.symantec.com/rosetta/showevent.asp?EvtID=13397
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Enterprise Vault " />
<EventID Qualifiers="32772">13397</EventID>
<Level>3</Level>
<Task>47</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2012-10-19T09:03:51.000000000Z" />
<EventRecordID>1974692</EventRecordID>
<Channel>Symantec Enterprise Vault</Channel>
<Computer>ev03.stf.nus.edu.sg</Computer>
<Security />
</System>
<EventData>
<Data>Provider=SQLOLEDB;Server=evdb01;Database=EVStudentVaultStore1;Trusted_Connection=Yes</Data>
<Data>11336</Data>
</EventData>
</Event>
Log Name: Symantec Enterprise Vault
Source: Enterprise Vault
Date: 10/19/2012 5:04:06 PM
Event ID: 13395
Task Category: Directory Service
Level: Warning
Keywords: Classic
User: N/A
Computer: ev03.stf.nus.edu.sg
Description:
The connection 'Provider=SQLOLEDB;Server=evdb01;Database=EnterpriseVaultDirectory;Trusted_Connection=yes' was lost and the system failed to reconnect (Thread Id: 5492)
For more information, see Help and Support Center at http://evevent.symantec.com/rosetta/showevent.asp?EvtID=13395
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Enterprise Vault " />
<EventID Qualifiers="32772">13395</EventID>
<Level>3</Level>
<Task>21</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2012-10-19T09:04:06.000000000Z" />
<EventRecordID>1974697</EventRecordID>
<Channel>Symantec Enterprise Vault</Channel>
<Computer>ev03.stf.nus.edu.sg</Computer>
<Security />
</System>
<EventData>
<Data>Provider=SQLOLEDB;Server=evdb01;Database=EnterpriseVaultDirectory;Trusted_Connection=yes</Data>
<Data>5492</Data>
</EventData>
</Event>
Log Name: Symantec Enterprise Vault
Source: Enterprise Vault
Date: 10/19/2012 5:04:22 PM
Event ID: 6578
Task Category: Migrator Server
Level: Error
Keywords: Classic
User: N/A
Computer: ev03.stf.nus.edu.sg
Description:
Abnormal error occurred
Object: CSSASCache
Reference: RE(1)/fe
For more information, see Help and Support Center at http://evevent.symantec.com/rosetta/showevent.asp?EvtID=6578
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Enterprise Vault " />
<EventID Qualifiers="49156">6578</EventID>
<Level>2</Level>
<Task>29</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2012-10-19T09:04:22.000000000Z" />
<EventRecordID>1974726</EventRecordID>
<Channel>Symantec Enterprise Vault</Channel>
<Computer>ev03.stf.nus.edu.sg</Computer>
<Security />
</System>
<EventData>
<Data>CSSASCache</Data>
<Data>RE(1)/fe</Data>
<Binary>5468652053514C2064617461626173652073657276657220666F72202332206973206E6F7420617661696C61626C653A202020202020436865636B2074686174207468652053514C205365727665722069732076616C696420616E642069732072756E6E696E67202020202020202020496E7465726E616C207265666572656E6365282338292020204465736372697074696F6E3A202020202020233120202020204164646974696F6E616C204D6963726F736F667420737570706C69656420696E666F726D6174696F6E3A2020202020536F757263653A2020202020202023332020204E756D6265723A20202020202020233420202053514C2053746174653A2020202023352020204E6174697665204572726F723A20233620202048524553554C54202023372020202020202020283078383030343334346129</Binary>
</EventData>
</Event
4. As we verified EV database cluster services and confirmed that the cluster resource are up.
5. As a workaround we performed cluster SQL database failover and failed back and the issue got fixed. Again we had same issue EV issue today.
Kindly find the below update.
n What type of mail archiving Domino or Exchange used?
Exchange 2007
n Version of Enterprise vault used??
EV 8.0
n EV server platform?
All EV Server are running windows 2008 R2
Solved! Go to Solution.
10-19-2012 07:44 AM
Well Gertjan started you off nicely i think, check the SQL Servers for any intermittment communication issues or cluster issues, so it could be that the SQL Servers are experiencing a "blip" in communications , and EV just isn't handling the disconnect well.
So check with the SQL Server, look at the event logs and any other sql logs that may be showing it disconnecting from the network, or any long procedures that may be disconnecting the EV Services
Also check out the technote he stated: http://www.symantec.com/docs/TECH66826
And from my side I asked whether restarting the admin service would have resolved the issue (without having to resort to failing over SQL Server) and whether you have limited any connections on the SQL Server itself, could it be that EV is exhausting the amount of connections given to it?
Also when this happens, do you see anything abnormal on the SQL Servers? 100% CPU usage, higher than normal disk usage etc
If a restart of the EV Services does not work but failing the SQL Server *does* work, then you would have to assume its an issue with the SQL Service or the node at that particular time.
Also look to see if anything has changed in the environment
have you recently updated EV? updated SQL? changed Storage for the SQL Databases?
Enabled a new bunch of users? started any big PST Migrations? Vault Cache builds etc?
If it becomes a much larger issue then your best bet is to get DTraces of the directoryService on the EV Server, get the application, system and Enterprise Vault logs from the Event Viewer, and also Application and System logs from the SQL Server as well as any other Error logs and a snapshot of the activity monitor...then open a case with symantec to help troubleshoot the issue
10-19-2012 05:04 AM
I assume you are on EV8SP4? lower versions are not supported on W2008R2.
Are you using DNS Alias for SQL server (evdb01)
Can you check: http://www.symantec.com/docs/TECH66826
Can you confirm there are no issues on the SQL-cluster (like quorum getting lost etc). Backup's maybe?
10-19-2012 06:09 AM
another thing, i take it restarting the Admin service didn't resolve the issue either?
You haven't limited the amount of sql connections that could be made by EV have you?
10-19-2012 07:01 AM
Hi Please give your recommendation what would be the problemmm....for intial troubleshooting to update my customer.....
10-19-2012 07:44 AM
Well Gertjan started you off nicely i think, check the SQL Servers for any intermittment communication issues or cluster issues, so it could be that the SQL Servers are experiencing a "blip" in communications , and EV just isn't handling the disconnect well.
So check with the SQL Server, look at the event logs and any other sql logs that may be showing it disconnecting from the network, or any long procedures that may be disconnecting the EV Services
Also check out the technote he stated: http://www.symantec.com/docs/TECH66826
And from my side I asked whether restarting the admin service would have resolved the issue (without having to resort to failing over SQL Server) and whether you have limited any connections on the SQL Server itself, could it be that EV is exhausting the amount of connections given to it?
Also when this happens, do you see anything abnormal on the SQL Servers? 100% CPU usage, higher than normal disk usage etc
If a restart of the EV Services does not work but failing the SQL Server *does* work, then you would have to assume its an issue with the SQL Service or the node at that particular time.
Also look to see if anything has changed in the environment
have you recently updated EV? updated SQL? changed Storage for the SQL Databases?
Enabled a new bunch of users? started any big PST Migrations? Vault Cache builds etc?
If it becomes a much larger issue then your best bet is to get DTraces of the directoryService on the EV Server, get the application, system and Enterprise Vault logs from the Event Viewer, and also Application and System logs from the SQL Server as well as any other Error logs and a snapshot of the activity monitor...then open a case with symantec to help troubleshoot the issue