cancel
Showing results for 
Search instead for 
Did you mean: 

EV 11.0.1 trying to connect to two sql servers after DR failover

nwalsh
Level 4

Hi all,

Wondering if someone could help me or if had similar issues.

i did a DR failover test from our production to our DR site.

1.eDiscovery

2. Mail Archive

3. Journal archive

our EV's are on vmware as too our SQL servers.

our Dba did a db mirror of the databases from our production sql server (SQL01) over to our DR sql server (SQL05)

the only change i applied with a sql alias change from sql01 to sql05

After failing back to production. i removed the sql alias from my mail archive server, so expected it would talk about to my production sql server SQL01

my Mail Archive server is trying to connect back to our Dr sql server. ive checked on the registry and did a dns flush. i had tech support on and they dont know how this is possible.

Due to the connectivity issues connecting to the sql server.

my Enterprise Vault services keeps shutting down. 

 

below are one of the examples of the error im getting.

Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault 
Date:          28/01/2016 15:03:07
Event ID:      6479
Task Category: Storage Crawler
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EUR0NWENTMASP01.domain.com
Description:
Error accessing Vault Store database.
 [Internal reference CStorageDBAccess::OpenDatabase()/AddConnection failed]
Database DSN: [EVVSMailboxVaultStore1_1] 

================================================================================================================

Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault 
Date:          28/01/2016 14:47:14
Event ID:      40966
Task Category: (132)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EUR0NWENTMASP01.domain.com
Description:
A program fault has raised an exception.

Exception: Cannot open database "EnterpriseVaultDirectory" requested by the login. The login failed.
Login failed for user Domain/'EURSVCEVAdmin'.
Diagnostic: Num:4060 State:1 Sev:11 Svr:EURSQL05\INFRA03 Proc: Line:65536
Type: System.Data.SqlClient.SqlException
Reference: DL/Archive/GetEntry/1AEB92A2AC3DE684CB520B1DA9119884B1110000ENTMASP01

 

any help would be great.

thanks,

Noel


   

1 ACCEPTED SOLUTION

Accepted Solutions

nwalsh
Level 4

Howdi All,

 

Just an update on this issue if you come across it and could not find the reason. The reason was down to the windows firewall port were only configured for 300 open ports. 

 

After having a 3rd party veritas vender in. the found the fault to do with there was not enough ports configured 

 

I found a tool also to use it may help called EV_Best_Practice_Analyzer_V2.0.7.exe its a very handy tool.

 

From the report it stated below.

DYNAMIC USER PORTS
Ports should start from 1500 and have 63000 available ports

Use the below command from cmd line or Powershell to 'correct' the Dynamic TCP User Port Settings:-
netsh int ipv4 set dynamicport tcp start=1500 num=63000

 

hope this helps ..

Thanks to everyone for all their help and input

Regards,

Noel

 

View solution in original post

10 REPLIES 10

JimmyNeutron
Level 6
Partner Accredited

Try looking in the ODBC.exe or ODBCAD32.exe for connections to the SQL servers. Also look into the update service locations in EV.

 

Update Service Location article

https://www.veritas.com/support/en_US/article.TECH215320

 

ODBC article

https://www.veritas.com/support/en_US/article.TECH66481

nwalsh
Level 4

Hi Jimmy,

 

i had check the odbc connections and they were confirmed connecting to sql01

 

i applied the update service location and i got he following results.

 

Update service locations failed

 

Admin Service on EUR0NWENTJASP01: OK

Admin Service on EUR0NWENTMASP01: OK

Admin Service on EUR0NWENTDASP01: OK

Admin Service on EUR0NWENTMASP01: OK

The site has 3 servers

Checking: Server [EUR0NWENTDASP01]

                Alias 1: EUR0NWENTDASP01.advisors.hal

                Enterprise Vault Indexing Service: OK

                Enterprise Vault Shopping Service: OK

                Enterprise Vault Directory Service: OK

                Enterprise Vault Task Controller Service: OK

 

Server cache is not configured for computer 'EUR0NWENTMASP01'

 

ive googled in to check this and i dont seem to have an answer. all help is greatly appreachated. 

here is my event viewer looking.

event archive.png

thanks

Noel

JimmyNeutron
Level 6
Partner Accredited

Put the sql01 IPs,Alises in the EV server host file and reboot the server. 

nwalsh
Level 4

Hi Jimmy,

 

yeh i added that in and rebooted the server and still the same issue.

GertjanA
Moderator
Moderator
Partner    VIP    Accredited Certified

Hi NW,

Are the databases on the target server Read Write? You mention something about DBA creating a mirror, did he revert the mirror?

If possible, logon to the SQL server with the VaultServiceAccount, and see if you can open SQL Management Studio. Then check to see if you see the databases, and what state they are in.

Regards. Gertjan

nwalsh
Level 4

Hi Gertjan,

Sorry in the late reply.

 

Yes the databases are RW and the DBA did mirror over the db and reverted back to the production server.

 

Yes i am able to RDP onto the sql server and able to access the databases using sql mangement studio.

Ive checked the connections with my other 2 ev servers and they are able to connect to the sql db with no problems or errors.

 

the strange thing is that the services stop as they cant make the connection. or it is overloaded. while the EV services are stopped. i am able to make an odbc connection to the directory and MailboxVaultStore. When i start the services again and then test. it failes and i get the following error message from event viewer

========================================================================================================================

A program fault has raised an exception.

Exception: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 0 - An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.)
Diagnostic: Num:10055 State:0 Sev:20 Svr: Proc: Line:0
Type: System.Data.SqlClient.SqlException
Reference: DL/Archive/GetEntry/1B1EF54BF39AF724AB731875D3206C7CB1110000ENTMASP01

Command Line: "e:\Program Files (x86)\Enterprise Vault\EVIndexVolumesProcessor.exe" -URL:tcp://x.x.x.197:49311/8ad99778_bca4_469c_a981_ce76afcb374b/KVS.EnterpriseVault.ProcessManager62954519 
Application Domain: EVIndexVolumesProcessor.exe
Process Id: 6540
Thread Id: 10968

 

========================================================================================================================

Then followed by

 

A program fault has raised an exception.

Exception: An attempt was made to access a socket in a way forbidden by its access permissions
Diagnostic: Error Code is 10013
Type: System.Net.Sockets.SocketException
Reference: MonitoringMiddleTier - MiddleTierDataManagerDriver::Main()

Command Line: "e:\Program Files (x86)\Enterprise Vault\MonitoringMiddleTier.exe" 9632 
Application Domain: MonitoringMiddleTier.exe
Process Id: 6452
Thread Id: 4068
Stack Trace:    at Symantec.EnterpriseVault.OperationsManager.MiddleTierDataAccessLayer.MiddleTierDataManagerDriver.ConfigureRemoting()
   at Symantec.EnterpriseVault.OperationsManager.MiddleTierDataAccessLayer.MiddleTierDataManagerDriver.Main(String[] args) 

========================================================================================================================

A program fault has raised an exception.

Exception: Could not connect to SQL database:'EVVSMailboxVaultStore1_1' at SQL Server:'EURSQL01\AMSDOC'
Diagnostic: 
Type: System.Exception
Reference: 

Command Line: "e:\Program Files (x86)\Enterprise Vault\EVStorageQueueBroker.exe" -URL:tcp://x.x.x.197:49291/4547eca7_db45_4faf_b8f0_330d5379497c/KVS.EnterpriseVault.ProcessManager33736294 
Application Domain: EVStorageQueueBroker.exe

========================================================================================================================

A periodic Index Admin check could not determine the status of one or more Index Volumes that are managed by this server. (Error: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 0 - An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.)) 

========================================================================================================================

i did also have an error that there are too many connections going to the sql server from this one. is there away to reduce the amount of connections from EV to SQL server?

 

I have an average between 476 to 268 attempted connections to the sql server and i believe the sql server cant handle them?

 

someone suggested in reinstalling the binarys and it should clear up the miss confusion? Currently i only have EV11.01.1 CH1 installed.

 

GertjanA
Moderator
Moderator
Partner    VIP    Accredited Certified

Hi Noel,

The EV server needs to talk to this SQLserver/Instance right? 'EURSQL01\AMSDOC'

If so, my first attempt would be to reboot the whole environment. Set backup mode on stores and indexes, then stop the EV-services on all EV servers, reboot SQL server. Wait for that to start fully.

When rebooted succesfully, reboot the EV servers one by one. Verify connectivity is properly established.

when ok, clear backup mode.

If that fails again for that one server, a binary re-install can be your second attempt. Install 11.0.1 and the CHF used, then reboot again.

Regards. Gertjan

nwalsh
Level 4

Hi Gertjan,

 

Yes you are correct EV needs to talk to the SQLserver/Instance.

 

I will attempt as you mentioned to set the ev in backup mode and get the sql server rebooted. only catch i have with the sql server its a live server that has other db connected so it might have to be done out of hrs. i wil have to check with our local sql dba. but i will attempt this and come back with the results.

 

And lastly if it doesnt work on the reboot ill try the reinstall of the binary and hopefully it will work.

 

Thanks for the input. ill come back when i have completed.

 

Thanks,

Noel

nwalsh
Level 4

Hi Gertjan,

 

Tried as you mentioned.. still the same issue event after the reinstall.

Also got an new error saying the following:

the application-specific permission settings do not grant Local Launch permission for the COM Server application with CLSID 
{7DB927A8-A2F9-4B72-B10B-359353D3A39F}
 and APPID 
{8FF8501E-1317-4386-AE86-B27388C26EAE}
 to the user Domain\EURSVCEVAdmin SID (S-1-5-21-367615377-828005543-9522986-83630) from address LocalHost (Using LRPC) running in the application container Unavailable SID (Unavailable). This security permission can be modified using the Component Services administrative tool.

 

I then checked the following technote and confirmed the firewall ports are open on both TCP and UDP

https://www.veritas.com/support/en_US/article.TECH87314

nwalsh
Level 4

Howdi All,

 

Just an update on this issue if you come across it and could not find the reason. The reason was down to the windows firewall port were only configured for 300 open ports. 

 

After having a 3rd party veritas vender in. the found the fault to do with there was not enough ports configured 

 

I found a tool also to use it may help called EV_Best_Practice_Analyzer_V2.0.7.exe its a very handy tool.

 

From the report it stated below.

DYNAMIC USER PORTS
Ports should start from 1500 and have 63000 available ports

Use the below command from cmd line or Powershell to 'correct' the Dynamic TCP User Port Settings:-
netsh int ipv4 set dynamicport tcp start=1500 num=63000

 

hope this helps ..

Thanks to everyone for all their help and input

Regards,

Noel