FSA agent upgrade to EV 10.0.2

[AndresMunoz]

Hi there,

I've upgraded a customer site to EV 10.0.2 (from 9.0.2), all upgrade has been successful, with exception of the FSA agent on two file clusters

Cluster configuration:

• OS: Windows 2008 R2
• Windows Failover cluster
• Cluster configuration: active-passive (2 nodes)

I've tried upgrading from both the VAC and locally from the nodes following the upgrade instructions.

By analysing the installation logs (attached) I get the following

MSI (s) (2C:A0) [21:48:04:047]: Hello, I'm your 32bit Elevated custom action server.
MSI (s) (2C!C0) [21:48:04:188]: Creating MSIHANDLE (494) of type 790531 for thread 8128
MSI (s) (2C!C0) [21:48:04:188]: Closing MSIHANDLE (494) of type 790531 for thread 8128
MSI (s) (2C!C0) [21:48:04:188]: Creating MSIHANDLE (495) of type 790531 for thread 8128
1: Exported Function Begin: PatchExistingProducts
MSI (s) (2C!C0) [21:48:04:188]: Closing MSIHANDLE (495) of type 790531 for thread 8128
MSI (s) (2C!C0) [21:48:04:188]: Creating MSIHANDLE (496) of type 790531 for thread 8128
1: C:\Users\vsACCOUNT\AppData\Local\Temp\{7DDC1AA4-2B64-4500-BF9F-3359D8B0CEE7}\RunSQL.exe
MSI (s) (2C!C0) [21:48:04:188]: Closing MSIHANDLE (496) of type 790531 for thread 8128
MSI (s) (2C!C0) [21:48:04:188]: Creating MSIHANDLE (497) of type 790531 for thread 8128
1: C:\Windows\Installer\a9ff59b.msi
MSI (s) (2C!C0) [21:48:04:188]: Closing MSIHANDLE (497) of type 790531 for thread 8128
MSI (s) (2C!C0) [21:48:04:188]: Creating MSIHANDLE (498) of type 790531 for thread 8128
1: C:\Users\vsACCOUNT\AppData\Local\Temp\{7DDC1AA4-2B64-4500-BF9F-3359D8B0CEE7}\{77F5B39C-D845-49CF-95E1-57BCB1CB6B7D}.sql
MSI (s) (2C!C0) [21:48:04:204]: Closing MSIHANDLE (498) of type 790531 for thread 8128
MSI (s) (2C!C0) [21:48:04:204]: Creating MSIHANDLE (499) of type 790531 for thread 8128
MSI (s) (2C!C0) [21:48:04:204]: Closing MSIHANDLE (499) of type 790531 for thread 8128
MSI (s) (2C!C0) [21:48:04:579]: Creating MSIHANDLE (500) of type 790531 for thread 8128
1: See $windir\temp\runsql.log for details.
MSI (s) (2C!C0) [21:48:04:579]: Closing MSIHANDLE (500) of type 790531 for thread 8128
MSI (s) (2C!C0) [21:48:04:579]: Creating MSIHANDLE (501) of type 790531 for thread 8128
1: Execution of CommandLine failed (exit code : 1)
MSI (s) (2C!C0) [21:48:04:579]: Product: Symantec Enterprise Vault - FSA Agent (x64) 10.0.2.1112 -- Error 25023.Error patching existing products. Please refer to the logs for more details.

MSI (s) (2C!C0) [21:48:04:579]: Closing MSIHANDLE (501) of type 790531 for thread 8128
Error 25023.Error patching existing products. Please refer to the logs for more details.
MSI (s) (2C:6C) [21:48:04:594]: Closing MSIHANDLE (493) of type 790536 for thread 8384
CustomAction PatchExistingProducts returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)
MSI (s) (2C:C0) [21:48:04:610]: User policy value 'DisableRollback' is 0
MSI (s) (2C:C0) [21:48:04:610]: Machine policy value 'DisableRollback' is 0
Action ended 21:48:04: InstallFinalize. Return value 3.

RunSQL.log indicates

Starting up

Cached Package: 'C:\Windows\Installer\a9ff59b.msi'
Script: 'C:\Users\vsACCOUNT\AppData\Local\Temp\{7DDC1AA4-2B64-4500-BF9F-3359D8B0CEE7}\{77F5B39C-D845-49CF-95E1-57BCB1CB6B7D}.sql'
Executing statement: "DELETE FROM `Dialog` WHERE `Dialog`='FilesInUse'"
Executing statement: "INSERT INTO `Property`(`Property`, `Value`) VALUES ('MSIRESTARTMANAGERCONTROL','Disable')"

Success.

Starting up

Cached Package: 'C:\Windows\Installer\a9ff59b.msi'
Script: 'C:\Users\vsACCOUNT\AppData\Local\Temp\{7DDC1AA4-2B64-4500-BF9F-3359D8B0CEE7}\{77F5B39C-D845-49CF-95E1-57BCB1CB6B7D}.sql'
Executing statement: "DELETE FROM `Dialog` WHERE `Dialog`='FilesInUse'"
Executing statement: "INSERT INTO `Property`(`Property`, `Value`) VALUES ('MSIRESTARTMANAGERCONTROL','Disable')"
 Unable to execute statement 'INSERT INTO `Property`(`Property`, `Value`) VALUES ('MSIRESTARTMANAGERCONTROL','Disable')'. Error 1627

Failure.

Starting up

Cached Package: 'C:\Windows\Installer\a9ff59b.msi'
Script: 'C:\Users\vsACCOUNT\AppData\Local\Temp\{7DDC1AA4-2B64-4500-BF9F-3359D8B0CEE7}\{77F5B39C-D845-49CF-95E1-57BCB1CB6B7D}.sql'
Executing statement: "DELETE FROM `Dialog` WHERE `Dialog`='FilesInUse'"
Executing statement: "INSERT INTO `Property`(`Property`, `Value`) VALUES ('MSIRESTARTMANAGERCONTROL','Disable')"
 Unable to execute statement 'INSERT INTO `Property`(`Property`, `Value`) VALUES ('MSIRESTARTMANAGERCONTROL','Disable')'. Error 1627

Failure.

I think that seems to be what is killing the agent upgrade. But I cannot figure out what is causing this.

I have also noticed that the cached installer (C:\Windows\Installer\a9ff59b.msi) is version 9.0.2.1061 (???)

Any ideas/suggestions, or a way forward would be greatly appreciated.

[JesusWept3]

Best idea I can think of uninstall/reinstall

[AndresMunoz]

JesusWept3, I think you are on the money there. I had already seek approval from the customer to try this over the weekend. I'll let you know how this goes. If anyone has any other sugestions, I'm all eyes ;)

[AndresMunoz · Accepted solution]

OK... I have solved this. Uninstall didn't work.. as I wasnt even able to uninstall it!!!! So after trying several things, I decided to use my admin account instead of VSA to upgrade the software. The VSA has local admin rights, and was the account used initially to install the software, so I had no reasons to suspect the account (which was not the problem) Before continuing, I had verified and checked evey possible symantec article, including this technote http://www.symantec.com/business/support/index?page=content&id=TECH179712 but the signature certificate indicated all was ok, and I was not getting the signature error described in the article. After I had run out of articles and ideas, only thing to do was to use a more privilegied account (mine) Once i used my account, the upgrade failed again, but the error on the logs DID match those described on the technote... namely Error code 0x800B0109: A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider.) DIFXAPP: INFO: Successfully removed {9BBEE86F-666A-4735-A1CF-2847669D3CEE} from reference list of driver store entry DIFXAPP: INFO: RETURN: DriverPackageInstallW (0x800B0100) DIFXAPP: ERROR encountered while installing driver package C:\Program Files\Enterprise Vault\drivers\evmf.inf DIFXAPP: InstallDriverPackages failed with error 0x800B0100 however, when verifying the package signature, the error on the certificate, unlike the technote, wasn't "the certificate in the signature cannot be verified", but "Digital signature information: One of the countersignatures is not valid. The file may have been altered." The countersignature certificate in question was the Symantec Time Stamping Services Signer - G3 certificate, yet the certificate chain was ok, BUT, on checking the CAPI2 event log on the fsa target, it indicated that could not verify the CRL of the certificate... BINGO!.. As these are government systems, there was no internet access alloed from the server. So I manually downloaded the CRL for both the time stamp certificate and the root ca for the signature certificate and voila! upgrade succeeded. I applied the CRL on all other 2 nodes and they also succeeded. Hope this helps anyone on the same predicament.

[StephenConnolly]

This issue with digital signatures and timestamping in FSA is problematic. Windows Drivers have to be properly signed and timestamped and any subtle issue like this will cause an installation problem. When you get problems like this, you have to check the signatures *on the machine where you have the problem*. The problem is typically not with the digital signatures, but with the system on which the signature is getting validated. We've seen this on systems where Windows Update is disabled and there is no internet access, which prevents Windows from getting Root Certificate updates automatically. If the signature depends on a root cert that is not installed locally, then the verification will fail as described above. There are TWO signatures in our digital signatures. The first is the main signature which the obvious one, but there is a second signature which is attached to the timestamp - this is the countersignature referred to in the error above. I suspect that on the target machine the root certs have not been updated recently, so that the cert chain for the signature on the timestamp is not valid. The timestamp guarantees the validity of the main signature in perpetuity, even after the certificate used for the signature has expired.

[AndresMunoz]

"The problem is typically not with the digital signatures, but with the system on which the signature is getting validated" Correct. This was exactly the problem, and yes Windows Updates was disabled with no internet access, as all the patching is done through SCCM. Since then,I've found out that the server team is also having issues with SEP updates. So i have pointed on the same direction as this issue.