Netbackup MSDP PostgreSQL service fails to start

[rsakimoto]

Hi,

Need help. PostgreSQL service stopped. When trying to start it says "Windows could not start the PostgreSQL Server 8.3 on Local Computer. Error 1053: The service did not respond to the start or control request in a timely fashion". Will this Article URL http://www.symantec.com/docs/TECH170440 solve it or the article pertains to a different scenario. If that article is a general solution, I will just execute this from the article solution - C:\Program Files\Veritas\pdde\pddb\bin>pg_resetxlog.exe -f x:\msdp\databases\pddb\data
Below postgresql logs from msdp path\log\pddb:

2014-08-27 17:44:09 MYT LOG:  database system was interrupted while in recovery at 2014-08-27 17:40:09 MYT
2014-08-27 17:44:09 MYT HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2014-08-27 17:44:09 MYT LOG:  database system was not properly shut down; automatic recovery in progress
2014-08-27 17:44:09 MYT LOG:  redo starts at 1112/40012348
2014-08-27 17:44:09 MYT FATAL:  the database system is starting up
2014-08-27 17:44:09 MYT LOG:  unrecognized win32 error code: 1106
2014-08-27 17:44:09 MYT CONTEXT:  writing block 164090 of relation 1663/16384/307404363
 xlog redo insert(init): rel 1663/16384/307404370; tid 58796/1
2014-08-27 17:44:09 MYT FATAL:  could not write block 164090 of relation 1663/16384/307404363: Invalid argument
2014-08-27 17:44:09 MYT CONTEXT:  writing block 164090 of relation 1663/16384/307404363
 xlog redo insert(init): rel 1663/16384/307404370; tid 58796/1
2014-08-27 17:44:09 MYT LOG:  startup process (PID 6404) exited with exit code 1
2014-08-27 17:44:09 MYT LOG:  aborting startup due to startup process failure
 

Netbackup version is 7.5.0.6.

Regards

[watsons]

I don't thnk that technote helps with your case - the message here looks like a data corruption:

This probably means that some data is corrupted and you will have to use the last backup for recovery.

Time to call up Netbackup support and have them provide you with crchk tool to run through the MSDP, and recover what is corrupted. And if your environment permits, once this is fixed upgrade to NB7.6 which will get rid of the usage of PostgreSQL service for deduplication.

[Mark_Solutions]

A few questions:

1. How is your MSDP drive for disk space

2. Is your MSDP drive fully excluded for all anti virus systems

3. What happened (if anything) that caused this .. system crash, reboot etc..

It does sound  like a support call to check how bad it is but best to also isolate what caused it anyway.

[rsakimoto]

Ok @ watsons. Already created ticket to support. Thanks.

[rsakimoto]

Hi MarK_Solutions,  MSDP space is a bit low. 1TB+ out of 9TB. Yes MSDP drive should be excluded to our antivirus since our antivirus is Symantec product as well. For some reason it just stop working. A regular maintenance reboot of server was the last activity done. Already created case to support. Thanks.

[Mark_Solutions]

OK - but just because you use End Point Protection (or similar) doesn't mean it will have any exclusions .. it doesn't auto detect other Symantec products!!

AV can be very damaging so do double check.

It sounds like the server had a reboot while it was in the middle of processing some transactions so hopefully support can do you a fix (usually just needs the odd file deleting and a re-start).

It is best practice to shut down NetBackup cleanly before re-booting a de-dupe server .. just for future reference..

[rsakimoto]

Hi Mark,

Symantec Support is already working on the case, they also advised to exclude msdp directory from local AV installed which I already did and ran some other windows patches. Based on the logs I have gathered them, netbackup .bn files were corrupted (some). Possible cause, they are pointing is AV. Though they have not exactly pin pointed it. Support advised to run recoverCR tool, but always hangs on Step 8 of the tool. They don't see any error too why the recoverCR tool hangs on Step 8 (Create new Crdb). Now they are recommending to re-create the msdp of the affected media server. Which yes I woul lose data. Thats the last option and recommendation they advised to do if (most likely) the corrupted .bn files cannot be recovered. So recoverCR also same as crchk tool didnt resolved my issue.

[rsakimoto · Accepted solution]

The only solution to this as per Symantec Support recommends is to re-create msdp of the affected media server. Decomission and recomission msdp. If the corrupted files cannot be recovered. Following technote http://www.symantec.com/docs/TECH150431 . To eliminate postgresql service is to upgrade to 7.6 version as what Watsons also have said.