Opscenter post upgrade wont start!

jim_dalton · ‎01-20-2014

Posting here as its sqlanywhere really and more traffic here:

POST UPGRADE LOG:

I. 01/16 14:50:56. Starting database "vxpmdb" (/opt/SYMCOpsCenterServer/db/data/vxpmdb.db) at Thu Jan 16 2014 14:50
I. 01/16 14:50:58. Opening dbspace 'symcOpscache' in file 'symcOpscache.db' for database 'vxpmdb'
I. 01/16 14:50:58. Opening dbspace 'symcopsscratchdb' in file 'symcopsscratchdb.db' for database 'vxpmdb'
I. 01/16 14:50:58. Opening dbspace 'symcsearchdb' in file 'symcsearchdb.db' for database 'vxpmdb'
I. 01/16 14:51:00. Transaction log: /opt/SYMCOpsCenterServer/db/data/vxpmdb.log
E. 01/16 14:51:00. Error: Cannot open transaction log file -- Can't use log file "/opt/SYMCOpsCenterServer/db/data/vxpmdb.log" since the database f
ile has been used more recently
E. 01/16 14:51:00. Cannot open transaction log file -- Can't use log file "/opt/SYMCOpsCenterServer/db/data/vxpmdb.log" since the database file has
been used more recently
I. 01/16 14:51:00. Database server shutdown due to startup error
I. 01/16 14:51:00. Database server stopped at Thu Jan 16 2014 14:51

ME DOING A RESTART TODAY LOG:

I. 01/20 13:04:45. SQL Anywhere Network Server Version 12.0.1.3873
I. 01/20 13:04:45. OEM Authenticated Edition, licensed only for use with authenticated OEM applications.
I. 01/20 13:04:45.
I. 01/20 13:04:45. Copyright (c) 2001-2013, iAnywhere Solutions, Inc.
I. 01/20 13:04:45. Portions copyright (c) 1988-2013, Sybase, Inc. All rights reserved.
I. 01/20 13:04:45. Use of this software is governed by the Sybase License Agreement.
I. 01/20 13:04:45. Refer to http://www.sybase.com/softwarelicenses.
I. 01/20 13:04:45.
I. 01/20 13:04:45. Processors detected: 1
I. 01/20 13:04:45. Processor limit (licensed processors): 128
I. 01/20 13:04:45. Maximum number of physical processors the server will use: 1
I. 01/20 13:04:45. This server is licensed to:
I. 01/20 13:04:45. NetBackup
I. 01/20 13:04:45. Symantec Corporation
I. 01/20 13:04:45. Running Linux 2.6.18-194.el5 #1 SMP Tue Mar 16 21:52:39 EDT 2010 on X86_64
I. 01/20 13:04:45. Server built for X86_64 processor architecture
I. 01/20 13:04:45. 25680K of memory used for caching
I. 01/20 13:04:45. Minimum cache size: 25680K, maximum cache size: 512000K
I. 01/20 13:04:45. Using a maximum page size of 8192 bytes
I. 01/20 13:04:45. Multiprogramming level: minimum:1, current:20, maximum:80
I. 01/20 13:04:45. Automatic tuning of multiprogramming level is enabled
I. 01/20 13:04:46. Starting database "vxpmdb" (/opt/SYMCOpsCenterServer/db/data/vxpmdb.db) at Mon Jan 20 2014 13:04
I. 01/20 13:04:48. Opening dbspace 'symcOpscache' in file 'symcOpscache.db' for database 'vxpmdb'
I. 01/20 13:04:48. Opening dbspace 'symcopsscratchdb' in file 'symcopsscratchdb.db' for database 'vxpmdb'
I. 01/20 13:04:48. Opening dbspace 'symcsearchdb' in file 'symcsearchdb.db' for database 'vxpmdb'
I. 01/20 13:04:50. Transaction log: /opt/SYMCOpsCenterServer/db/data/vxpmdb.log
E. 01/20 13:04:50. Error: Cannot open transaction log file -- Can't use log file "/opt/SYMCOpsCenterServer/db/data/vxpmdb.log" since the database f
ile has been used more recently
E. 01/20 13:04:50. Cannot open transaction log file -- Can't use log file "/opt/SYMCOpsCenterServer/db/data/vxpmdb.log" since the database file has
been used more recently
I. 01/20 13:04:50. Database server shutdown due to startup error
I. 01/20 13:04:50. Database server stopped at Mon Jan 20 2014 13:04

So the upgrade screwed the db. Nice work.

-r--r--r-- 1 root root 1410654208 Jan 20 13:04 /opt/SYMCOpsCenterServer/db/data/vxpmdb.db
[root@server01 bin]# !1217
ls -l /opt/SYMCOpsCenterServer/db/data/vxpmdb.log
-rw------- 1 root root 8192 Jan 16 14:49 /opt/SYMCOpsCenterServer/db/data/vxpmdb.log

How come the db has a timestamp of today? Seems the startup logic is screwed if the db timestamp can be applied without correct startup.

The only oddity of the upgrade was the script fails to check the space available (in /var) so when it decompresses certain data it can fill up the filesys but I worked around thsi via a symlink to /opt.The upgrade didnt start when this happened as it has the sense to check the prerequisites all worked before touching the db.

I have not yet attempted any bodge, like modifying the timestamp.

So I'm looking for a steer to fix if anyone has one.

Thanks in advance, forum members,Jim.

Mark_Solutions · ‎01-20-2014

You could try following the process for when you have a corrupted database http://www.symantec.com/docs/TECH147217

jim_dalton · ‎01-20-2014

Its an idea Mark but a quick read suggests it wont work:-

The first command is a defrag and I cant defrrag a db that wont start (as it says...Stopping OpsCenter database, which thus must mean its running in the first place, and it wont).

I did try it: yuk.

Stopping Symantec NetBackup OpsCenter Server (15) ...
SQL Anywhere Stop Server Utility Version 12.0.1.3873
Unable to stop database server -- Database server not found.
Unloading the vxpmdb database to /opt/SYMCOpsCenterServer/db/data/defrag/vxpmdb
Error unloading vxpmdb
Error Unloading vxpmdb database
SQL Anywhere Start Server In Background Utility Version 12.0.1.3873
DBSPAWN ERROR: -935
Unable to start specified database: Log file error
Unable to start Symantec NetBackup OpsCenter Database Server
Starting Symantec OpsCenter Server...
Symantec OpsCenter Server started, pid = 18236
Symantec OpsCenter WebServer Service started

Thanks,Jim

Mark_Solutions · ‎01-20-2014

It rather sounds like this issue - but i am guessing you are at 7.5 or 7.6? http://www.symantec.com/business/support/index?page=content&pmv=print&impressions=&viewlocale=&id=TE...

Not sure of this will help or if it is too late?

http://www.symantec.com/docs/TECH144683

jim_dalton · ‎01-20-2014

That Mark could offer a solution indirectly as I did take the precaution of a backup before I started.

I went from 7504 to 7601. The db is about 1.4G, dont think the ten minute rule applies/applied however one things for sure it is corrupted. Lets hope the backup isnt...and if I can upgrade manually then it should be good.

Thanks for the pointer.

Jim

jim_dalton · ‎01-20-2014

And I should add that the 10 minute timeout also exists in the 7601 upgrade perl. Oh dear.

Jim

jim_dalton · ‎01-20-2014

I tried to upgrade from the backed up version: it failed. I'm really loving the product.

Executing the query with id: 1, SQL: IF (isPrimaryKeyExists('PK_APP_DISK','app_Disk') = 1 AND isColumnExists('enclosureId','app_Disk') = 1)
                        THEN
                        ALTER TABLE DBA.app_Disk
                        DROP CONSTRAINT PK_APP_DISK;

ALTER TABLE DBA.app_Disk
MODIFY "enclosureId" INTEGER NOT NULL;

                        ALTER TABLE DBA.app_Disk
                        ADD CONSTRAINT PK_APP_DISK
                        PRIMARY KEY (masterServerId,hostName,diskId,enclosureId);
                        END IF;
JZ006: Caught IOException: com.sybase.jdbc3.jdbc.SybConnectionDeadException: JZ0C0: Connection is already closed.
java.sql.SQLException: JZ006: Caught IOException: com.sybase.jdbc3.jdbc.SybConnectionDeadException: JZ0C0: Connection is already closed.
        at com.sybase.jdbc3.jdbc.ErrorMessage.createIOEKilledConnEx(Unknown Source)
        at com.sybase.jdbc3.jdbc.ErrorMessage.raiseErrorCheckDead(Unknown Source)
        at com.sybase.jdbc3.tds.Tds.a(Unknown Source)
        at com.sybase.jdbc3.tds.Tds.a(Unknown Source)
        at com.sybase.jdbc3.tds.Tds.cancel(Unknown Source)
        at com.sybase.jdbc3.tds.Tds.cancel(Unknown Source)
        at com.sybase.jdbc3.jdbc.SybStatement.a(Unknown Source)

blah blah blah.

ERROR: Failed to Install database. Exit Code is : 1

This is looking more and more like a support call.

Will_Restore · ‎01-20-2014

I tried to upgrade OpsCenter at several sites and it failed something like 5 of 8 attempts.

Thankfully the NetBackup upgrades went a lot smoother!

Mark_Solutions · ‎01-21-2014

I dont really know any other way - if you took a backup initially then it would be worth rolling back to 7.5.0.4 (or uninstalling and re-installing to get to 7.5.0.4) and then do a database import from your original database:

http://www.symantec.com/docs/HOWTO42874

#Edit#

and then set the timeout values and try the upgrade again

jim_dalton · ‎01-22-2014

Opscentre failed 5/8 times WR...thats terrible!

I have a call logged...

VOX

Opscenter post upgrade wont start!