Cannot open transaction log file -- Can't use log file "/NBAZDB.log" since

[Raaavan]

Hi,

We are not aware how this NBAZDB got placed in root location but here i
can give you background of this issue

- hbdakcmstn2 were the active node , while switch over on passive node i.e
hbdakcmstn1 we encountered issue with below error message since NBU
service failed to start on hbdakcmstn1

Cannot open transaction log file -- Can't use log file "/NBAZDB.log" since
the database file has been used more recently

- Here the netbackup is searching for the transaction log under
"/NBAZDB.log instead of /nbucat/db/data/NBAZDB.log.
We dont know how it redirected towards "/ " instead of it's actual path
"VXDBMS_NB_TLOG = /nbucat/db/data"

- As temporary solution we copied same file from hbchanmstn2 to
hbchanmstn1 under root mount point & started NBU services successfully
(Note - NBAC is not implemented )

- Requestto provide proper guide line & steps to move this files to
it's actual location.
we need to fix this issue on priority
multiple activity lined up on master server which need frequent switch
over between nods.

Environment details -

Virtual IP - 10.226.7.187 dakcmstnetbkp

Node -
10.226.7.197 hbdakcmstn1 -- active node
10.226.7.198 hbdakcmstn2

NBU version - 7.5.0.4
OS - Solaris 10

VCS cluster version 6.0.1

[hbdakcmstn1]# cat vxdbms.conf
VXDBMS_NB_SERVER = NB_dakcmstnetbkp
VXDBMS_NB_REMOTE_SERVER = NB_dakcmstnetbkp
VXDBMS_NB_PORT = 13785
VXDBMS_NB_DATABASE = NBDB
VXDBMS_AZ_DATABASE = NBAZDB
VXDBMS_NB_DATA = /nbucat/db/data
VXDBMS_NB_INDEX = /nbucat/db/data
VXDBMS_NB_TLOG = /nbucat/db/data
VXDBMS_NB_MLOG = /nbucat/db/data
VXDBMS_NB_STAGING = /nbucat/db/staging
VXDBMS_NB_PASSWORD = f11686d76ace21182cbc1beced60026b9ac338cb4fe01507
AZ_DB_PASSWORD = Jj8mkP3sKTo=
VXDBMS_NB_FULL_KEYWORD = NBDB:277221:1384223400:F
VXDBMS_NB_INCREMENTAL = NBDB.log.1
VXDBMS_AZ_INCREMENTAL = NBAZDB.log.1
VXDBMS_BACKUP_POLICY = Hot_Catalog_Backup
VXDBMS_BACKUP_SCHEDULE_TYPE = 0
 

/nbucat/ is shared between both nodes

[Nicolai]

Usually the shared disk is mount under /opt/VRTSnbu in a VCS cluster. /nbucat look like a custom configuration. Has the functionality of the cluster been verified before ?

Can you post /etc/VRTSvcs/conf/config/main.cf ?

[Yasuhisa_Ishika]

when did you get this error? If you got this while manually starting NetBackup on inactive node of cluster or system bootup time, this is what bring you this error.

As shared disk(s) and NetBackup services are under control of cluster software, you should(hmm, have to) start NetBackup though cluster software.

For bootup time issue, I usually add some trick in rc script. Default rc script always try to start server daemons - this startup is out of cluster control. It is better to add some lines into rv script so that startup of server daemons is disabled. it is easy to determine if the node is clusterd or not - check NBU_RSP is placed under /usr/openv/netbackup/bin/cluster.

[Raaavan]

I have attached main.cf 

I would like to make some note here 

 "/nbucat" is netbackup catalog mount point & will switch with node ..this is not exactly issue i am looking for 

 -when i attempted to do switch from active node hbdakcmstn2 (which was active from last 5-6 month ) to hbdakcmstn1

NBU services failed to start 

- in server.log below error message addressed 

Cannot open transaction log file -- Can't use log file "/NBAZDB.log" since
the database file has been used more recently ---> this is in hbdakcmstn1's server.log

- when i checked  hbdakcmstn2  there was file available in " / " path  "NBAZDB.log"

 i copied that from hbdakcmstn2 to hbdakcmstn1's " / " path  (on which i did switch & found error )

- after that services are up since NBU found recently used logged file .

it's not like that issue is on node1 it is on both the node

- what i am expecting , have any one came across such issue 

- proper TN to move this file to it's actual location in this senario 

- or any configuration changes that will make net backup to search this file to its actual location which is already mentioned in vxdbms.com 

[Nicolai]

I did not find somthing in main.cf that can explain to behavior.

[Deb_Wilmot]

Please complete:

1. Shutdown NetBackup processes.

2.  Check the /usr/openv/var/global/databases.conf file - make sure the path is correct in there for NBAZDB

3. Start Sybase (nbdbms_start_server) and emm (nbemm &)

4. Rename the NBAZDB.log file to NBAZDB.gol

5.  Run: dbeng11 -f  /nbucat/db/data/NBAZDB.db

6.  Start NetBackup.

7.  Verify that there is a new NBAZDB.log file in the /nbucat/db/data

Hopefully that will work <fingers crossed>!

Deb

[Raaavan]

we attmpted copy & rename oprtion did not worked

Server.log

I. 11/13 16:59:06. 64 physical processor(s) detected.
I. 11/13 16:59:06. Processor limit (licensed processors): 128
I. 11/13 16:59:06. Maximum number of physical processors the server will use: 64
I. 11/13 16:59:06. This server is licensed to:
I. 11/13 16:59:06.     NetBackup
I. 11/13 16:59:06.     Symantec Corporation
I. 11/13 16:59:06. Running SunOS 5.10 Generic_148888-02 on SPARC
I. 11/13 16:59:06. Server built for SPARC processor architecture
I. 11/13 16:59:06. 524288K of memory used for caching
I. 11/13 16:59:06. Minimum cache size: 524288K, maximum cache size: 2097152K
I. 11/13 16:59:06. Using a maximum page size of 4096 bytes
I. 11/13 16:59:06. Starting database "NBAZDB" (/nbucat/db/data/NBAZDB.db) at Wed Nov 13 2013 16:59
I. 11/13 16:59:06. Starting database "NBDB" (/nbucat/db/data/NBDB.db) at Wed Nov 13 2013 16:59
I. 11/13 16:59:06. Transaction log: /NBAZDB.log
I. 11/13 16:59:06. Transaction log mirror: /nbucat/db/data/NBAZDB.m.log
I. 11/13 16:59:07. Opening dbspace 'EMM_DATA' in file '/nbucat/db/data/EMM_DATA.db' for database 'NBDB'
I. 11/13 16:59:07. Opening dbspace 'EMM_INDEX' in file '/nbucat/db/data/EMM_INDEX.db' for database 'NBDB'
I. 11/13 16:59:07. Opening dbspace 'DBM_DATA' in file '/nbucat/db/data/DBM_DATA.db' for database 'NBDB'
I. 11/13 16:59:07. Opening dbspace 'DBM_INDEX' in file '/nbucat/db/data/DBM_INDEX.db' for database 'NBDB'
I. 11/13 16:59:07. Opening dbspace 'DARS_DATA' in file '/nbucat/db/data/DARS_DATA.db' for database 'NBDB'
I. 11/13 16:59:07. Opening dbspace 'DARS_INDEX' in file '/nbucat/db/data/DARS_INDEX.db' for database 'NBDB'
I. 11/13 16:59:07. Opening dbspace 'SEARCH_DATA' in file '/nbucat/db/data/SEARCH_DATA.db' for database 'NBDB'
I. 11/13 16:59:07. Opening dbspace 'SEARCH_INDEX' in file '/nbucat/db/data/SEARCH_INDEX.db' for database 'NBDB'
I. 11/13 16:59:07. Opening dbspace 'JOBD_DATA' in file '/nbucat/db/data/JOBD_DATA.db' for database 'NBDB'
I. 11/13 16:59:07. Transaction log: /nbucat/db/data/NBDB.log
I. 11/13 16:59:07. Transaction log mirror: /nbucat/db/data/NBDB.m.log
I. 11/13 16:59:07. Starting checkpoint of "NBDB" (NBDB.db) at Wed Nov 13 2013 16:59
I. 11/13 16:59:07. Finished checkpoint of "NBDB" (NBDB.db) at Wed Nov 13 2013 16:59
I. 11/13 16:59:07. Database "NBDB" (NBDB.db) started at Wed Nov 13 2013 16:59
I. 11/13 16:59:07. This database is licensed for use with:
I. 11/13 16:59:07.     Application: CommonRepository
I. 11/13 16:59:07.     Company: VERITAS
I. 11/13 16:59:07. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Wed Nov 13 2013 16:59
I. 11/13 16:59:07. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Wed Nov 13 2013 16:59
I. 11/13 16:59:07. Database "NBAZDB" (NBAZDB.db) started at Wed Nov 13 2013 16:59
I. 11/13 16:59:07. This database is licensed for use with:
I. 11/13 16:59:07.     Application: CommonRepository
I. 11/13 16:59:07.     Company: VERITAS
I. 11/13 16:59:07. Database server started at Wed Nov 13 2013 16:59
I. 11/13 16:59:07. Trying to start SharedMemory link ...
I. 11/13 16:59:07.     SharedMemory link started successfully
I. 11/13 16:59:07. Trying to start TCPIP link ...
I. 11/13 16:59:07. Starting on port 13785
I. 11/13 16:59:07.     TCPIP link started successfully
 

[Deb_Wilmot]

What is the contents of the databases.conf file?

Also - the server.conf file (in the same location???)

 /usr/openv/var/global/databases

Deb

[Raaavan]

[hbdakcmstn1]# cat databases.conf
"/nbucat/db/data/NBDB.db" -n NBDB
"/nbucat/db/data/NBAZDB.db" -n NBAZDB
 

Yes server.conf is in same location i.e /usr/openv/var/global/

[Deb_Wilmot]

What are the contents of the server.conf file?

[Raaavan]

[hbdakcmstn1]# cat server.conf
 -n NB_dakcmstnetbkp
   -x tcpip(LocalOnly=YES;ServerPort=13785) -gn 27 -gp 4096 -gd DBA -gk DBA -gl DBA -ti 0 -c 512M -ch 2048M -cl 512M -zl -os 1M  -o /usr/openv/db//log/server.log
 -ud -m
 

[Deb_Wilmot]

Ok - I'm stumped.... Possibly disabling mirroring might be helpful... I'm not sure though.

Please open a case with support.  Ask that they assist you using the nbdbadmin.exe or nbdbadm tool to disable mirroring and to see if that is helpful.

If that doesn't work, ask the TSE to give me a call and I'll take a look at it with them.  Thanks,
 

Deb W

[Raaavan]

Thnaks D/A

Done call log with TSE & will Post resolution here