β07-24-2017 12:23 PM
Hi,
Need your help . We have a master/ media server on LINUX with NBU version 6.5 . I am not able to bring UP a drive . It says :
oprd returned abnormal status (96)
IPC Error: Daemon may not be running
Here are the logs :
#############################
[root@veritas log]# grep "DOWN" /var/log/messages-20170716
Jul 10 11:24:53 veritas ltid[18095]: Operator/EMM server has DOWN'ed drive IBM.ULTRIUM-HH6.000 (device 0)
Jul 19 09:52:21 veritas ltid[21804]: ltid can not be started while resources are assigned to the host.
/usr/openv/volmgr/debug - Logs
09:56:22.491 [22231] <2> wait_oprd_ready: oprd response: EXIT_STATUS 278
09:56:22.491 [22231] <2> put_string: cannot write data to network: Broken pipe (32)
09:56:22.491 [22231] <16> send_string: unable to send data to socket: Broken pipe (32), stat=-5
09:56:22.491 [22233] <16> oprd: device management error: IPC Error: Daemon may not be running
[root@veritas bin]# /usr/openv/volmgr/bin/vmoprcmd
HOST STATUS
Host Name Version Host Status
========================================= ======= ===========
veritas 656000 ACTIVE-DISK
PENDING REQUESTS
<NONE>
DRIVE STATUS
Drive Name Label Ready RecMID ExtMID Wr.Enbl. Type
Host DrivePath Status
=============================================================================
[root@veritas ltid]# /usr/openv/volmgr/bin/tpconfig -l
Device Robot Drive Robot Drive Device Second
Type Num Index Type DrNum Status Comment Name Path Device Path
robot 0 - TLD - - - - /dev/sg6
drive - 0 dlt3 1 DOWN - IBM.ULTRIUM-HH6.000 /dev/nst2
drive - 1 dlt3 2 UP - IBM.ULTRIUM-HH6.001 /dev/nst5
drive - 2 dlt3 4 UP - IBM.ULTRIUM-HH6.002 /dev/nst1
drive - 3 dlt3 3 UP - IBM.ULTRIUM-HH6.003 /dev/nst3
drive - 4 dlt3 5 UP - IBM.ULTRIUM-HH6.004 /dev/nst4
drive - 5 dlt3 6 UP - IBM.ULTRIUM-HH6.005 /dev/nst0
[root@veritas bin]# /usr/openv/volmgr/bin/tpconfig -d
Id DriveName Type Residence
Drive Path Status
****************************************************************************
0 IBM.ULTRIUM-HH6.000 dlt3 TLD(0) DRIVE=1
/dev/nst2 DOWN
1 IBM.ULTRIUM-HH6.001 dlt3 TLD(0) DRIVE=2
/dev/nst5 UP
2 IBM.ULTRIUM-HH6.002 dlt3 TLD(0) DRIVE=4
/dev/nst1 UP
3 IBM.ULTRIUM-HH6.003 dlt3 TLD(0) DRIVE=3
/dev/nst3 UP
4 IBM.ULTRIUM-HH6.004 dlt3 TLD(0) DRIVE=5
/dev/nst4 UP
5 IBM.ULTRIUM-HH6.005 dlt3 TLD(0) DRIVE=6
/dev/nst0 UP
[root@veritas ~]# /usr/openv/volmgr/bin/vmoprcmd -up 0
oprd returned abnormal status (96)
IPC Error: Daemon may not be running
--------------------------------------------------------------------------------------------------------------------
Reports :
[root@veritas ~]# /usr/openv/volmgr/bin/tpautoconf -report_disc
======================= New Device (Robot) =======================
Inquiry = "SPECTRA PYTHON 2000"
Serial Number = 9111005888
Robot Path = /dev/sg9
Number of Drives = 6
Number of Slots = 120
Number of Media Access Ports = 8
Drive = 1, Drive Name = /dev/nst5, Serial Number = 1012005888
Drive = 2, Drive Name = /dev/nst2, Serial Number = 1013005888
Drive = 3, Drive Name = /dev/nst1, Serial Number = 1014005888
Drive = 4, Drive Name = /dev/nst3, Serial Number = 1015005888
Drive = 5, Drive Name = /dev/nst0, Serial Number = 1016005888
Drive = 6, Drive Name = /dev/nst4, Serial Number = 1017005888
======================= New Device (Drive) =======================
Inquiry = "IBM ULTRIUM-HH6 G9P1"
Serial Number = 1012005888
Drive Path = /dev/nst5
Found as drive=1 in new robot
Robot's inquiry = "SPECTRA PYTHON 2000"
Robot's Serial Number = 9111005888
Robot's device path = /dev/sg9
======================= New Device (Drive) =======================
Inquiry = "IBM ULTRIUM-HH6 G9P1"
Serial Number = 1017005888
Drive Path = /dev/nst4
Found as drive=6 in new robot
Robot's inquiry = "SPECTRA PYTHON 2000"
Robot's Serial Number = 9111005888
Robot's device path = /dev/sg9
======================= New Device (Drive) =======================
Inquiry = "IBM ULTRIUM-HH6 G9P1"
Serial Number = 1015005888
Drive Path = /dev/nst3
Found as drive=4 in new robot
Robot's inquiry = "SPECTRA PYTHON 2000"
Robot's Serial Number = 9111005888
Robot's device path = /dev/sg9
======================= New Device (Drive) =======================
Inquiry = "IBM ULTRIUM-HH6 G9P1"
Serial Number = 1013005888
Drive Path = /dev/nst2
Found as drive=2 in new robot
Robot's inquiry = "SPECTRA PYTHON 2000"
Robot's Serial Number = 9111005888
Robot's device path = /dev/sg9
======================= New Device (Drive) =======================
Inquiry = "IBM ULTRIUM-HH6 G9P1"
Serial Number = 1014005888
Drive Path = /dev/nst1
Found as drive=3 in new robot
Robot's inquiry = "SPECTRA PYTHON 2000"
Robot's Serial Number = 9111005888
Robot's device path = /dev/sg9
======================= New Device (Drive) =======================
Inquiry = "IBM ULTRIUM-HH6 G9P1"
Serial Number = 1016005888
Drive Path = /dev/nst0
Found as drive=5 in new robot
Robot's inquiry = "SPECTRA PYTHON 2000"
Robot's Serial Number = 9111005888
Robot's device path = /dev/sg9
[root@veritas ~]#
[root@veritas ltid]# /usr/openv/volmgr/bin/tpconfig -l
Device Robot Drive Robot Drive Device Second
Type Num Index Type DrNum Status Comment Name Path Device Path
robot 0 - TLD - - - - /dev/sg6
drive - 0 dlt3 1 DOWN - IBM.ULTRIUM-HH6.000 /dev/nst2
drive - 1 dlt3 2 UP - IBM.ULTRIUM-HH6.001 /dev/nst5
drive - 2 dlt3 4 UP - IBM.ULTRIUM-HH6.002 /dev/nst1
drive - 3 dlt3 3 UP - IBM.ULTRIUM-HH6.003 /dev/nst3
drive - 4 dlt3 5 UP - IBM.ULTRIUM-HH6.004 /dev/nst4
drive - 5 dlt3 6 UP - IBM.ULTRIUM-HH6.005 /dev/nst0
[root@veritas volmgr]# /usr/openv/volmgr/bin/tpautoconf -t
TPAC60 IBM ULTRIUM-HH6 G9P1 1012005888 -1 -1 -1 -1 /dev/nst5 - -
TPAC60 IBM ULTRIUM-HH6 G9P1 1017005888 -1 -1 -1 -1 /dev/nst4 - -
TPAC60 IBM ULTRIUM-HH6 G9P1 1015005888 -1 -1 -1 -1 /dev/nst3 - -
TPAC60 IBM ULTRIUM-HH6 G9P1 1013005888 -1 -1 -1 -1 /dev/nst2 - -
TPAC60 IBM ULTRIUM-HH6 G9P1 1014005888 -1 -1 -1 -1 /dev/nst1 - -
TPAC60 IBM ULTRIUM-HH6 G9P1 1016005888 -1 -1 -1 -1 /dev/nst0 - -
Tried this ....
###########################
Ran nbrbutil -resetMediaServer <media server>
Ran nbrbutil -resetall
Restarted NetBackup daemons on media server
Rebooted Media/Master Server
Rebooted Tape Library
Did not find any problems with Tape Library
Thanks,
Krishma.
β08-07-2017 08:10 AM
β08-07-2017 08:13 AM
Are you able to see all log files ?
β08-07-2017 10:18 AM
β08-07-2017 10:52 AM
β08-07-2017 10:52 AM
β08-07-2017 10:57 AM
Are you now able to see nbemm log files as well ?
β08-07-2017 12:52 PM
β08-07-2017 01:07 PM
So, do you think problem is within the database or is it possible that problem can be somewhere else too (network,hardware etc etc) ..
Thanks,
Krishma.
β08-07-2017 01:13 PM
Hi @Krishma,
Can you check this TN?
https://www.veritas.com/support/en_US/article.TECH46255
Regards,
Thiago Ribeiro
β08-08-2017 12:32 AM
That TN applies to Windows only.
I have searched Google for this error seen in the bottom of server.log:
*** ERROR *** Assertion failed: 102300 (9.0.2.3900) File associated with given page id is invalid or not open
I found this TN that is not exactly the same, but nevertheless similar enough to assume data corruption: https://www.veritas.com/support/en_US/article.000017748
Another search found the exact error in a Sybase forum where the error definely points to corruption: http://nntp-archive.sybase.com/nntp-archive/action/article/%3C3B9D3340.B20562F5@medinotes.com%3E
Please believe me when I say that you need help.
If your management is really in the process of renewing your software, then your Veritas account manager can approach Support to assist you as a once-off favour.
β08-08-2017 07:44 AM
Thank you all for you guidance and support . Really appreciate it . We are in process of renewing the software . Hopefully we will get some support for this issue.
Thanks again,
Krishma.