cancel
Showing results forΒ 
Search instead forΒ 
Did you mean:Β 

Media server -- Drive not visible when i run vmoprcmd command

Krishma
Level 4

Hi,

Need your help . We have a master/ media server on LINUX with NBU version 6.5 . I am not able to bring UP a drive . It says :

oprd returned abnormal status (96)
IPC Error: Daemon may not be running

Here are the logs :


#############################

[root@veritas log]# grep "DOWN" /var/log/messages-20170716
Jul 10 11:24:53 veritas ltid[18095]: Operator/EMM server has DOWN'ed drive IBM.ULTRIUM-HH6.000 (device 0)

Jul 19 09:52:21 veritas ltid[21804]: ltid can not be started while resources are assigned to the host.

/usr/openv/volmgr/debug - Logs

09:56:22.491 [22231] <2> wait_oprd_ready: oprd response: EXIT_STATUS 278
09:56:22.491 [22231] <2> put_string: cannot write data to network: Broken pipe (32)
09:56:22.491 [22231] <16> send_string: unable to send data to socket: Broken pipe (32), stat=-5

09:56:22.491 [22233] <16> oprd: device management error: IPC Error: Daemon may not be running

[root@veritas bin]# /usr/openv/volmgr/bin/vmoprcmd

HOST STATUS
Host Name Version Host Status
========================================= ======= ===========
veritas 656000 ACTIVE-DISK

PENDING REQUESTS


<NONE>

DRIVE STATUS

Drive Name Label Ready RecMID ExtMID Wr.Enbl. Type
Host DrivePath Status
=============================================================================

[root@veritas ltid]# /usr/openv/volmgr/bin/tpconfig -l
Device Robot Drive Robot Drive Device Second
Type Num Index Type DrNum Status Comment Name Path Device Path
robot 0 - TLD - - - - /dev/sg6
drive - 0 dlt3 1 DOWN - IBM.ULTRIUM-HH6.000 /dev/nst2
drive - 1 dlt3 2 UP - IBM.ULTRIUM-HH6.001 /dev/nst5
drive - 2 dlt3 4 UP - IBM.ULTRIUM-HH6.002 /dev/nst1
drive - 3 dlt3 3 UP - IBM.ULTRIUM-HH6.003 /dev/nst3
drive - 4 dlt3 5 UP - IBM.ULTRIUM-HH6.004 /dev/nst4
drive - 5 dlt3 6 UP - IBM.ULTRIUM-HH6.005 /dev/nst0

[root@veritas bin]# /usr/openv/volmgr/bin/tpconfig -d
Id DriveName Type Residence
Drive Path Status
****************************************************************************
0 IBM.ULTRIUM-HH6.000 dlt3 TLD(0) DRIVE=1
/dev/nst2 DOWN
1 IBM.ULTRIUM-HH6.001 dlt3 TLD(0) DRIVE=2
/dev/nst5 UP
2 IBM.ULTRIUM-HH6.002 dlt3 TLD(0) DRIVE=4
/dev/nst1 UP
3 IBM.ULTRIUM-HH6.003 dlt3 TLD(0) DRIVE=3
/dev/nst3 UP
4 IBM.ULTRIUM-HH6.004 dlt3 TLD(0) DRIVE=5
/dev/nst4 UP
5 IBM.ULTRIUM-HH6.005 dlt3 TLD(0) DRIVE=6
/dev/nst0 UP

 

[root@veritas ~]# /usr/openv/volmgr/bin/vmoprcmd -up 0
oprd returned abnormal status (96)
IPC Error: Daemon may not be running


--------------------------------------------------------------------------------------------------------------------


Reports :

[root@veritas ~]# /usr/openv/volmgr/bin/tpautoconf -report_disc
======================= New Device (Robot) =======================
Inquiry = "SPECTRA PYTHON 2000"
Serial Number = 9111005888
Robot Path = /dev/sg9
Number of Drives = 6
Number of Slots = 120
Number of Media Access Ports = 8
Drive = 1, Drive Name = /dev/nst5, Serial Number = 1012005888
Drive = 2, Drive Name = /dev/nst2, Serial Number = 1013005888
Drive = 3, Drive Name = /dev/nst1, Serial Number = 1014005888
Drive = 4, Drive Name = /dev/nst3, Serial Number = 1015005888
Drive = 5, Drive Name = /dev/nst0, Serial Number = 1016005888
Drive = 6, Drive Name = /dev/nst4, Serial Number = 1017005888
======================= New Device (Drive) =======================
Inquiry = "IBM ULTRIUM-HH6 G9P1"
Serial Number = 1012005888
Drive Path = /dev/nst5
Found as drive=1 in new robot
Robot's inquiry = "SPECTRA PYTHON 2000"
Robot's Serial Number = 9111005888
Robot's device path = /dev/sg9
======================= New Device (Drive) =======================
Inquiry = "IBM ULTRIUM-HH6 G9P1"
Serial Number = 1017005888
Drive Path = /dev/nst4
Found as drive=6 in new robot
Robot's inquiry = "SPECTRA PYTHON 2000"
Robot's Serial Number = 9111005888
Robot's device path = /dev/sg9
======================= New Device (Drive) =======================
Inquiry = "IBM ULTRIUM-HH6 G9P1"
Serial Number = 1015005888
Drive Path = /dev/nst3
Found as drive=4 in new robot
Robot's inquiry = "SPECTRA PYTHON 2000"
Robot's Serial Number = 9111005888
Robot's device path = /dev/sg9
======================= New Device (Drive) =======================
Inquiry = "IBM ULTRIUM-HH6 G9P1"
Serial Number = 1013005888
Drive Path = /dev/nst2
Found as drive=2 in new robot
Robot's inquiry = "SPECTRA PYTHON 2000"
Robot's Serial Number = 9111005888
Robot's device path = /dev/sg9
======================= New Device (Drive) =======================
Inquiry = "IBM ULTRIUM-HH6 G9P1"
Serial Number = 1014005888
Drive Path = /dev/nst1
Found as drive=3 in new robot
Robot's inquiry = "SPECTRA PYTHON 2000"
Robot's Serial Number = 9111005888
Robot's device path = /dev/sg9
======================= New Device (Drive) =======================
Inquiry = "IBM ULTRIUM-HH6 G9P1"
Serial Number = 1016005888
Drive Path = /dev/nst0
Found as drive=5 in new robot
Robot's inquiry = "SPECTRA PYTHON 2000"
Robot's Serial Number = 9111005888
Robot's device path = /dev/sg9
[root@veritas ~]#


[root@veritas ltid]# /usr/openv/volmgr/bin/tpconfig -l
Device Robot Drive Robot Drive Device Second
Type Num Index Type DrNum Status Comment Name Path Device Path
robot 0 - TLD - - - - /dev/sg6
drive - 0 dlt3 1 DOWN - IBM.ULTRIUM-HH6.000 /dev/nst2
drive - 1 dlt3 2 UP - IBM.ULTRIUM-HH6.001 /dev/nst5
drive - 2 dlt3 4 UP - IBM.ULTRIUM-HH6.002 /dev/nst1
drive - 3 dlt3 3 UP - IBM.ULTRIUM-HH6.003 /dev/nst3
drive - 4 dlt3 5 UP - IBM.ULTRIUM-HH6.004 /dev/nst4
drive - 5 dlt3 6 UP - IBM.ULTRIUM-HH6.005 /dev/nst0

[root@veritas volmgr]# /usr/openv/volmgr/bin/tpautoconf -t
TPAC60 IBM ULTRIUM-HH6 G9P1 1012005888 -1 -1 -1 -1 /dev/nst5 - -
TPAC60 IBM ULTRIUM-HH6 G9P1 1017005888 -1 -1 -1 -1 /dev/nst4 - -
TPAC60 IBM ULTRIUM-HH6 G9P1 1015005888 -1 -1 -1 -1 /dev/nst3 - -
TPAC60 IBM ULTRIUM-HH6 G9P1 1013005888 -1 -1 -1 -1 /dev/nst2 - -
TPAC60 IBM ULTRIUM-HH6 G9P1 1014005888 -1 -1 -1 -1 /dev/nst1 - -
TPAC60 IBM ULTRIUM-HH6 G9P1 1016005888 -1 -1 -1 -1 /dev/nst0 - -


Tried this ....
###########################

Ran nbrbutil -resetMediaServer <media server>
Ran nbrbutil -resetall
Restarted NetBackup daemons on media server
Rebooted Media/Master Server
Rebooted Tape Library
Did not find any problems with Tape Library

Thanks,

Krishma.

50 REPLIES 50

Uploading server.txt

Are you able to see all log files ?

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
It is 'after hours' in my part of the world and I am on VOX using my mobile phone. I cannot open zip files. Only .txt files. I will have a look at server.txt.

nbemm log file

Are you now able to see nbemm log files as well ? 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
I have tried to go through the logs but they are massive and I have no easy way of identifying the issue.
I do not even know if the bits and pieces I have seen are real errors or a red herring...

This seems serious to me:

08/07/2017 10:17:48.995 [Application] NB 51216 nbemm 111 PID:30275 TID:4043307888 File ID:111 [No context] [Error] V-111-1049 EMMServer generic error = Can't connect to the database ([Sybase][ODBC Driver][Adaptive Server Anywhere]Database server not found <-1> )

Hopefully @mph999 has a moment or two to give us his opinion?

I have no idea if this can be fixed without a catalog restore.

I truly believe that you need assistance from Veritas Support.

Sorry... I have reached the limit of my capabilities.

So, do you think problem is within the database or  is it possible that problem can be somewhere else too (network,hardware etc etc) ..

Thanks,

Krishma. 

Thiago_Ribeiro
Moderator
Moderator
Partner    VIP    Accredited

Hi @Krishma,

Can you check this TN? 

https://www.veritas.com/support/en_US/article.TECH46255

 

Regards,

 

Thiago Ribeiro

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

@Thiago_Ribeiro

That TN applies to Windows only.

@Krishma

I have searched Google for this error seen in the bottom of server.log:

 *** ERROR *** Assertion failed: 102300 (9.0.2.3900)
 File associated with given page id is invalid or not open

I found this TN that is not exactly the same, but nevertheless similar enough to assume data corruption: https://www.veritas.com/support/en_US/article.000017748

Another search found the exact error in a Sybase forum where the error definely points to corruption: http://nntp-archive.sybase.com/nntp-archive/action/article/%3C3B9D3340.B20562F5@medinotes.com%3E

Please believe me when I say that you need help.

If your management is really in the process of renewing your software, then your Veritas account manager can approach Support to assist you as a once-off favour.

 

 

Thank you all for you guidance and support . Really appreciate it . We are in process of renewing the software . Hopefully we will get some support for this issue.

Thanks again,

Krishma.