network protocol error (39)

[robertngara]

I get this error when i run the command vmoprcmd -d on my media server. what could be wrong? Also the tape drives are in AVR state instead of TLD. I am running Netbackup 6.5.6. My master server is running Sles 11 and my media server has AIX 5.3.

[Yasuhisa_Ishika]

How is the output of "nbemmcmd -listhosts", "bpps - x"? How about "vmoprcmd" on your master server?

[robertngara]

hi yasuhisa

its a new setup.

do you want me to run thr nbemmcmd and bpps commands on the master or media server?

here is the output of vmoprcmd on the master server:

sivls008:/usr/openv/volmgr/bin # ./vmoprcmd

                           HOST STATUS
Host Name                                  Version   Host Status
=========================================  =======   ===========
sivls008                                   656000    ACTIVE
mp2rs018                                   650000    OFFLINE
sivrs055                                   656000    ACTIVE

                                PENDING REQUESTS

                                    <NONE>

                                  DRIVE STATUS

Drive Name               Label   Ready  RecMID  ExtMID  Wr.Enbl.  Type
    Host                       DrivePath                            Status
=============================================================================
IBM.ULTRIUM-TD3.000      No      No                     No        hcart3
    sivls008                   /dev/nst2                            TLD

IBM.ULTRIUM-TD3.001      No      Yes            SV0043  Yes       hcart3
    sivls008                   /dev/nst3                            ACTIVE

IBM.ULTRIUM-TD3.002      Yes     Yes    SV0034  SV0034  Yes       hcart3
    sivls008                   /dev/nst1                            ACTIVE

IBM.ULTRIUM-TD3.003      No      No                     No        hcart3
    sivls008                   /dev/nst0                            TLD

IBM.ULTRIUM-TD3.004      No      No                     No        hcart3
    sivrs055                   /dev/rmt4.1                          AVR

IBM.ULTRIUM-TD3.005      No      No                     No        hcart3
    sivrs055                   /dev/rmt7.1                          AVR

IBM.ULTRIUM-TD3.006      No      No                     No        hcart3
    sivrs055                   /dev/rmt6.1                          AVR

IBM.ULTRIUM-TD3.007      No      No                     No        hcart3
    sivrs055                   /dev/rmt5.1                          AVR
sivls008:/usr/openv/volmgr/bin #
 

[Marianne · Accepted solution]

Do 'nbemmcmd -listhosts -verbose' on master

and 'bpps -x' on media server.

Is media server perhaps behind firewall? Port 1556 (PBX) open in both directions?

Where is the robotic control? If not master, you need to ensure PBX comms (bi-directional) between media server and robot control host as well.

Proper forward and reverse lookup of host/IP in all directions? If FQDN is used anywhere, it must be able to resolve everywhere.

Easiest is to add /etc/hosts entries on master and media server (and robot control host) with short and FQDN, e.g

on master:

10.10.10.1 master master.domain loghost
10.10.10.2 media media.domain

on media

10.10.10.2 media media.domain loghost
0.10.10.1 master master.domain

Add appropriate entries if robot control host is another media server.

[robertngara]

here is output of bpps -x from the media server:

[root@sivrs055 /usr/openv/netbackup/bin]# ./bpps -x
NB Processes
------------
    root 335920      1   0 16:17:16      -  0:00 /usr/openv/netbackup/bin/bpcompatd
    root 381078      1   0 16:17:18      -  0:00 /usr/openv/netbackup/bin/nbsl
    root 454766      1   0 16:17:18      -  0:03 /usr/openv/netbackup/bin/nbsvcmon
    root 471110      1   0 16:17:17      -  0:17 /usr/openv/netbackup/bin/nbrmms

MM Processes
------------
    root 311508 438372   0 16:17:23      -  0:00 tldd
    root 364620      1   0 16:17:25      -  0:00 tldcd
    root 438372      1   0 16:17:15      -  0:01 /usr/openv/volmgr/bin/ltid
    root 479262 438372   0 16:17:25      -  0:03 avrd
    root 499718      1   0 16:17:16      -  0:00 vmd

Shared Symantec Processes
-------------------------
    root 258190      1   0   Oct 12      -  0:00 /opt/VRTSpbx/bin/pbx_exchange
[root@sivrs055 /usr/openv/netbackup/bin]#

and nbemmcmd -listhosts from the master server:

sivls008:/usr/openv/netbackup/bin/admincmd # ./nbemmcmd -listhosts
NBEMMCMD, Version:6.5.6
The following hosts were found:
server          sivls008
master          sivls008
media           mp2rs018
media           sivrs055
Command completed successfully.
sivls008:/usr/openv/netbackup/bin/admincmd #
 

yes i am using the /etc/hosts file for the host names resolution.

there is no firewall and the port 1556 is open as i can telnet: telnet server-name 1556.

[robertngara]

hi Marianne

something interesting here:

I have defined my media server as sivrs055 but from the /usr/openv/netbackup/remote_versions folder i am getting a FQDN (sivrs055.eskom.co.za) and the file doesnt contain the netbackup version. its had 0.

[Marianne]

Where is the robot control? I see tldcd is running on this media server meaning it's the robot control host?

Please paste output of your device config on this server:

/usr/openv/volmgr/bin/tpconfig -l

How did you do device config - manually or via the GUI wizard?

Enable Media Manager verbose logging by adding VERBOSE line to /usr/openv/volmgr/bin/vm.conf.

Stop/start NBU daemons.

Ensure syslog is enabled on the AIX media server to enable media manager logging to /var/adm/messages (the syslogd daemon must be active).

[robertngara]

Hi Marianne

Yes the robot control is on that media server. There is an HBA connected to the VTL from the media server.

Here is the output of the tpconfig -l command:

[root@sivrs055 /usr/openv/volmgr/bin]# ./tpconfig -l
Device Robot Drive       Robot                    Drive                Device
Type     Num Index  Type DrNum Status  Comment    Name                 Path
robot      1    -    TLD    -       -  -          -                    /dev/ovpass2
  drive    -    0 hcart3    1      UP  -          IBM.ULTRIUM-TD3.004  /dev/rmt4.1
  drive    -    1 hcart3    4      UP  -          IBM.ULTRIUM-TD3.005  /dev/rmt7.1
  drive    -    2 hcart3    3      UP  -          IBM.ULTRIUM-TD3.006  /dev/rmt6.1
  drive    -    3 hcart3    2      UP  -          IBM.ULTRIUM-TD3.007  /dev/rmt5.1
[root@sivrs055 /usr/openv/volmgr/bin]#
 

I did configure the devices using GUI and it picked the 4 drives and one robot.

I have added VERBOSE line in the vm.conf file.

[Marianne]

Looks like PBX level comms is fine from master to media server (GUI wizard worked) but problematic when initiated from the media server.

Have you tried the FQDN aliases in /etc/hosts as I've suggested previously?

You are going to need the logging that I've mentioned in the previous post as well as the following NBU logs:

/usr/openv/volmgr/debug/daemon

/usr/openv/volmgr/debug/reqlib

When you create these log directories, also add the following line to vm.conf (to prevent /usr filling up):

DAYS_TO_KEEP_LOGS = 3

Verify that syslogd is running and that system messages are logged to /var/adm/messages. Stop/start NBU.

[robertngara]

Hi Marianne

This is what i am getting from /usr/openv/volmgr/debug/reqlib, check the line in bold:

[root@sivrs055 /usr/openv/volmgr/debug/reqlib]# more log.101410
14:18:56.516 [225336] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2048: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
14:18:56.516 [225336] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2062: service: vmd
14:18:56.522 [225336] <2> get_long: (2) premature end of file (byte 1)
14:18:56.522 [225336] <16> getrequestack: unable to get request ack string: There is an input or output error. (5), stat=-9
14:18:56.522 [225336] <16> vmdbm_terminate: unable to send request to server:  network protocol error (39)
[root@sivrs055 /usr/openv/volmgr/debug/reqlib]#
 

[Marianne]

So it seems I was right in previous post? Media Server initiated comms are not getting to the master. You need to check that all is well at TCP/IP comms level.

Have you tried to alias FQDN in /etc/hosts as I've suggested initially?

[robertngara]

Hi Marianne

Strange enough i had to add this line in the /etc/hosts of the media server itsself for it to work

147.110.197.101 sivrs055

Thats the ip address and hostname of the media server.

Thanks a lot for all the help, i really appreciate.

[Andy_Welburn]

the media server didn't have an entry in /etc/hosts for itself?

[robertngara]

yes the media server didnt have the entry in /etc/hosts for itself. I never thought it was necessay beause my DNS is working fine. I only realised it when i got an error after running the command bpclntcmd -ip <media-server-ip> on the media server itsself and got an error.

[Marianne]

Please select a post that answered your question and mark as Solution?