adding a new media server, using SSO & ACSLS

[rjrumfelt]

All servers are solaris 10, NBU 6.5.4

When attempting to inventory NBU, getting error 204, and the drives are showing up as AVR

Any suggestions?

Do you guys need any further info?  What I've provided isnt much.

[Nicolai]

The tapes drives should be in ACS mode not AVR. You may have a ACSLS commutation problem.

Does acstest work ?

Does the /var/log/messages  contain info from the acsd daemon ?

[Marianne]

acstest and/or robtest will probably return an error code - something like ACS STATUS = 54 or 72.
This TechNote is excellent to troubleshoot these comms problems: http://seer.entsupport.symantec.com/docs/267306.htm

One of the most common problems is when the media server have two NICs. The hostname to be used for comms with the ACSLS sever must be added in vm.conf on the media server: ACS_SSI_HOSTNAME = <hostname>.
The other common problem is firewall comms between the media server and ACSLS, but I'm pretty sure you would've mentioned it if that was the case....

[rjrumfelt]

yesterday was hectic to say the least, and I was not able to get back here.

Nicolai, I havent had a chance to try acstest - I will get to that shortly.  I've checked the messages log, and there are no entries for acsd, or any other comm or hardware entries.

Marianne, we are communicating with ACSLS over the mgt rail, and I added the ACS_SSI_HOSTNAME = <hostname> entry when we started yesterday, and no luck.  Not sure what its worth, but I've verified that I can telnet to 30031 from the media servers.  Below is something similar to our vm.conf file:

ACS_SSI_HOSTNAME=<IP_of_MGT_RAIL>
ACS_TCP_RPCSERVICE
ACS_CSI_HOSTPORT=<IP_of_ACSLS> 30031
ACS_SSI_INET_PORT=<IP_of_ACSLS> 30031
INVENTORY_FILTER=stuff

Not sure if I'm missing anything.

I've also tried substituing the actual hostname for the ACS_SSI_HOSTNAME entry, with no luck.

[Nicolai · Accepted solution]

When you add ACS_SSI_HOSTNAME,  make sure to restart acsd and acsssi and delete previous registered RPC entries.


Here is a article I wrote some time ago: Netbackup and the ACSLS firewall feature

[Marianne]

Nicolai, was that you who wrote that article?! I have a bookmark set to that URL and especially love point 9...

Jason, you might have to add VERBOSE entry in vm.conf to force logging to /var/adm/messages.
Please run robtest from media server. Select the acs robot and wait for an error message to be generated. Please post the error message.
Also check the acsls event log (/export/home/ACSSS/log/acsss_event.log) for comms request from your media server and resulting return comms.
Have you verified that RPC is running on the media server? Check output of rpcinfo.
Also test and verify the following on the media server:

How to Verify the ACSSS Program Registration
#rpcinfo -t {acsls_hostname} 300031 2
program 300031 version 2 ready and waiting
#rpcinfo -t {acsls_hostname} 300031 1
program 300031 version 1 ready and waiting

Also ensure that vm.conf entries are actual hostnames, not IP addresses. See this TechNote:
http://seer.entsupport.symantec.com/docs/306739.htm
Extract:

(Note: for <remote_acsls_hostname>, DO NOT USE A IP ADDRESS.)

Example:
ACS_TCP_RPCSERVICE
ACS_CSI_HOSTPORT = acsls 30031
ACS_SSI_INET_PORT = acsls 30031

[rjrumfelt]

when running acstest I get an ACS STATUS = 72

RPC entries look good

Just to rule anything out, I went back and used hostnames instead of IP's for the vm.conf file, and it didnt change anything.

Marianne, when I attempt to verify ACSSS registration, I get the following, after waiting some time:

"Unable to obtain Query Server acknowledge response, ACS status = 72, STATUS_PENDING
rpcinfo: RPC: Rpcbind failure - RPC: Timed out
program 300031 version 2 is not available

Working on the other suggestions as we speak

[Marianne]

The TechNote that I posted yesterday says the following about status 72:

Event log from the ACSLS :
2003-05-22 11:33:51 CSI[0]:
1022 N csi_net_send.c 1 474
ONC RPC: csi_net_send(): status:STATUS_NI_TIMEDOUT; failed: st_net_send() Cannot send message to NI:discarded, Network timeout Errno = 0 (none) Remote Internet address: 207.169.154.59, Port: 53429

2003-05-22 11:33:51 CSI[0]:
1026 N csi_freeqmem.c 1 142
ONC RPC: csi_freeqmem(): status:STATUS_QUEUE_FAILURE;
Dropping from Queue: Remote Internet address: 207.169.154.59,Port: 53429 , ssi_identifier: 1, Protocol: 2, Connect type: 1

2003-05-22 11:33:51 ACSSA[0]:
1432 N sa_demux.c 1 273
Server System network interface timeout.

Cause:

The errors above indicate that NetBackup is able to reach the ACSLS server with a status request, but the ACSLS server is unable to respond.

By default, the IP address sent to the ACSLS as part of the packet STATUS request is that of the primary hostname of the media server, i.e. the hostname given by uname -a. This error occurs when the ACSLS cannot resolve reverse name or is configured (via routing) to only reach a secondary interface on the media server.

If the issue is failure to do reserve name lookup, add the media server's IP to the domain name service (DNS) reverse tables or to the ACSLS /etc/hosts file.

If the ACSLS server cannot route to the media server's hostname, override the default behavior by using "ACS_SSI_HOSTNAME = <hostname>" in the /usr/openv/volmgr/vm.conf file where the value for <hostname> is a hostname associated with an IP address the ACSLS can reach on the media server.

[Nicolai]

You only got 30031open i one direction. Both sides must be able to initiate traffic at 30031 (bidirectional opening).

If you do a "snoop 30031"  at the ACSLS server side you should be able to see ACSLS try 30031 but then fall backup to RPC port 110 communication.

[Nicolai]

that home page is my hideout :)

[rjrumfelt]

The one thing I hadnt checked yet is communication to the media server from the acsls server.  And guess what?  That communication isnt there.  The bad news is it may take an act of the Almighty to get this rectified.

I'll keep everyone posted - there's a chance that an issue will still remain once connectivity has been resolved.

[Nicolai]

It will work - Seen that error before. All our media server are ACSLS attached and we are highly firewalled. Been there where you are :-))

[rjrumfelt]

and now I can ping the media server from acsls, but still cant telnet to 30031 on the media server from acsls

Odd thing is though, I get a connection refused to the master server as well, and it has been working just fine with acsls for a long time.

[Nicolai]

I dont think the portproblem is related to DNS. On my ACSLS server I have none of the media servers listed.

[rjrumfelt]

and the drives no longer show up as AVR.

The final solution was to have the security folks put in bidirectional rules for 30031, which we thought was done when the rules were first submitted.

Thanks Nic and Marianne for all of the help, you were both extremely helpful.  I am always torn when it comes to giving the nod when everyone collaborated.

I haven't attempted a backup yet, as we're not to that point in the process.  I may have to btt this thread, or create a new one if I have further issues.

[Marianne]

You can mark Nicolai's post where he says that 30031 must be opened in both directions.
He is also my hero of the year for his article - you have no idea how much that has helped me in the past...

[Nicolai]

Thanks for the nominations. I am glad you like the article.

[rjrumfelt]

I'm glad someone made the decision for me :)  Its probably not a big deal, but I really gave it a lot of thought.  I will give Nic the nod.

Thanks again for the help.