Media Server is picking wrong IP when backup Initiated from Master Server

[2013]

Hello All,

A strange issue i would like to highligt.

Master Server - 7.1 (Unix)

Media Server - 6.5 (unix)

Our Media Server is Offline.Whenever the backup start it pick the wrong IP 130.8.37.177 to 144.160.231.154.It should pick IP 130.8.47.171 which is our Master Server IP.

04:02:49.580 [8218] <16> bpcd main: Server access denied
04:02:55.306 [8230] <2> ListenForConnection: Unexpected working directory: /root
04:02:55.306 [8230] <2> bpcd main: offset to GMT 18000
04:02:55.306 [8230] <2> logconnections: BPCD ACCEPT FROM 130.8.37.177.33793 TO 144.160.231.154.13724
04:02:55.306 [8230] <2> bpcd main: setup_sockopts complete
04:02:55.309 [8230] <2> ParseConfigExA: Unknown configuration option on line 6: VERBOSE 5
04:02:55.309 [8230] <2> bpcd peer_hostname: Connection from host asprd212-ebr.aldc.att.com (130.8.37.177) port 33793
04:02:55.309 [8230] <2> bpcd valid_server: comparing asprd212.aldc.att.com and asprd212-ebr.aldc.att.com
04:02:55.312 [8230] <2> bpcd valid_server: comparing asprd472.enaf.aldc.att.com and asprd212-ebr.aldc.att.com
04:02:55.313 [8230] <4> bpcd valid_server: asprd212-ebr.aldc.att.com is not a master server
04:02:55.313 [8230] <16> bpcd valid_server: asprd212-ebr.aldc.att.com is not a media server either
04:02:55.313 [8230] <2> bpcd main: output socket port number = 1
04:02:55.349 [8230] <2> bpcd main: Duplicated vnetd socket on stderr
04:02:55.350 [8230] <2> bpcd main: <---- NetBackup 6.5 0 ------------initiated
04:02:55.350 [8230] <2> bpcd main: VERBOSE = 0
04:02:55.350 [8230] <2> bpcd exit_bpcd: exit status 46  ----------->exiting
04:02:55.350 [8230] <4> bpcd exit_bpcd: FTL - BPCD EXIT STATUS 46
-----------------------------------------------------------------------------------------------------

BP.conf file is correct:

SERVER = asprd212.aldc.att.com
SERVER = asprd472.enaf.aldc.att.com
CLIENT_NAME = asprd472.enaf.aldc.att.com
EMMSERVER = asprd212.aldc.att.com
ALLOW_MEDIA_OVERWRITE = ANSI
VERBOSE 5
CLIENT_CONNECT_TIMEOUT = 300
CLIENT_READ_TIMEOUT = 1800
REQUIRED_INTERFACE = asprd472.enaf.aldc.att.com
----------------------------------------------------------------------------------------------------------

Master Server to Media Server

asprd212: ./bpclntcmd -pn
expecting response from server asprd212.aldc.att.com
asprd212 asprd212.aldc.att.com 130.8.47.171 46004

-----------------------------------

/etc/hosts

asprd212: cat /etc/hosts | grep asprd472.enaf.aldc.att.com
144.160.231.154  asprd472.enaf.aldc.att.com


---------------------------------------------------------------------------------
Media Server to Master

asprd472: ./bpclntcmd -sv
7.1
asprd472: ./bpclntcmd -pn
expecting response from server asprd212.aldc.att.com
asprd472.enaf.aldc.att.com asprd472.enaf.aldc.att.com 144.160.231.154 42254

---------------------------------------------------------------------------------------------------------------------

/etc/hosts

asprd472: cat /etc/hosts | grep asprd212
130.8.47.171  asprd212.aldc.att.com

-----------------------------------------------------------------------------------------------------------------------

Previously ebr interface was added in Media Server bp.conf file which was removed but still whenever the Backup initiate it pick the same ebr 130.8.37.177 interface......We want that whenever backup start it should pick the correct interface 130.8.47.171.....Any help would be appreciated......
 

[sksujeet]

Yes indeed it is picking the same interface which was removed. When did you remove it?
04:02:55.313 [8230] <4> bpcd valid_server: asprd212-ebr.aldc.att.com is not a master server
04:02:55.313 [8230] <16> bpcd valid_server: asprd212-ebr.aldc.att.com is not a media server either

Please run this command bprdreq -rereadconfig to update the changes of conf file and see if it works.

[Marianne]

Is 130.8.37.177 a second interface on the master server? 

Is each interface/IP bound to a different hostname?

Please show us output of the following on the master server:

grep asprd212 /etc/hosts

grep 130.8.37.177 /etc/hosts

Which STU is defined in policy? Check Attributes as well as Schedule.
Check STU properties.
 

Please read through this section in chapter 2 of NBU Troubleshooting Guide:

About troubleshooting networks and hostnames

In a configuration with multiple networks and clients with more than one

hostname, NetBackup administrators must configure the policy entries carefully.

They must consider the network configuration (physical, hostnames and aliases,

NIS/DNS, routing tables, and so on). If administrators want to direct backup and

restore data across specific network paths, they especially need to consider these

things.

For a backup, NetBackup connects to the host name as configured in the policy.

The operating system’s network code resolves this name and sends the connection

across the network path that is defined by the system’s routing tables.

The bp.conf file is not a factor making this decision.

[2013]

Yesterday i removed the ebr entry from the Media Server.afterwards i bounced the services also....but whenever the backup started it pick the wrong ip....I am not getting from where it is geeting....I executed the command but same result....

[Marianne]

I have in the meantime edited my previous post.

Please see the extract from Troubleshooting Guide.

[2013]

asprd212: cat /etc/hosts | grep asprd212
130.8.47.171    asprd212                asprd212.aldc.att.com   loghost
130.8.47.172    asprd212-prod1-if1      asprd212-prod1-if1.aldc.att.com
130.8.47.173    asprd212-prod1-if2      asprd212-prod1-if2.aldc.att.com
130.8.37.176    asprd212-ebr            asprd212-ebr.aldc.att.com
130.8.37.177 asprd212-ebr-bkup asprd212-ebr-bkup.aldc.att.com
130.8.37.178 asprd212-ebr-if1 asprd212-ebr-if1.aldc.att.com
130.8.37.179 asprd212-ebr-if2 asprd212-ebr-if2.aldc.att.com
130.1.178.43    asprd212-nas1     asprd212-nas1.aldc.att.com
130.1.178.44    asprd212-nas1-if1 asprd212-nas1-if1.aldc.att.com
130.1.178.45    asprd212-nas1-if2 asprd212-nas1-if2.aldc.att.com
130.1.188.44    asprd212-nebr     asprd212-nebr.aldc.att.com
130.1.188.45    asprd212-nebr-if1 asprd212-nebr-if1.aldc.att.com
130.1.188.46    asprd212-nebr-if2 asprd212-nebr-if2.aldc.att.com

-------------------------------------------------------------------------
asprd212: cat /etc/hosts | grep 130.8.37.177
130.8.37.177 asprd212-ebr-bkup asprd212-ebr-bkup.aldc.att.com
-------------------------------------------------------------------------
Which STU is defined in policy? Check Attributes as well as Schedule.
Check STU properties.

STu-asprd472_vtl

Sending the snapshot of interface.....

[2013]

Thanks for the update...I have checked with SA team and Network Team......But the question is that from Master Server Host Properies we are not able to connect the Media Server......it always go to the wrong ip....Is there any specific we can do on media server that it connect to the correct IP....

[Marianne]

Have you tried 'traceroute' from master server to client name in the policy?

This should show you which route is chosen.

[2013]

I bold the IP..please check

output of netstat -nr of Master

asprd212: netstat -nr

Routing Table: IPv4
  Destination           Gateway           Flags  Ref     Use     Interface
-------------------- -------------------- ----- ----- ---------- ---------
default              130.8.47.131         UG        1       1362
10.112.120.0         130.1.188.3          UG        1        166
10.112.120.0         130.1.188.3          UG        1         55
130.1.176.66         130.1.178.43         UH        1          1 nxge1:1
130.1.178.0          130.1.178.43         U         1     112603 nxge1:1
130.1.178.0          130.1.178.43         U         1          0 nxge1
130.1.178.0          130.1.178.43         U         1        111 nxge5
130.1.184.0          130.1.188.3          UG        1         47
130.1.188.0          130.1.188.44         U         1        629 nxge9:1
130.1.188.0          130.1.188.44         U         1          0 nxge9
130.1.188.0          130.1.188.44         U         1         69 nxge11
130.8.37.0           130.8.37.176         U         1      53699 nxge0:1
130.8.37.0           130.8.37.177         U         1      58908 nxge4:1
130.8.37.0           130.8.37.176         U         1          0 nxge0
130.8.37.0           130.8.37.177         U         1          0 nxge4
130.8.37.0           130.8.47.171         UG        1          0
130.8.41.0           130.8.37.1           UG        1        294
130.8.47.128         130.8.47.171         U         1          9 bge0:1
130.8.47.128         130.8.47.171         U         1          0 bge0
130.8.47.128         130.8.47.171         U         1         68 bge2
144.160.231.154      130.8.37.1           UGH       1        206
192.168.224.1        192.168.224.2        UH        1          1 sppp0
224.0.0.0            130.8.47.171         U         1          0 bge0:1
127.0.0.1            127.0.0.1            UH      216     367053 lo0
 

144.160.231.154 - This is my client Ip....

130.8.47.171 ---Backup should pick this IP but its picking wrong IP...130.8.37.177

Any idea....

[2013]

Below output of Traceroute from Master to Client IP....

@asprd212: traceroute 144.160.231.154
traceroute: Warning: Multiple interfaces found; using 130.8.37.176 @ nxge0:1
traceroute to 144.160.231.154 (144.160.231.154), 30 hops max, 40 byte packets
 1  alprgaedgrddcb42-v339.dc.sest.att.com (130.8.37.1)  1.116 ms  0.827 ms  1.095 ms
 2  alprgaedgrbdca01-v256.dc.sest.att.com (135.53.0.190)  0.398 ms  0.390 ms  0.224 ms
 3  alprgaedcrbindl01-ge000.pmtr.sest.att.com (135.28.160.91)  0.190 ms  0.256 ms  0.201 ms
 4  gaalxbm12a-diglife_1-bbintvirt.pst.cso.att.com (135.28.191.38)  0.628 ms  0.805 ms  0.638 ms
 5  asprd472.enaf.aldc.att.com (144.160.231.154)  1.063 ms  1.220 ms  1.080 ms

asprd212: traceroute asprd472.enaf.aldc.att.com
traceroute: Warning: Multiple interfaces found; using 130.8.37.177 @ nxge4:1
traceroute to asprd472.enaf.aldc.att.com (144.160.231.154), 30 hops max, 40 byte packets
 1  alprgaedgrddcb42-v339.dc.sest.att.com (130.8.37.1)  1.057 ms  0.909 ms  0.861 ms
 2  alprgaedgradca01-v255.dc.sest.att.com (135.53.0.126)  0.415 ms  0.493 ms  0.385 ms
 3  alprgaedcraindl01-ge000.pmtr.sest.att.com (135.28.160.59)  0.196 ms  0.460 ms  0.396 ms
 4  gaalxbm12a-diglife_1-bbintvirt.pst.cso.att.com (135.28.191.38)  0.634 ms  0.755 ms  0.641 ms
 5  asprd472.enaf.aldc.att.com (144.160.231.154)  1.295 ms  1.207 ms  1.078 ms

 

[2013]

Seem to be on Master Server netstat -nr is the culprit.....Media Server IP added in netstat

144.160.231.154      130.8.37.1           UGH       1        206

which is routing to 130.8.37.1 gateway.....I thing we need to put the gateway  130.8.47.171.....

wat u say....

[RamNagalla]

Hi,

remove the below route from your master server, and try, it should go fine.

130.8.37.0           130.8.37.177         U         1          0 nxge4

[2013]

Thanks for the update...

But as per my experience 144.160.231.154 is our Media Server which is having gateway 130.8.37.1..I am thinking to change the gateway to 130.8.47.171 so that whenever the Backup starts it pick the correct gateway 130.8.47.171.....

Correct me if i am wrong....

[Yasuhisa_Ishika]

I am thinking to change the gateway to 130.8.47.171 so that whenever the Backup starts it pick the correct gateway 130.8.47.171.....

No luck, I think. By this routing this post send packets to itself.

Do you have any gateway on 130.8.47.128 which can pass packet to 144.160.231.154? If so, use it as gawate to 144.160.231.154.
If you want to set source IP as 130.8.47.171, 130.8.47.1 can not be used. This host use 130.8.37.176 or 130.8.37.177 beacuse there interfaces are nearest ones.

[RamNagalla · Accepted solution]

hey, 

seem did mis read your routing table..

but still your routing table has the muttiple route for the same destinations, which you man need to condiser

130.8.37.0           130.8.37.177         U         1          0 nxge4
130.8.37.0           130.8.47.171         UG        1          0

130.8.47.128         130.8.47.171         U         1          9 bge0:1

130.8.47.128         130.8.47.171         U         1          0 bge0
130.8.47.128         130.8.47.171         U         1         68 bge2

#################

comming to the curret issue..

yes , your approach is looking fine to me

remove the route

144.160.231.154      130.8.37.1           UGH       1        206

and see if your media  server is pinging to the gateway 130.8.47.1, if yes

use the below route

destination :-144.160.231.154 

gateway:-130.8.47.1

subnetmask:-255.255.255.255

if media server is not pinging to 130.8.47.1.

then use below

destination :-144.160.231.154 

gateway:-130.8.47.171

subnetmask:-255.255.255.255

remember unless you have proper route in media server it will not work, 

if it will not work, please provide the belowout puts from both master and media servers, attache to the post

ifconfig -a

netstat -nr

[2013]

Hi Yasuhisa/Nagalla,

You are correct...

I am thinking to change the gateway to 130.8.47.171 so that whenever the Backup starts it pick the correct gateway 130.8.47.171..

I did that but no luck....so revert backup the changes....

--------------------------------------------------------------------------------------------------------------------

Do you have any gateway on 130.8.47.128 which can pass packet to 144.160.231.154? If so, use it as gawate to 144.160.231.154.

No Gateways 130.8.47.128....

----------------------------------------------------------------------------------------------------------------------------------------

Now i an re-sending the output as requested by Nagalla for both Master and Media....Please check and let me know what route needs to be added or removed.

Waiting for the response.....

[RamNagalla]

hi,

1) When you try to add the Gateway 130.8.47.171, did you remove the below route before doing that?

144.160.231.154 130.8.37.1 UGH 1 206

2)I am not seeing any Route in the Media server to reach any of the master Network.

so you would need to add route in the media server as below as a trail.

Destination :-130.8.47.171

Gateway:-144.160.231.154

subnetmask:-255.255.255.255

its a host based route , so subnet mask is all 255., go ahead a and try the above route in media server 

along with the master server route 

destination :-144.160.231.154 

gateway:-130.8.47.171

subnetmask:-255.255.255.255

if this did not work..

then need to look back in how it design level.

route are just elable the gate in host side, unless there is a route define in Network routers, all hosts routes are useless.

so you need to look and involve someone who know you backup Nework Desing.

the master server IP is subnetted network.

130.8.47.171  is sub-network with the mask 255.255.255.128.

do you aware if you have 2 differet Network, few are in 1 Gig network and Few are in 10 Gig network?

besides:- what is the Issue that you are facing becasue of using the IP 130.8.37.177 when its trying to communicate with Media server.

[2013]

From Master when i issue command bptestbpcd its giving the below error:

./bptestbpcd -host mediaserver.enaf.aldc.att.com -verbose
<16>bptestbpcd main: Function ConnectToBPCD(mediaserver.xxxx.xxx.att.com) failed: 46
server not allowed access

afterwards enabled the bpcd logs and from there we are still working on it....
 

[Marianne]

bpcd log on client will tell you which media server IP address is trying to connect to this client.
You will also see in bpcd log if client is able to resolve this IP address to a hostname.
Next, resolved hostname will be compared with Server entries on the client. If client cannot find resolved name in Server list, you will see 'access denied'.

[2013]

Marianne you are absolutely correct.....In my pervious post  i have attached the route output....could you please review once ....