Netbackup 7.5.0.4 Master not communicating with 6.5.4 San media Server

[Sammeer_Pednega]

Hi,

We have upgraded to a new Solaris 10 T4 master server yesterday and we have upgraded Netbackup from 6.5.4 to 7.5.0.4 on the same box.

Now we are facing an issue wherein all the San Media server are facing intermittent corba communication issues wherein backups fail with status 195.

We are not facing any issues on the lan clients.

We have logged a case with technical support which is currently in progress however there is no resolution yet in sight.

Kindly suggest.

[Yasuhisa_Ishika]

Did you replace the existing master server with new T4? Have you already checked network communiation between master server and SAN Media server? Check if SAN media server is listed in master's bp.conf as SERVER.

[Sammeer_Pednega]

Hi Yashuhisa

We replace the existing Master V440 solaris 10 with a new T4 Master Server.

We installed 6.5.4 and performed a catalog recovery on the new T4 Master.

Tested Lan client backups and found no issues.

When we tested San Media server backups it failed with Status 195.

Tried vmoprcmd from Master Server to San media server and got Corba communication errors intermittently.

The only change between the old master and new master is Network Teaming was not present on the old master while the new master we have incorporated teaming.

Please suggest

[thesanman]

You mention you started with NetBackup v6.5.4 on your new hardware; did you test SAN Media server backups after the initial catalog recovery?  Did they work or have they started breaking after the Master Server upgrade to v7.5.0.4?

[Yasuhisa_Ishika]

Tried vmoprcmd from Master Server to San media server and got Corba communication errors intermittently.

intermittently? Not always got failed?
Please check network communication between master server and SAN media server without NetBackup. Try telnet, ftp, ssh or so and observe if any error or significant perforamce degration occurs.

[Yasuhisa_Ishika]

To verify your configuration, please post output of following commands on both host. bpps -x bpclntcmd -self bpclntcmd -pn grep SERVER /usr/openv/netbackup/bp.conf bptestbpcd -verose -host bpclntcmd -hn bpclntcmd -hn

[Sammeer_Pednega]

Tried all above commands  and found no errors

[Yasuhisa_Ishika]

please post output of commands, Sammeer.

[Sammeer_Pednega]

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2012.11.05 14:47:12 =~=~=~=~=~=~=~=~=~=~=~=
clear
# clear
# bpps -x
NB Processes
------------
    root  2330     1   0 12:55:53 ?           0:01 /usr/openv/netbackup/bin/bpcd -standalone
    root  2415     1   0 12:56:01 ?           0:02 /usr/openv/netbackup/bin/nbaudit
    root  2581     1   0 12:56:10 ?           1:20 /usr/openv/netbackup/bin/nbjm
    root  2289     1   0 12:55:53 ?           0:02 /usr/openv/netbackup/bin/vnetd -standalone
    root  2591  2581   0 12:56:11 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbjm
    root  2546     1   0 12:56:05 ?           7:09 /usr/openv/netbackup/bin/nbemm
    root  2411     1   0 12:55:59 ?           0:59 /usr/openv/netbackup/bin/nbevtmgr
    root  2398     1   0 12:55:57 ?           6:04 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/
    root  2588     1   0 12:56:11 ?           0:10 /usr/openv/netbackup/bin/nbpem
    root  2550     1   0 12:56:06 ?           1:22 /usr/openv/netbackup/bin/nbrb
    root  2586  2584   0 12:56:10 ?           0:07 /usr/openv/netbackup/bin/bpjobd
    root  2566     1   0 12:56:09 ?           0:02 /usr/openv/netbackup/bin/bprd
    root  2594     1   0 12:56:11 ?           0:03 /usr/openv/netbackup/bin/nbstserv
    root  2584     1   0 12:56:10 ?           0:01 /usr/openv/netbackup/bin/bpdbm
    root  2572     1   0 12:56:09 ?           0:03 /usr/openv/netbackup/bin/bpcompatd
    root  2626     1   0 12:56:16 ?           0:01 /usr/openv/netbackup/bin/nbars
    root  2623     1   0 12:56:15 ?           0:25 /usr/openv/netbackup/bin/nbsl
    root  2649     1   0 12:56:17 ?           0:00 /usr/openv/netbackup/bin/nbvault
    root  2599     1   0 12:56:12 ?           0:03 /usr/openv/netbackup/bin/nbim
    root  2602     1   0 12:56:13 ?           0:05 /usr/openv/netbackup/bin/nbrmms
    root 11831 11736   0 13:08:34 ?           0:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 C /usr/openv/java/auth.conf 1
    root  2652     1   0 12:56:17 ?           0:00 /usr/openv/netbackup/bin/nbcssc -a NetBackup
    root  2660     1   0 12:56:18 ?           0:01 /usr/openv/netbackup/bin/nbsvcmon
    root  2670  2588   0 12:56:18 ?           0:04 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root 11860  2623   0 13:09:05 ?           0:00 sh -c "/usr/openv/netbackup/bin/nbproxy" dblib -mgrIORFile -PolicyManager-3.ior
    root 12404 12400   0 13:13:22 ?           0:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 C /usr/openv/java/auth.conf 1
    root  6275     1   0 12:57:07 ?           0:00 /usr/openv/netbackup/bin/bpjava-susvc nbhelp -1 -1 C /usr/openv/java/auth.conf
    root 12402 12400   0 13:13:20 ?           0:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 C /usr/openv/java/auth.conf 1
    root 21575  2623   0 14:43:28 ?           0:00 sh -c "/usr/openv/netbackup/bin/nbproxy" dblib -mgrIORFile -StorageService-5.io
    root 12400     1   0 13:13:19 ?           0:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 C /usr/openv/java/auth.conf 1
    root  7287  6275   0 12:57:57 ?           0:00 /usr/openv/netbackup/bin/bpjava-susvc nbhelp -1 -1 C /usr/openv/java/auth.conf
    root 12565 12564   0 13:15:14 ?           0:03 /usr/openv/netbackup/bin/nbproxy dblib -mgrIORFile -PolicyManager-4.ior.mgr -pr
    root 11861 11860   0 13:09:05 ?           0:03 /usr/openv/netbackup/bin/nbproxy dblib -mgrIORFile -PolicyManager-3.ior.mgr -pr
    root 14849     1   0 13:31:15 ?           0:00 bpbrm -backup -S SF440 -c DRDWMSOFF -ct 0 -ru root -cl BCP_MD_W_DRDWMSOFF -sche
    root 21502 14660   0 14:43:04 ?           0:00 /usr/openv/netbackup/bin/bpjava-susvc nbhelp -1 -1 C /usr/openv/java/auth.conf
    root 11863  2623   0 13:09:05 ?           0:00 sh -c "/usr/openv/netbackup/bin/nbproxy" dblib -mgrIORFile -StorageService-3.io
    root 15891 15824   0 13:41:45 ?           1:47 bptm -pid 15800 -den 20 -rt 8 -rn 1 -cj 5 -mpx 5 -reqid -1352100903 -jm -brm -p
    root 21573 21572   0 14:43:28 ?           0:03 /usr/openv/netbackup/bin/nbproxy dblib -mgrIORFile -PolicyManager-5.ior.mgr -pr
    root 14854 14849   0 13:31:18 ?           0:09 bptm -w -c DRDWMSOFF -den 20 -rt 8 -rn 1 -stunit SF440-hcart3-robot-tld-1 -cl B
    root 12744  2588   0 13:16:20 ?           0:01 /usr/openv/netbackup/bin/nbproxy dblib nbpem_cleanup
    root 14664 14660   0 13:29:17 ?           0:00 /usr/openv/netbackup/bin/bpjava-susvc nbhelp -1 -1 C /usr/openv/java/auth.conf
    root 14662 14660   0 13:29:15 ?           0:00 /usr/openv/netbackup/bin/bpjava-susvc nbhelp -1 -1 C /usr/openv/java/auth.conf
    root 15824     1   0 13:41:14 ?           0:08 bptm -pid 15800 -den 20 -rt 8 -rn 1 -cj 5 -mpx 5 -reqid -1352100903 -jm -brm -p
    root 11744 11736   0 13:07:51 ?           0:01 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 C /usr/openv/java/auth.conf 1
    root 11736     1   0 13:07:48 ?           0:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 C /usr/openv/java/auth.conf 1
    root 21577 21575   0 14:43:28 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib -mgrIORFile -StorageService-5.ior.mgr -p
    root 11741 11736   0 13:07:48 ?           0:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 C /usr/openv/java/auth.conf 1
    root 12564  2623   0 13:15:14 ?           0:00 sh -c "/usr/openv/netbackup/bin/nbproxy" dblib -mgrIORFile -PolicyManager-4.ior
    root 11865 11863   0 13:09:05 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib -mgrIORFile -StorageService-3.ior.mgr -p
    root 15578     1   0 13:37:45 ?           0:01 bpbrm -backup -S SF440 -c vldb02 -ct 13 -ru root -cl BCP_D_vldb02 -sched D_vldb
    root 15583     1   0 13:37:46 ?           0:10 bptm -pid 15578 -den 20 -rt 8 -rn 1 -cj 5 -mpx 5 -reqid -1352100902 -jm -brm -p
    root 20751 20749   0 14:35:25 ?           0:00 /usr/openv/netbackup/bin/bpjava-susvc dpcbcp -1 -1 C /usr/openv/java/auth.conf
    root 20749     1   0 14:35:25 ?           0:00 /usr/openv/netbackup/bin/bpjava-susvc dpcbcp -1 -1 C /usr/openv/java/auth.conf
    root 14856 14854   0 13:31:19 ?           1:30 bptm -w -c DRDWMSOFF -den 20 -rt 8 -rn 1 -stunit SF440-hcart3-robot-tld-1 -cl B
    root 15586 15578   0 13:37:47 ?           0:00 bpbrm -backup -S SF440 -c vldb02 -ct 13 -ru root -cl BCP_D_vldb02 -sched D_vldb
    root 15585 15583   0 13:37:47 ?           1:28 bptm -pid 15578 -den 20 -rt 8 -rn 1 -cj 5 -mpx 5 -reqid -1352100902 -jm -brm -p
    root 15800     1   0 13:40:51 ?           0:01 bpbrm -backup -S SF440 -c INFORAPP -ct 13 -ru root -cl BCP_D_INFORAPP -sched D_
    root 14660     1   0 13:29:15 ?           0:00 /usr/openv/netbackup/bin/bpjava-susvc nbhelp -1 -1 C /usr/openv/java/auth.conf
    root 15892 15800   0 13:41:45 ?           0:00 bpbrm -backup -S SF440 -c INFORAPP -ct 13 -ru root -cl BCP_D_INFORAPP -sched D_
    root 21506 14660   0 14:43:14 ?           0:00 /usr/openv/netbackup/bin/bpjava-susvc nbhelp -1 -1 C /usr/openv/java/auth.conf
    root 21572  2623   0 14:43:28 ?           0:00 sh -c "/usr/openv/netbackup/bin/nbproxy" dblib -mgrIORFile -PolicyManager-5.ior

MM Processes
------------
    root  2556     1   0 12:56:07 ?           0:01 /usr/openv/volmgr/bin/ltid
    root  2562     1   0 12:56:08 ?           0:01 vmd -v
    root  7224  2556   0 12:57:45 ?           0:00 avrd -v
    root  7244     1   0 12:57:46 ?           0:01 tldcd -v
    root  7221  2556   0 12:57:43 ?           0:00 tldd -v
    root 21879 21878   0 14:44:17 ?           0:00 /usr/openv/volmgr/bin/vmoprcmd -h HBFCDRODGBKPDB1 -extall

Shared Symantec Processes
-------------------------
    root  2261     1   0 12:55:48 ?           0:03 /opt/VRTSpbx/bin/pbx_exchange
# bpclntcmd -self
yp_get_default_domain failed: (6) internal yp server or client error
NIS does not seem to be running: (1) args to yp function are bad
gethostname() returned: SF440
host SF440: SF440 at 10.225.4.44
aliases:     SF440     10.225.4.44
getfqdn(SF440) returned: sf440.com
# bpclntcmd -self    pn
expecting response from server SF440
SF440 SF440 10.225.4.44 43229
# bpclntcmd -pn\\\               
#
# grep

[Marianne]

Try nbdna and/or bptestnetconn to test network connectivity.

http://www.symantec.com/docs/HOWTO43776 

http://www.symantec.com/docs/HOWTO43773

[Sammeer_Pednega]

Hi Marianne

We have backline Engg with us who have found there are servers which are flooding the master server with pings.

We have found there are four servers which are causing this scenario and they no longer exist in  netbackup configuration. We do not have access to these servers hence would like to know how to block these from pinging the netbackup master.

Regards

[badri]

i think the network guys can help you here. they can perhaps block connection or may be you can use the OS firewall setting to block request from those servers