Netbackup 7.0.1 - Catalog backup problem

[Sega_Ken_Chan]

Dear supports,

I am facing a catalog backup problem in these two days, sytem nothing had been changed included network.

Can help to solve it ?

There are the message prompted:

10/8/2013 9:01:55 AM - requesting resource P7BU0101-PDC-VTL-1
10/8/2013 9:01:55 AM - requesting resource p7bu0101.NBU_CLIENT.MAXJOBS.p7bu0101
10/8/2013 9:01:55 AM - requesting resource p7bu0101.NBU_POLICY.MAXJOBS.HotCatalogBackupToVTL
10/8/2013 9:01:55 AM - granted resource p7bu0101.NBU_CLIENT.MAXJOBS.p7bu0101
10/8/2013 9:01:55 AM - granted resource p7bu0101.NBU_POLICY.MAXJOBS.HotCatalogBackupToVTL
10/8/2013 9:01:55 AM - granted resource DV5399
10/8/2013 9:01:55 AM - granted resource p7bu0101_TLD_hcart_Drive_007
10/8/2013 9:01:55 AM - granted resource p7bu0101-hcart-robot-tld-0
10/8/2013 9:01:55 AM - estimated 0 Kbytes needed
10/8/2013 9:01:55 AM - started process bpbrm (5736)
10/8/2013 9:01:55 AM - connecting
10/8/2013 9:01:56 AM - Error bpbrm(pid=5736) cannot open progress file /D/Program?Files/Veritas/NetBackup/Logs/user_ops/dbext/logs/vxbsa.1381194111.3308.6888.prog.pcb_std on client p7bu0101     
10/8/2013 9:01:56 AM - Error bpbrm(pid=5736) client open errno = 22        
10/8/2013 9:01:56 AM - Error bpbrm(pid=5736) listen for client protocol error - couldn't write necessary information on /D/Program?Files/Veritas/NetBackup/Logs/user_ops/dbext/logs/vxbsa.1381194111.3308.6888.prog.pcb_std
10/8/2013 9:01:56 AM - end writing
cannot connect on socket(25)

Many thanks.

Ken

[Sega_Ken_Chan]

I checked that there is not file permission problem, attached the vxbsa.prog and bpbrm log file for your reference.

Many thanks.

[Marianne]

So, p7bu0101 is the master server and is also the media server?

We see in vxbsa log that the master server cannot to itself.

Server status = 25
Backup by SYSTEM on client p7bu0101 using policy HotCatalogBackupToVTL, sched Full:cannot connect on socket

Have you tried to stop and restart all NBU processes/services, including PBX?

That would be the first thing that I will try, followed be a reboot.

If restart or reboot does not solve the problem, we need to get to cmd and perform name lookup and connection tests to find out where the problem is.

**** EDIT ****

See these 2 TNs:
check output of  netstat -an  on the master server to see if there is a large number of ports in TIME_WAIT status.

A reboot will be needed to clear this.

Adjust TcpTimedWaitDelay as per the TNs.

http://www.symantec.com/docs/TECH50811

http://www.symantec.com/docs/TECH29790

[Sega_Ken_Chan]

p7bu0101 is master server, we owned another one for media server.

I'm also strange that it can't connected by itself. I tried to reboot, but still failed.

How the prove the connection issue?  connection between media and master server ?

nslookup:

c:\nslookup p7bu0101

Name:    p7bu0101.tfprod.taifook.com
Address:  172.28.21.151

[Marianne]

Nothing wrong with netstat output.

Have you tried to backup something else on the master to this STU?
Something small like a TEMP folder using a MS-Windows policy?

We need to see if the problem is with Catalog backup only or a general problem with master internal comms.

How many network cards in the master server?

Check output of commands such as:

bpclntcmd -self

bpclntcmd -pn

nbemmcmd -listhosts -verbose

nbemmcmd -getemmserver 

**** EDIT ****

I just had another look at bpbrm log:

hosts_equal: name1=tfdb1  name2=p7bu0101

Please help us to understand where these 2 hostnames fit in?

And then seemingly 2 IP addresses tied to the same hostname?

hosts_equal: host2 p7bu0101 P7BU0101.tfprod.taifook.com addr 172.28.10.11 (0xb0a1cac)
hosts_equal: host2 p7bu0101 P7BU0101.tfprod.taifook.com addr 172.28.21.151 (0x97151cac)

[Sega_Ken_Chan]

D:\Program Files\Veritas\NetBackup\bin>bpclntcmd -self
gethostname() returned: p7bu0101
host p7bu0101: p7bu0101 at 172.28.21.151
aliases:     p7bu0101     172.28.21.151

D:\Program Files\Veritas\NetBackup\bin>bpclntcmd -pn
expecting response from server p7bu0101
p7bu0101 p7bu0101 172.28.21.151 52737

No this command "nbemmcmd"

The master server got two IP, 172.28.21.151 is internal network and 172.28.10.11 is backup network

[Marianne]

nbemmcmd is in D:\Program Files\Veritas\NetBackup\bin\admincmd.

Are separate hostnames configured for the 2 NICs/IPs?

[Sega_Ken_Chan]

no, only one server name "p7bu0101"

For some 172.28.10.x client, we use the 172.28.10.11 network to backup.

For the others, will through 172.28.21.151

[Marianne]

Please bind 172.28.10.11 on the master server to a different hostname.

TCP/IP on the master server will round-robin between the 2 IP addresses for internal comms on the master server.

Please add the following entries to hosts file on the master server:

172.28.21.151  p7bu0101 
172.28.10.11  p7bu0101-bkp

Run 'bpclntcmd -clear_host_cache', then try to manually run Catalog backup.

If this does not solve the problem, please log a Support call with Symantec. Your master server is at risk as long as Catalog backup is unsuccessful.