Status Codes 205 and 219: Is my master server run out of its network sockets

[Anton_Panyushki]

Hello,

I've got backup jobs that abnormally finish with status codes 205 and 219. When examining bpsched log on master server I came accross the following messages:

05:01:59.400 [12398] <2> logconnections: BPDBM CONNECT FROM XXX.YYY.184.144.34675 TO XXX.YYY.184.144.13721
05:01:59.499 [12398] <2> nb_getsockconnected: rresvport() failed, Address already in use (125)
05:01:59.499 [12398] <2> bpcr_connect: Can't connect to client client-ge0
05:01:59.500 [12398] <16> start_bpbrm: cannot connect to client-ge0
05:01:59.500 [12398] <16> start_bpbrm: bpcd exit:  cannot connect to server backup restore manager (205)
05:01:59.511 [12398] <2> nb_getsockconnected: rresvport() failed, Address already in use (125)
05:01:59.511 [12398] <2> bpcr_connect: Can't connect to client client-ge0
05:01:59.511 [12398] <2> append_to_client_log: cannot connect to bpcd on client client-ge0, Address already in use
05:01:59.511 [12398] <8> process_new_job: failure logging message to client client-ge0 in log /usr/openv/netbackup/logs/user_ops/dbext/logs/7429.0.1186362109:  can't connect to client (58)
05:01:59.511 [12398] <16> log_in_errorDB: failure logging message to client client-ge0 in log /usr/openv/netbackup/logs/user_ops/dbext/logs/7429.0.1186362109:  can't connect to client (58)
05:01:59.514 [12398] <32> child_bpsched: error processing first new job:  cannot connect to server backup restore manager (205)
05:01:59.531 [29067] <2> set_job_details: Done
05:01:59.531 [29067] <2> job_end_try: Done
05:01:59.531 [29067] <2> ?: ------------------------------------
05:01:59.531 [29067] <2> ?: <<  CLIENT: client-ge0 (oracle) BIRTH: 1186362119  PRIORITY: 50
05:01:59.532 [29067] <2> ?:     POLICY: UNX-ORA-SASMA-ARC  SCHED: Default-Application-Backup(1)  PART 0 JOBID 3585646 GRPID 3585646
05:01:59.532 [29067] <2> ?:     STUNIT: client-ge0-hcart2-robot-tld-0 (client-ge0)
05:01:59.532 [29067] <2> ?:     PID: 12398  STATUS: 205  TRIES: 1  SPID: 12389 (UB)
05:01:59.532 [29067] <2> ?: ------------------------------------
05:01:59.532 [29067] <2> ?: client-ge0 exited with status 205 (cannot connect to server backup restore manager)
05:01:59.545 [29067] <2> job_enddata: Done
05:01:59.560 [29067] <2> nb_getsockconnected: rresvport() failed, Address already in use (125)
05:01:59.560 [29067] <2> bpcr_connect: Can't connect to client client-ge0
05:01:59.560 [29067] <16> start_bptm: cannot connect to client-ge0
05:01:59.560 [29067] <16> start_bptm: bpcd exit:  cannot connect to server backup restore manager (205)
05:01:59.560 [29067] <16> log_in_errorDB: get_num_avail_drives failed with stat 205 for stunit client-ge0-hcart2-robot-tld-0
05:01:59.564 [29067] <16> process_job_complete: get_num_avail_drives failed with stat 205 for stunit client-ge0-hcart2-robot-tld-0
05:01:59.565 [12389] <16> log_in_errorDB: backup by oracle on client client-ge0 using policy UNX-ORA-SASMA-ARC:  cannot connect to server backup restore manager
05:01:59.574 [12389] <16> bpsched: backup by oracle on client client-ge0 using policy UNX-ORA-SASMA-ARC:  cannot connect to server backup restore manager
05:01:59.575 [29067] <2> run_backups: number_of_stunits = 22
05:01:59.589 [12515] <2> unmount_all_tapes: initiating bptm -U on client-ge0
05:01:59.590 [12515] <2> bpcr_ex_client_cmd: getservbyname returned port 13782
05:01:59.591 [12389] <2> nb_getsockconnected: Connect to client-ge0 on port 982
05:01:59.592 [12389] <2> logconnections: BPCD CONNECT FROM XXX.YYY.184.144.982 TO XXX.YYY.184.40.13782
05:01:59.594 [12515] <2> nb_getsockconnected: Connect to client-ge0 on port 760
05:01:59.594 [12515] <2> logconnections: BPCD CONNECT FROM XXX.YYY.184.144.760 TO XXX.YYY.184.40.13782
05:01:59.608 [12389] <2> append_to_client_log: cannot connect to bpcd on client client-ge0, Address already in use
05:01:59.608 [12389] <16> log_in_errorDB: failure logging message to client client-ge0 in log /usr/openv/netbackup/logs/user_ops/dbext/logs/7429.0.1186362109:  can't connect to client (58)
05:01:59.609 [12515] <2> bpcr_ex_client_cmd: Can't connect to client-ge0

I don't know if master server is running of its sockets or there is some problem with IP adresses duplication.

[Tekkdude]

As you have already figured out, it looks like a socket issue. Try this technote:

http://support.veritas.com/docs/247262 . It should help.

Jeff Redington

[Anton_Panyushki]

Hi, Jeff.

I've already seen that technote. You see this errors don't ocuur during vaulting and that puzzles me.

[Jeff_Redington]

Based on the error it looks like something else is trying to use the ports that NetBackup is using. You could check \etc\services and see if anything is trying to reserve NetBackup ports. Or you could try stopping all non-essential services on the server and see if it happens again. Also, try running netstat -a and see what is keeping the port open.

Jeff

[Stumpr2]

Jeff,

Welcome back :)

How come you have credit for so few posts? Man, we were running neck-to-neck a few years back when they had the points system. You must have answered at least 500 posts during that time.

[adwhite]

G'day,

I'm having the same problems as well - It seems to only be on busy schedule nights where a few hundred jobs will kick off at once.

Did you have any luck with it?

Cheers

Andrew

[redfox]

hi Andrew, did you fix this rc219 issue  and how did you fix it, could you post it out?