Backup for SAN client is failing with Error #11: system call failed

[JollyJ]

Master server : Win2K8 NBU :7.5.0.4

FT Media server : Solaris 10  NBU :7.5.0.4

SAN Client : Linux  2.6.9-103.ELsmp NBU :6.5.5

Since I have upgraded my master and Media servers to 7.5.0.4 my FT backup has stopped working.

I ran below cammand from my FT media server but I am getting error. However cammands are successful from master server.

bash-3.00# ./nbftconfig -getconfig -verbose
Encountered with an error, check the log for details

bash-3.00# ./nbftconfig -listtargets -verbose
Encountered with an error, check the log for details

bash-3.00# ./nbftconfig -rescanclient esidwdlp01
;Encountered with an error, check the log for details

bash-3.00# ./nbhba -l
Unable to communicate with HBA ports.
Please ensure that this system is set up correctly with nbftsrv_config.
(Must not be set up as FT Server.)
Error #11: system call failed

Job details:

6/28/2013 11:25:17 AM - Info nbjm(pid=15216) starting backup job (jobid=109117) for client esidwdlp01, policy EDW_esidwdlp01_Apps, schedule UNIX-Full 
6/28/2013 11:25:17 AM - Info nbjm(pid=15216) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=109117, request id:{9876B71C-AF8A-4AE3-ADF4-71015C5791F5}) 
6/28/2013 11:25:17 AM - requesting resource esibkpmedia04-robot-tld-0
6/28/2013 11:25:17 AM - requesting resource esibkpmaster01.NBU_CLIENT.MAXJOBS.esidwdlp01
6/28/2013 11:25:17 AM - requesting resource esibkpmaster01.NBU_POLICY.MAXJOBS.EDW_esidwdlp01_Apps
6/28/2013 11:25:19 AM - granted resource esibkpmaster01.NBU_CLIENT.MAXJOBS.esidwdlp01
6/28/2013 11:25:19 AM - granted resource esibkpmaster01.NBU_POLICY.MAXJOBS.EDW_esidwdlp01_Apps
6/28/2013 11:25:19 AM - granted resource HC4284
6/28/2013 11:25:19 AM - granted resource HP.ULTRIUM3-SCSI.005
6/28/2013 11:25:19 AM - granted resource esibkpmedia04-robot-tld-0
6/28/2013 11:25:19 AM - granted resource TRANSPORT
6/28/2013 11:25:19 AM - estimated 1818122073 Kbytes needed
6/28/2013 11:25:19 AM - Info nbjm(pid=15216) started backup (backupid=esidwdlp01_1372433119) job for client esidwdlp01, policy EDW_esidwdlp01_Apps, schedule UNIX-Full on storage unit esibkpmedia04-robot-tld-0
6/28/2013 11:25:21 AM - started process bpbrm (12428)
6/28/2013 11:25:22 AM - end writing
6/28/2013 12:56:19 PM - Error bpbrm(pid=12428) bpcd on esibkpmedia04 exited with status 59: access to the client was not allowed
6/28/2013 12:56:20 PM - Critical bpbrm(pid=12428) could not set non-blocking I/O on media socket, Bad file number (9)

Output from master server:

SAN Client Name      : esidwdlp01
SAN Client Version   : 6.5
SAN Client State     : active
Master Server Name   : esibkpmaster01
FT Server Connections: 1
Client Ports/Server  : 2
Usage Preference     : preferred
Backup Wait Time     : 15
Restore Wait Time    : 5
SAN Client HBA Port  : 0
        SAN Client Device State: active
        Media Server Name      : esibkpmedia04
        Media Sever State      : active
        Media Server HBA Port  : 0
        Media Server Port Mode : FABRIC
        Media Server LUN       : 0

SAN Client HBA Port  : 16
        SAN Client Device State: active
        Media Server Name      : esibkpmedia04
        Media Sever State      : active
        Media Server HBA Port  : 1
        Media Server Port Mode : FABRIC
        Media Server LUN       : 0

[huanglao2002]

If your master server is configure as cluster.

please add the physical and virtual host resolve to /etc/host and must add " physical and virtual host name" to bp.conf.

[V4]

verify if FT and SAN cilent services are running. (try recycling them if you observe they are stalled)

Also verify from zoning part. Try removing and creating zones if possible

We had incidents where we had to re-install OS and then build it back from scratch to bring back healthy state. It was OS which was not behaving properly. 

Do follow release notes from 7.5.0.4  (also try to update client to same release and check)

If SAN clients are cluster, then don't forget to add them under nbemmcmd as app_cluster

also add virtual name to Device -> SAN client

[JollyJ]

Thanks for your reply,

Master server is not in cluster.

# bpclntcmd -pn  out fine at client side. its not host resolution problem because

  host file are updated on master/media and at client end.

To add it to more information  I have two FT media server and non of the SAN client backup are using FT.

here is the  output from master server for perticular client.

E:\VERITAS\NetBackup\bin\admincmd>nbftconfig -lc -verbose

SAN Client Name      : <Client Name>
SAN Client Version   : 6.5
SAN Client State     : active
Master Server Name   : <Master server>
FT Server Connections: 1
Client Ports/Server  : 2
Usage Preference     : preferred
Backup Wait Time     : 15
Restore Wait Time    : 5
SAN Client HBA Port  : 0
        SAN Client Device State: active
        Media Server Name      : <Media server>
        Media Sever State      : active
        Media Server HBA Port  : 0
        Media Server Port Mode : FABRIC
        Media Server LUN       : 0

SAN Client HBA Port  : 16
        SAN Client Device State: active
        Media Server Name      : <Media server>
        Media Sever State      : active
        Media Server HBA Port  : 1
        Media Server Port Mode : FABRIC
        Media Server LUN       : 0

[watsons]

Your client is on 6.5.5, any reason why not to upgrade it to 7.x?

There was a bug in 6.5.6 (possibly affect 6.5.5 too) that may affect FT device discovery. Look for other bugs in release 6.5.6 too if this does not match.

http://www.symantec.com/docs/TECH157326

[JollyJ]

Working on reconfiguring FT media server.

[Marianne]

You have hopefully upgraded your clients in the meantime to NBU 7.5.0.4 as well?

[JollyJ]

Yes, I am targetting them as well.

[JollyJ]

I Configured the FT media server, however the WWN number of HBA detected by netbackup is different than that visible to OS. I am unable to get good techsupport guy to work on it as well. #fcinfo hba-port | grep WWN HBA Port WWN: 23000003ba9b25c7 Node WWN: 22000003ba9b25c7 HBA Port WWN: 10000000c95e50d3 Node WWN: 20000000c95e50d3 HBA Port WWN: 10000000c998fa72 Node WWN: 20000000c998fa72 bash-3.00# bash-3.00# /usr/openv/netbackup/bin/admincmd/nbhba -l 1 2422 21:00:00:1B:32:9D:73:C7 "QLA2462 " 0 1 8134 2 2422 21:01:00:1B:32:BD:73:C7 "QLA2462 " 1 1 8134