Active Directory GRT restore : socket open failed

[yguilloux]

Hello

I have configured a GRT backup of a seconday AD server, NBU services are running under a domain administrator service account, NFS client services is enabled on the AD server.

The GRT backup is running fine, but when I try to browse the active directory tree in BAR interface , i get the following message : ERROR : socket open failed. (using master BAR or AD server BAR)

Any help would be appreciated

Thanks in advance

Best regards

[RiaanBadenhorst]

Did you open the BAR with "run as administrtor"?

Did the backup exit with status ZERO and contain the message saying that granular processing completed successfully?

HAve you tried to mount the image using the process in this note https://support.symantec.com/en_US/article.TECH124810.html ?

[yguilloux]

Hello

yes BAR was opened "as administrator"

Th exit staus of the backup is 0 and i have the following line in the detailed status :

 Info bpbrm(pid=152579) from client svbll-dw1: TRV - Successfully finished granular backup processing for (System State\Active Directory)

I successfully have mouted the NFS export using the technote you provided ....

best regards

[Marianne]

Do you have bprd log folder on the master?
If not, please create the folder and restart NBU.
Try to browse again.
After the error, please copy the log to bprd.txt and upload as File attachment.

One more thought:

Have you verified that these firewall ports are open?

■ TCP port 111 (portmapper) needs to be open from the client to the media server.
■ TCP port 3794 (nbfsd) needs to be open from the client to the media server.

Check Windows firewall as well.

[yguilloux]

in addition, each time i am trying to expand the "active directory" tree, a "restore job" is ran (?!?!) :

02/09/2015 09:43:57 - requesting resource @aaaag
02/09/2015 09:43:57 - granted resource MediaID=@aaaag;DiskVolume=PureDiskVolume;DiskPool=ms01_sgie-pril-inf38;Path=PureDiskVolume;StorageServer=sgie-pril-inf38;MediaServer=svie-priw-inf35
the requested operation was successfully completed (0)

[yguilloux]

Hello Marianne,

windows firewall is disabled, and both ports are opened

The bprd.txt file is attached to this post

[Marianne]

Something wrong with socket connections on the master server - bprd -> bpdbm using 13721 instead of 1556?

09:55:55.558 [4764.900] <2> async_connect: [vnet_connect.c:1691] connect async 
CONNECT FROM 10.224.48.9.51366 TO 10.224.48.9.13721

09:55:55.667 [4764.900] <2> fileslist: begin db communication
09:55:55.667 [4764.900] <2> fileslist: criteria sent to db mgr
09:55:58.464 [4764.900] <2> isDONE: reply DONE 21 <?xml version="1.0" encoding="utf-8"?><StatusMsgDoc><StatusMsg NBUStatus="21" Severity="2" VXULoid="271" VXULtid="21"><MsgString>socket&#32;open&#32;failed</MsgString></StatusMsg></StatusMsgDoc>
09:55:58.464 [4764.900] <2> fileslist: unexpected return value from db_FLISTreceive: socket open failed 21
09:55:58.464 [4764.900] <2> process_request: fileslist failed - EC status = socket open failed
09:55:58.464 [4764.900] <2> process_request: EXIT STATUS 21

I have no explanation for this.
Are there perhaps any 'Connect Options' configured for the master server?

[yguilloux]

no "connect options" configured ... and telnet <master> 1556 from the client is working ....

[Marianne]

The issue is with port connection on the master - from itself to itself.

bprd (on the master) is trying to connect to connect to bpdbm (on the master) on port 13721 instead of 1556 (note the 'from' and 'to' IP address): 

CONNECT FROM 10.224.48.9.51366 TO 10.224.48.9.13721 

[RiaanBadenhorst]

Can you post the nbfsd log too please.

[yguilloux]

i have tried bptestbpcd from the master to itself :

C:\WINDOWS\system32>bptestbpcd -debug -verbose -client svie-priw-inf35
11:10:06.457 [408.6892] <2> bptestbpcd: VERBOSE = 0
11:10:06.457 [408.6892] <8> cache_item_to_file: [vnet_addrinfo.c:6434] rename() failed 13 0xd
11:10:06.457 [408.6892] <8> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1705] cache_item_to_file() failed 0 0x0
11:10:06.457 [408.6892] <2> logconnections: BPCD CONNECT FROM 10.224.48.9.51110 TO 10.224.48.9.13782 fd = 520
11:10:06.551 [408.6892] <2> bpcr_get_version_rqst: bpcd version: 07610001
1 0 2
10.224.48.9:51110 -> 10.224.48.9:13782
10.224.48.9:4829 <- 10.224.48.9:51111
11:10:06.551 [408.6892] <2> bpcr_get_peername_rqst: Server peername length = 25
11:10:06.551 [408.6892] <2> bpcr_get_hostname_rqst: Server hostname length = 15
11:10:06.551 [408.6892] <2> bpcr_get_clientname_rqst: Server clientname length = 15
11:10:06.551 [408.6892] <2> bpcr_get_version_rqst: bpcd version: 07610001
11:10:06.551 [408.6892] <2> bpcr_get_platform_rqst: Server platform length = 7
11:10:06.551 [408.6892] <2> bpcr_get_version_rqst: bpcd version: 07610001
11:10:06.551 [408.6892] <2> bpcr_patch_version_rqst: theRest == > <
11:10:06.551 [408.6892] <2> bpcr_get_version_rqst: bpcd version: 07610001
11:10:06.551 [408.6892] <2> bpcr_patch_version_rqst: theRest == > <
11:10:06.551 [408.6892] <2> bpcr_get_version_rqst: bpcd version: 07610001
PEER_NAME = SVIE-PRIW-INF35.sie.local
HOST_NAME = svie-priw-inf35
CLIENT_NAME = svie-priw-inf35
VERSION = 0x07610001
PLATFORM = win_x64
PATCH_VERSION = 7.6.1.2
SERVER_PATCH_VERSION = 7.6.1.2
MASTER_SERVER = svie-priw-inf35
EMM_SERVER = svie-priw-inf35
NB_MACHINE_TYPE = MASTER_SERVER
10.224.48.9:1460 <- 10.224.48.9:51112
<2>bptestbpcd: EXIT status = 0
11:10:06.598 [408.6892] <2> bptestbpcd: EXIT status = 0

please find below the nbfsd log of the client

[Marianne]

NOT the default connect options:

1 0 2

Extract from in-house bptestbpcd:

1 1 1
10.5.1.190:57697 -> 10.5.1.190:1556
10.5.1.190:57698 -> 10.5.1.190:1556
10.5.1.190:57701 -> 10.5.1.190:1556 

Extract from Commands manual:

■ The first digit is 0 if reserved ports are used and 1 if non-reserved ports are used.
■ The second digit is 0 if you use legacy (random port) callback and 1 if you use vnetd callback.
■ The third digit is 1 if the connection is initiated on the PBX or vnetd port number.
   The third digit is 2 if the connection is initiated on the legacy bpcd port number. 

The default is 1 1 1.

You can see that your master is not using defaults.

[yguilloux]

Really strange ... I had nothing in master server properties/client attributes ...

I just added master and AD server in client list and forced BPCD connect back to "VNETD port" and Daemon connection port to "VNETD Only"

So now the bptestbpcd looks better :

1 1 1
10.224.48.9:52447 -> 10.224.48.9:1556
10.224.48.9:52448 -> 10.224.48.9:1556
10.224.48.9:52451 -> 10.224.48.9:1556

But BAR browsing of active directory is still providing the same error and the bprd log is the same :

11:48:05.634 [7528.5032] <2> ConnectionCache::connectAndCache: Acquiring new connection for host svie-priw-inf35, query type 79
11:48:05.634 [7528.5032] <2> vnet_sortaddrs: [vnet_addrinfo.c:3852] sorted addrs: 1 0x1
11:48:05.634 [7528.5032] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4717] Local [strong] check, using interface  ANY
11:48:05.634 [7528.5032] <2> async_connect: [vnet_connect.c:1499] connect in progress 1 0x1
11:48:05.634 [7528.5032] <2> async_connect: [vnet_connect.c:1691] connect async CONNECT FROM 10.224.48.9.52413 TO 10.224.48.9.13721 fd = 28
11:48:05.634 [7528.5032] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO svie-priw-inf35 10.224.48.9 bpdbm
11:48:05.634 [7528.5032] <2> logconnections: BPDBM CONNECT FROM 10.224.48.9.52413 TO 10.224.48.9.13721 fd = 28
11:48:05.634 [7528.5032] <2> vnet_check_vxss_client_magic_with_info_ex: [vnet_vxss_helper.c:939] Ignoring VxSS authentication 2 0x2
11:48:05.743 [7528.5032] <2> fileslist: begin db communication
11:48:05.743 [7528.5032] <2> fileslist: criteria sent to db mgr
11:48:08.571 [7528.5032] <2> isDONE: reply DONE 21 <?xml version="1.0" encoding="utf-8"?><StatusMsgDoc><StatusMsg NBUStatus="21" Severity="2" VXULoid="271" VXULtid="21"><MsgString>socket&#32;open&#32;failed</MsgString></StatusMsg></StatusMsgDoc>
11:48:08.587 [7528.5032] <2> fileslist: unexpected return value from db_FLISTreceive: socket open failed 21
11:48:08.587 [7528.5032] <2> process_request: fileslist failed - EC status = socket open failed
11:48:08.587 [7528.5032] <2> process_request: EXIT STATUS 21
11:48:08.712 [7528.5032] <2> bprd: D:\Program Files\Veritas\NetBackup\bin\userreq_notify.cmd list svbll-dw1 root

[RiaanBadenhorst]

Maybe the same properties are incorrectly set on the media server?

[Marianne]

Still the same: 

CONNECT FROM 10.224.48.9.52413 TO 10.224.48.9.13721 fd = 28 
.....
fileslist: unexpected return value from db_FLISTreceive: socket open failed 21

I suggest that you log a Support call with Symantec/Veritas.

Extract from 'working' bprd-> bpdbm connection:

10:45:55.175 [6756.8156] <2> logconnections: BPDBM CONNECT FROM 158.119.166.62.59107 TO 158.119.166.62.1556 fd = 512
....
10:45:55.237 [6756.8156] <2> fileslist: begin db communication
10:45:55.237 [6756.8156] <2> fileslist: criteria sent to db mgr
10:45:55.299 [6756.8156] <2> process_request: EXIT STATUS 0

[yguilloux]

media server does not seem to be involved at this point, but bptestbpcd on media server is ok :

1 1 1
10.224.48.9:53120 -> 10.224.48.193:1556
10.224.48.9:53121 -> 10.224.48.193:1556
10.224.48.9:53122 -> 10.224.48.193:1556

[yguilloux]

i restarted one more time the NB agent on AD server and it is working now ! even if still connecting on port 13721 ...

I really don't understand what happend ....

But thanks for helping !

Best regards

NEW EDIT : it just stopped working ....

I have asked the AD server to be rebooted as i was told that after activating NFS client service it might be required ...

[yguilloux · Accepted solution]

a colleague of mine suggested to create the file USE_BACKUP_MEDIA_SERVER_FOR_RESTORE in <NBU_ome>\db\config and this worked !!!

I haven't figured out why, but it is really working , the restore ran successfully and browsing is working everytime !

[nurgul]

great solution :) its works for us as well. NetBackup should add this parameter by default.