Cross restore problme. cannot write data to socket, 10054

[Ramiro-Magan]

Hi again.

So, I am trying to do a cross restore, but keep getting this error message. I have tried a bunch of things, yet nothing seems to work.

Server: Windows 2008 R2, Netbackup 7.1.0.4

Source client: windows 2003, netbackup 7.1.0.4 client. Physical server.

Destination client: Windows 2008 R2, Netbackup 7.1.0.4. Virtual server, with VMware. Has a 1tb lun assigned.

This is a file server I am trying to virtualize, so I want to move all my data from the source to the destination.

I tried restoring files from the source to the source, and the destination, to the destination. No problem there. 

But each time I try to go from Source to Destination, I keep getting the same error:

22/08/2012 05:26:46 p.m. - begin Restore

22/08/2012 05:28:45 p.m. - 1 images required

22/08/2012 05:28:45 p.m. - media HPZ950 required

22/08/2012 05:28:45 p.m. - media HPZ963 required

22/08/2012 05:28:48 p.m. - restoring image kamino_1345355900

22/08/2012 05:28:49 p.m. - Info bpbrm(pid=7872) korasa is the host to restore to      

22/08/2012 05:28:51 p.m. - Info bpbrm(pid=7872) telling media manager to start restore on client     

22/08/2012 05:28:53 p.m. - Info bptm(pid=5528) Waiting for mount of media id HPZ950 (copy 1) on server myarsw00403bk01. 

22/08/2012 05:28:53 p.m. - started process bptm (5528)

22/08/2012 05:28:53 p.m. - mounting HPZ950

22/08/2012 05:28:53 p.m. - Info bptm(pid=5528) INF - Waiting for mount of media id HPZ950 on server myarsw00403bk01 for reading.

22/08/2012 05:28:53 p.m. - requesting resource HPZ950

22/08/2012 05:28:53 p.m. - granted resource HPZ950

22/08/2012 05:28:53 p.m. - granted resource HP.ULTRIUM4-SCSI.000

22/08/2012 05:28:54 p.m. - Info bpbrm(pid=7120) korasa is the host to restore to      

22/08/2012 05:28:56 p.m. - connecting

22/08/2012 05:28:57 p.m. - Info bpbrm(pid=7120) start tar32 on client         

22/08/2012 05:28:59 p.m. - Info tar32(pid=3644) Restore started.           

22/08/2012 05:28:59 p.m. - connected; connect time: 00:00:03

22/08/2012 05:29:33 p.m. - mounted; mount time: 00:00:40

22/08/2012 05:29:33 p.m. - Info bptm(pid=5528) HPZ950            

22/08/2012 05:29:34 p.m. - Info bptm(pid=5528) INF - Waiting for positioning of media id HPZ950 on server myarsw00403bk01 for reading.

22/08/2012 05:29:34 p.m. - positioning HPZ950 to file 81

22/08/2012 05:30:50 p.m. - positioned HPZ950; position time: 00:01:16

22/08/2012 05:30:52 p.m. - begin reading

22/08/2012 05:30:53 p.m. - Error bptm(pid=5516) cannot write data to socket, 10054       

22/08/2012 05:30:53 p.m. - Info bpbrm(pid=7872) child done, status 13         

22/08/2012 05:30:53 p.m. - Info bpbrm(pid=7872) sending message to media manager: STOP RESTORE kamino_1345355900     

22/08/2012 05:30:53 p.m. - Error bptm(pid=5516) The following files/folders were not restored:     

********************** a bunch of files ******************

22/08/2012 05:30:53 p.m. - Error bptm(pid=5516) more than 10 files were not restored, remaining ones are shown in the progress log.

22/08/2012 05:32:56 p.m. - Warning bprd(pid=5896) Restore must be resumed prior to first image expiration on 19/09/2012 02:58:20 a.m.

22/08/2012 05:32:56 p.m. - end Restore; elapsed time: 00:06:10

socket write failed(24)

22/08/2012 05:32:56 p.m. - Info bpbrm(pid=7872) media manager for backup id kamino_1345355900 exited with status 150: termination requested by administrator

****************************************************************

So now I am stuck, tried opening the firewall, putting everything in the host file, tried bptestbpcd, bpclntcmd, bpcoverage, bpgetconfig, and I can see every single thing, yet, I can not do a restore. The backups work ok.

What might I have missed? Please assist.

Thanks.

[Ramiro-Magan]

oh, one more thing.

The server is in one domain, and both clients are on another domain.

There is a trust relashinship between the AD, but I don´t know if that could cause the problem.

[Marianne]

Connection seems to be broken very soon after restore is started. Anything in the logs? On media server: bpbrm and bptm On destination client: bpcd and tar Please rename logs to reflect process names (e.g.bpbrm.txt) and post as attachments.

[Ramiro-Magan]

Here are the logs Marianne.

I noticed something strange on the Tar log, a line that goes:

09:24:36.395 a.m.: [3116.3552] <2> ov_log::V_GlobalLog: WRN - ubsDetermineExchangeVersion(): RegOpenKeyEx() failed for HKLM\SOFTWARE\MICROSOFT\ExchangeServer\v14\Setup - 0x2.

There is no exchange involved in this restore. Dunno, just caught my eye.

Thanks for your help.

[Mark_Solutions]

09:26:41.934 [7672.1548] <16> bpbrm readline: socket read failed, An existing connection was forcibly closed by the remote host.  (10054)
 

The exchange message is just as it does its usual checks before doing anything so dont worry about that one.

The above line is the issue - the network connection gets closed

Also the tar log just seems to stop as if it crashes or something blocks it

Any Anti-Virus or Access Protection or firewall that could slam the door shut when you try to write files across the link to this server?

Connection seems OK but then everything stops - check the Application event log on the destination client to see if there are any clues

[Marianne]

Seems "something" terminated the network connection before data could be transferred.

bpbrm started bptm (media manager) process at 09:24:29 and established connection to client at 09:24:31

Client (bpcd) confirmed connection at 09:24:35 and started tar process.

Last entry in tar log is at 09:24:36 while bptm was still loading tape and positioning it. (no errors in tar log - message that you saw is not an error: <2> is notice/debug info)

bptm finished positioning at 09:26:37 and started reading at 09:26:39

Here bptm reports the 'boken connection':

09:26:41.934 [7404.7552] <16> write_to_out: cannot write data to socket, 10054

We cannot say why connection was broken - just that it happened.
tar was ready to receive data, bptm tried to send data.

Any possibility of monitoring network connection between client and media server?
Any errors in Event Viewer on client?
Any errors/messages/logs on firewall between media server and client?
 

[Ramiro-Magan]

but that is the strange thing.

I CAN do a restore from DESTINATION to DESTINATION, and from SOURCE to SOURCE, what I can´t do is from SOURCE to DESTINATION, so I assume it must be something related with the cross restore.

some permits that the source has and the destination doesn´t, something like that.

There was a firewall but it was deactivated, and also the windows firewall though it is off, I set an inbound and outboud rule to permit any.

[Mark_Solutions]

Many Anti-Virus programs are set to trust local files by default - but may not trust cross domain files and may try to block them

If you have AV turned on the try turning it off (including all of it add-ons) whilst the restore runs

[Marianne]

Anything in Event Viewer logs? Check all of them - Application, Security...

All we know for sure is that "something" broke the connection.
Because NBU did not terminate the connection, all we can do is REPORT the broken connection.