HP teaming with backup exec issue?

[Johnny_mnemonic]

Hi,

I am wondering whether anyone encounter any problem with hp teaming and backup exec?

I had been having the connection to the target systems has been lost.. error  and was troubleshooting with Symantec for near to 2 weeks. Till to date, it had not been solve and i am thinking whether could it be an issue when my server is using the hp nic teaming.

Hope that some can advise me on that. Thanks

[MrBuskin]

We run all of our client backups off a DL380 with pairing setup as follows

Nic1 -
        |--- Intranet - 10.0.103.243
Nic2 -

We've not had any problems

[Dave_Woo]

I've not heard of any issues with HP teaming, the most common occurance of that error, is where the backup exec remote agent drops the connection this can happen for several reasons one of the more frequent, is tamper protection setting on anti virus products, firewall blocking, or somthing as simple as the network dropping out.

We would have had you run a debug on the backup from one of the target servers. can you past the contents into here?

[Johnny_mnemonic]

Hi guys,

Below are part of the debug log from a remote agent. There is alwys this error mesg: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.

Hope that some experts can give me some valuable advice to solve my issue.

In fact today i had just spoken to one the symantec technical support and he is asking me to spilt the backup job into 2 portion. 1 for the system state/Volume shadow and the other for C: and I:. His reason being i do not have AOFO enable and the backup of System State /Shadow Copy component was failing with "lost connection" error.

I am very in doubt that this will help..but will try it out tonight.

0xfa0
[5020] 10/15/07 03:11:08 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:11:08 ERROR: sending message 0xf204: encoding request header.
[5020] 10/15/07 03:11:08 ndmp_writeit: len:4000
[5020] 10/15/07 03:11:08 ndmp_writeit: NDMP Socket Error when trying to write message. Socket 0x4dc len 0xfa0
[5020] 10/15/07 03:11:08 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:11:08 ERROR: ndmpdApiLog: error sending log message request.
[5020] 10/15/07 03:11:08 ndmpdGenerateFileHistoryInfo: name:Apps\Housekeeping\Backup\Sun\SHIPOUT lba:13994654
[5020] 10/15/07 03:11:08 ndmpSendRequest: message:0xf337
[5020] 10/15/07 03:11:08 ndmp_writeit: len:4000
[5020] 10/15/07 03:11:08 ndmp_writeit: NDMP Socket Error when trying to write message. Socket 0x4dc len 0xfa0
[5020] 10/15/07 03:11:08 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:11:08 ERROR: sending message 0xf337: encoding request header.
[5020] 10/15/07 03:11:08 ndmp_writeit: len:4000
[5020] 10/15/07 03:11:08 ndmp_writeit: NDMP Socket Error when trying to write message. Socket 0x4dc len 0xfa0
[5020] 10/15/07 03:11:08 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:11:08 ERROR: ndmpdApiExtFileHistoryFileExt: error sending ndmp_vrts_fh_add_file_request.
[5020] 10/15/07 03:11:08 ndmpFreeMessage: message:0x0
[5020] 10/15/07 03:11:08 ndmpdApiLog: version:3
[5020] 10/15/07 03:11:08 ndmpdApiLog: message_id = 10, log_type = 0, entry_len=26
[5020] 10/15/07 03:11:08 ndmpSendRequest: message:0xf204
[5020] 10/15/07 03:11:08 ndmp_writeit: len:4000
[5020] 10/15/07 03:11:08 ndmp_writeit: NDMP Socket Error when trying to write message. Socket 0x4dc len 0xfa0
[5020] 10/15/07 03:11:08 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:11:08 ERROR: sending message 0xf204: encoding request header.
[5020] 10/15/07 03:11:08 ndmp_writeit: len:4000
[5020] 10/15/07 03:11:08 ndmp_writeit: NDMP Socket Error when trying to write message. Socket 0x4dc len 0xfa0
[5020] 10/15/07 03:11:08 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:11:08 ERROR: ndmpdApiLog: error sending log message request.
[5020] 10/15/07 03:11:08 ndmpdGenerateFileHistoryInfo: name:SHIPOUT.DBS lba:13994655
[5020] 10/15/07 03:11:08 ndmpSendRequest: message:0xf337
[5020] 10/15/07 03:11:08 ndmp_writeit: len:4000
[5020] 10/15/07 03:11:08 ndmp_writeit: NDMP Socket Error when trying to write message. Socket 0x4dc len 0xfa0
[5020] 10/15/07 03:11:08 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:11:08 ERROR: sending message 0xf337: encoding request header.
[5020] 10/15/07 03:11:08 ndmp_writeit: len:4000
[5020] 10/15/07 03:11:08 ndmp_writeit: NDMP Socket Error when trying to write message. Socket 0x4dc len 0xfa0
[5020] 10/15/07 03:11:08 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:11:08 ERROR: ndmpdApiExtFileHistoryFileExt: error sending ndmp_vrts_fh_add_file_request.
[5020] 10/15/07 03:11:08 ndmpFreeMessage: message:0x0
[5020] 10/15/07 03:11:11 ndmpdApiLog: version:3
[5020] 10/15/07 03:11:11 ndmpdApiLog: message_id = 10, log_type = 0, entry_len=26
[5020] 10/15/07 03:11:11 ndmpSendRequest: message:0xf204
[5020] 10/15/07 03:11:11 ndmp_writeit: len:4000
[5020] 10/15/07 03:11:11 ndmp_writeit: NDMP Socket Error when trying to write message. Socket 0x4dc len 0xfa0
[5020] 10/15/07 03:11:11 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:11:11 ERROR: sending message 0xf204: encoding request header.
[5020] 10/15/07 03:11:11 ndmp_writeit: len:4000
[5020] 10/15/07 03:11:11 ndmp_writeit: NDMP Socket Error when trying to write message. Socket 0x4dc len 0xfa0
[5020] 10/15/07 03:11:11 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:11:11 ERROR: ndmpdApiLog: error sending log message request.
[5020] 10/15/07 03:11:11 ndmpdGenerateFileHistoryInfo: name:Apps\Housekeeping\Backup\Sun\SHIPPIC lba:14023388
[5020] 10/15/07 03:11:11 ndmpSendRequest: message:0xf337
[5020] 10/15/07 03:11:11 ndmp_writeit: len:4000
[5020] 10/15/07 03:11:11 ndmp_writeit: NDMP Socket Error when trying to write message. Socket 0x4dc len 0xfa0
[5020] 10/15/07 03:11:11 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:11:11 ERROR: sending message 0xf337: encoding request header.
[5020] 10/15/07 03:11:11 ndmp_writeit: len:4000
[5020] 10/15/07 03:11:11 ndmp_writeit: NDMP Socket Error when trying to write message. Socket 0x4dc len 0xfa0
[5020] 10/15/07 03:11:11 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:11:11 ERROR: ndmpdApiExtFileHistoryFileExt: error sending ndmp_vrts_fh_add_file_request.
[5020] 10/15/07 03:11:11 ndmpFreeMessage: message:0x0
[5020] 10/15/07 03:11:11 ndmpdApiLog: version:3
[5020] 10/15/07 03:11:11 ndmpdApiLog: message_id = 10, log_type = 0, entry_len=26
[5020] 10/15/07 03:11:11 ndmpSendRequest: message:0xf204
[5020] 10/15/07 03:11:11 ndmp_writeit: len:4000
[5020] 10/15/07 03:11:11 ndmp_writeit: NDMP Socket Error when trying to write message. Socket 0x4dc len 0xfa0
[5020] 10/15/07 03:11:11 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:11:11 ERROR: sending message 0xf204: encoding request header.
[5020] 10/15/07 03:11:11 ndmp_writeit: len:4000
[5020] 10/15/07 03:11:11 ndmp_writeit: NDMP Socket Error when trying to write message. Socket 0x4dc len 0xfa0
[5020] 10/15/07 03:11:11 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:11:11 ERROR: ndmpdApiLog: error sending log message request.
[5020] 10/15/07 03:11:11 ndmpdGenerateFileHistoryInfo: name:SHIPPIC.DBS lba:14023389
[5020] 10/15/07 03:11:11 ndmpSendRequest: message:0xf337
[5020] 10/15/07 03:11:11 ndmp_writeit: len:4000
[5020] 10/15/07 03:11:11 ndmp_writeit: NDMP Socket Error when trying to write message. Socket 0x4dc len 0xfa0
[5020] 10/15/07 03:11:11 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:11:11 ERROR: sending message 0xf337: encoding request header.
[5020] 10/15/07 03:11:11 ndmp_writeit: len:4000
[5020] 10/15/07 03:11:11 ndmp_writeit: NDMP Socket Error when trying to write message. Socket 0x4dc len 0xfa0
[5020] 10/15/07 03:11:11 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:11:11 ERROR: ndmpdApiExtFileHistoryFileExt: error sending ndmp_vrts_fh_add_file_request.
[5020] 10/15/07 03:11:11 ndmpFreeMessage: message:0x0
[5020] 10/15/07 03:17:56 TF_GetNextTapeRequest( ) Error = E00084F9
[5020] 10/15/07 03:17:56 TF_GetNextTapeRequest( :( Recalled in Error State
[5020] 10/15/07 03:17:56 TF_GetNextTapeRequest( :( Recalled in Error State
[5020] 10/15/07 03:17:56 ndmpdApiLog: version:3
[5020] 10/15/07 03:17:56 ndmpdApiLog: message_id = 12, log_type = 0, entry_len=17
[5020] 10/15/07 03:17:56 ndmpSendRequest: message:0xf204
[5020] 10/15/07 03:17:56 ndmp_writeit: len:4000
[5020] 10/15/07 03:17:56 ndmp_writeit: NDMP Socket Error when trying to write message. Socket 0x4dc len 0xfa0
[5020] 10/15/07 03:17:56 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:17:56 ERROR: sending message 0xf204: encoding request header.
[5020] 10/15/07 03:17:56 ndmp_writeit: len:4000
[5020] 10/15/07 03:17:56 ndmp_writeit: NDMP Socket Error when trying to write message. Socket 0x4dc len 0xfa0
[5020] 10/15/07 03:17:56 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:17:56 ERROR: ndmpdApiLog: error sending log message request.
[5020] 10/15/07 03:17:56 ndmpdApiLog: version:3
[5020] 10/15/07 03:17:56 ndmpdApiLog: message_id = 7, log_type = 0, entry_len=4
[5020] 10/15/07 03:17:56 ndmpSendRequest: message:0xf204
[5020] 10/15/07 03:17:56 ndmp_writeit: len:4000
[5020] 10/15/07 03:17:56 ndmp_writeit: NDMP Socket Error when trying to write message. Socket 0x4dc len 0xfa0
[5020] 10/15/07 03:17:56 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:17:56 ERROR: sending message 0xf204: encoding request header.
[5020] 10/15/07 03:17:56 ndmp_writeit: len:4000
[5020] 10/15/07 03:17:56 ndmp_writeit: NDMP Socket Error when trying to write message. Socket 0x4dc len 0xfa0
[5020] 10/15/07 03:17:56 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:17:56 ERROR: ndmpdApiLog: error sending log message request.
[5020] 10/15/07 03:17:56 TF_CloseSet()
[5020] 10/15/07 03:17:56 Fatal error in channel.
[5020] 10/15/07 03:17:56 FreeFormatEnv( cur_fmt=0 )
[5020] 10/15/07 03:17:56 Detach from \\bp1iomap001\I:
[5020] 10/15/07 03:17:56 TF_FreeDriveContext( 4C8A558 )
[5020] 10/15/07 03:17:56 TF_FreeTapeBuffers: from 10 to 0 buffers
[5020] 10/15/07 03:17:56 ndmpdDataError: 0x1
[5020] 10/15/07 03:17:56 ndmpSendRequest: message:0x501
[5020] 10/15/07 03:17:56 ndmp_writeit: len:4000
[5020] 10/15/07 03:17:56 ndmp_writeit: NDMP Socket Error when trying to write message. Socket 0x4dc len 0xfa0
[5020] 10/15/07 03:17:56 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:17:56 ERROR: sending message 0x501: encoding request header.
[5020] 10/15/07 03:17:56 ndmp_writeit: len:4000
[5020] 10/15/07 03:17:56 ndmp_writeit: NDMP Socket Error when trying to write message. Socket 0x4dc len 0xfa0
[5020] 10/15/07 03:17:56 ndmp_writeit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:17:56 ERROR: ndmpdDataError: error sending notify_data_halted request.
[5020] 10/15/07 03:17:56 @@@@@@@MyCloseSocket called with sockfd = 1860(0x744) retval = -1
[5020] 10/15/07 03:17:56 Backup Job Stop(0)  - Mon Oct 15 03:17:56 2007
[5020] 10/15/07 03:17:56 ndmpFreeMessage: message:0x0
[5020] 10/15/07 03:17:56 ndmp_readit: len:4000
[5020] 10/15/07 03:17:56 ndmp_readit: Caught message on closed connection. Socket 0x4dc len 0xffffffff
[5020] 10/15/07 03:17:56 ndmp_readit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[5020] 10/15/07 03:17:56 ndmp_process_messages: no more messages in stream buffer
[5020] 10/15/07 03:17:56 ndmpFreeMessage: message:0x0
[5020] 10/15/07 03:17:56 ndmpDestroyConnection
[5020] 10/15/07 03:17:56 @@@@@@@MyCloseSocket called with sockfd = 1244(0x4dc) retval = 0

Message Edited by Johnny mnemonic on 10-15-2007 02:13 AM

[Johnny_mnemonic]

Hi guys,

below is the reply from the symantec technical support. What do u all think??

As I said I have seen the issue few times when customer had no AOFO installed. The backup of System State /Shadow Copy component was failing with "lost connection" error. The solution was to split the jobs as I suggested to you. It may have something to do with the VSS writer status. When you back up System State Windows by default uses VSS to create a snapshot and if you create separate job for that backup then VSS writer has enough time to finish and return to stable state.

Please run few test jobs to see if that helps.

[ESabo]

I am interested in this thread as well, I switched today and teamed two adapters in my server, which I never got any errors until the two adapters were teamed, tonight 2 out of 5 backup jobs got errors.

Are there any fixes for this issue?   I checked with HP and I am running the latest drivers and software.

[Johnny_mnemonic]

I am still very unwilling to remove the team to test out as i found out alot of people are not having any backup problem with hp teaming.

Since you are the first one to highlight the problem, we shall see whether is there anyone else facing this problem and hope that some one can give us some good advice or solution.

[ESabo]

Well my backups were just fine until I introduced the HP Teaming, I am probably going to revert back to a single network card because it did not make any difference in the speed.    Maybe I got 25MB/min or 50 MB/min or more throughput.

[ESabo]

I would say the failure rate on backup jobs using teaming is about 20% so far.   I am going to give it one more night before I stop the HP teaming.

I wonder why this is happening.

[Johnny_mnemonic]

Hi,

My backup ran successfully after i had split the job into 2 as advised by Symantec. I will monitor it closely for a few days and will update all here again. Thanks

Message Edited by Johnny mnemonic on 10-16-2007 05:58 PM

[CaptainCanoe]

We just added two HP DL580 servers with HP NC371i Gig NICs in them. There is no teaming set up on these units but there is on our Backup Exec server. The backup job rate of these two servers was terible. Anywhere from 50MB/Minute to 300MB/Minute. After a week of trying everything I could think of.....I disabled teaming on the Backup Exec Server and the rate jumped to 3.5GB/Minute. Before I disabled teaming the NIC Utilization (in task manager) seldom got over 2% during the backup. Afterwards it was a steady 50%-60%.

These new servers are replaceing two HP DL560 servers with HP NC7781 Gig NICs. The configuration of the old servers is almost identical to the new servers (no teaming) and the job rate has always been excellent. So I'm guessing it's something with the NIC or the configuration of it on the new servers. In any case I'm done babysitting this backup job. Fingers Crossed!

[ESabo]

It is just strange some servers are backing up with very good speeds which would be 4 servers in total.   But 2 servers are failing almost every night.    What can one look for to try and figure out what is happening?    Teaming shouldn't be an issue here.    I have the latest drivers and correct settings on both servers.

[Johnny_mnemonic]

However, it is strange enough that i am not able to search any documents or kb in symantec with regards to issue with HP teaming. Even though when i log a call to symantec, they didnt even mention to tear down the teaming and test on the backup.

Btw my last night back ran successfully again. But i am thinking if my whole server crash, i will need to restore the system state/volume shadowcopy first and then C: drive with other Data drive. Is that more troublesome when i just need to restore one copy of the backup which consist of everything. Anyone can advise?

[Bald_From_Netwo]

I have spent approx 4-5 weeks working with Hp Advanced Multi-Vendor Support Engineers

(their words not mine) and Cisco support on teaming with DL & ML server connecting to

various Cisco switches and I have reams of Beta White Papers

(OOOOhh I'm special (not at all) ) and I'd be glad to share this life experience with you, but

the results will depend on a lot of little config variations. I would be glad to speak to you if you

would like. Email I just don't think will cut it. Plus I'm a very slow typist and bad speller .

Contact me if you want.

P.S. Symantec was also very involved. For a couple of days I had one of their field engineers onsite.

[Johnny_mnemonic]

Hi,

Glad to hear that.

Would you  mind to share some of your beta white paper to me?

Can send to johnnytanki@hotmail.com

Thanks a lot.

[Bald_From_Netwo]

That's the problem I had to sign a will not redistribute agreement, but the engineer said that does not cover just "Talking" about the information in a non-public manner. I sent you my land line. Call me and we can have a dialog :) Non-public of course.

[Johnny_mnemonic]

Sigh !!!

After running fine for a week, another problem intermittently appear.

Now the error message:

Error with network connection to the Backup exec remote agent has been lost.  

An inconsistency was encountered on the storage media ....

The data being read from media is inconsistent...

Anyone can help???

[Johnny_mnemonic]

Hi,

Need one more advices from all.

Is it a requirement to install the veritas/symantec tape driver rather than using the OEM manufacturer driver in BEXEC11d?  I noticed the driver that i am currently using belong to HP. Will that cause any problem? should i change it to symantec tape driver? Thanks

[Bald_From_Netwo]

I think the answer is yes support wants you to use the drivers they supply. One it gives then a base point of having tested the exact version of the driver in their labs.

[Bald_From_Netwo]

Was this error during a verify?