Media server drops out under load

[PostmanPat]

Hi - I'm hoping there may be someone who has come across this or a simliar problem and may be able to provide a fix.

We have a Windows 2003 media server using 10Gbit ethernet with jumbo frames enabled. Each time we initiate a backup that uses this media server the network connection to the server drops out.

The server itself does not report any errors - ther is nothing to indicate problems in the windows event log and no other network activity causes the same problem - only backup traffic.

Has anyone experienced this same issue?

[Will_Restore]

maybe something in the network switch logs?

[Mark_Solutions]

It depends how busy it gets when running backups.

It may run out of TCPIP ports and so sorts of just dissapear from the network for 4 minutes and then arrive back again!

Try the following (needs a reboot once you have done it):

Create two registry keys - both DWORDS, both decimal values given here:

HKLM\System\CurrentControlSet\Services\tcpip\parameters\

first one MaxUserPort with a value of 65534

second one TcpTimedWaitDelay with a value of 30

This increase the number of posts and also reduces the wait state of the used ports freeing things up quicker

Hope this helps

[Nicolai]

Windows 2003 has known problems with Reciver Side Scaling and TCP chimmeny. Make sure you have the most resent 10G NIC driver. 

TN: Network connectivity tuning to avoid network read/write failures and increase performance

http://www.symantec.com/docs/TECH60844

http://support.microsoft.com/kb/912222 - The Microsoft Windows Server 2003 Scalable Networking Pack release

 http://support.microsoft.com/kb/945977 - Some problems occur after installing Windows Server 2003 SP2

 http://support.microsoft.com/kb/948496/en-us - An update to turn off default SNP features is available for Windows Server 2003-based and Small Business Server 2003-based computers
 

 http://support.microsoft.com/kb/936594/ - You may experience network-related problems after you install Windows Server 2003 SP2 or the Scalable Networking Pack on a Windows Server 2003-based computer
 

 http://support.microsoft.com/kb/939421 - Best practices and known issues when you install Windows Server 2003 Service Pack 2 on a Windows Small Business Server 2003-based computer

[Omar_Villa]

what is the buffer size you are using maybe you set it up to big and is killing the NIC, windows is well know to be realy bad with large blocks.

[schmaustech]

I have seen where having a bad ethernet port and/or cable can also cause the problem described.  Network appears to function under normal loads, like RDP connection, however as soon as you pass backup traffic the ethernet appears to drop and you end up with network timeouts.

[Kevin_Good]

If you haven't enabled Jumbo Frames on both sides of the conenction, the network would drop the packets.

[PostmanPat]

Thanks for the suggestions - we have disabled the teaming of NIC's on the server as a workaround at present which has eliminated the dropouts. I will endeavour to try all the suggested fixes and post if we have any luck.

[Nicolai]

If a active teaming is used e.g. LACP the switch side need to be configured for port channel activity. If not you will loose 50% of all network trafic because the switch does not understand the load balance protocol.

The HP default teaming configuration is active/passive - this teaming does not need any active switch configuration.

[Michael_G_Ander]

Have had a similar experience with our Master/Media which at that time was a Windows 2003 x64 Cluster, we discovered that our problem was the desktop was to small, we had eventid 26 in the application event log. Used dheapmon8.1 to see the desktop heap sizes.

Got this from support: http://www.symantec.com/business/support/index?page=content&id=TECH48099