Info bpbkar32(pid=3124) done. status: 44: network write failed network write failed(44)

[Zahid_Haseeb]

Environment

Veritas Netbackup = 7.1

OS of Netbackup = win2008

Tape Library attached with six drives

Problem

I am doing Catalog backup. While doing backup on Tape Cartridge at around 80% completion the backup got failed.

1/26/2012 10:36:15 AM - Info nbjm(pid=4108) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=143574, request id:{12B409E1-989A-442B-A1EA-8F10361D8B3D})  
1/26/2012 10:36:15 AM - requesting resource NBU-Server-hcart-robot-tld-0
1/26/2012 10:36:15 AM - requesting resource NBU-Server.NBU_CLIENT.MAXJOBS.NBU
1/26/2012 10:36:15 AM - requesting resource NBU-Server.NBU_POLICY.MAXJOBS.Catalog_Backup
1/26/2012 10:36:15 AM - awaiting resource NBU-Server-hcart-robot-tld-0 - No drives are available
1/26/2012 10:39:53 AM - Info bpbrm(pid=5288) NBU-Server is the host to backup data from     
1/26/2012 10:39:53 AM - Info bpbrm(pid=5288) reading file list from client        
1/26/2012 10:39:53 AM - granted resource NBU-Server.NBU_CLIENT.MAXJOBS.NBU
1/26/2012 10:39:53 AM - granted resource NBU-Server.NBU_POLICY.MAXJOBS.Catalog_Backup
1/26/2012 10:39:53 AM - granted resource 0014L5
1/26/2012 10:39:53 AM - granted resource IBM.ULT3580-TD5.002
1/26/2012 10:39:53 AM - granted resource NBU-Server-hcart-robot-tld-0
1/26/2012 10:39:53 AM - estimated 48899691 Kbytes needed
1/26/2012 10:39:53 AM - Info nbjm(pid=4108) started backup job for client NBU-Server, policy Catalog_Backup, schedule Full on storage unit NBU-hcart-robot-tld-0
1/26/2012 10:39:53 AM - started process bpbrm (5288)
1/26/2012 10:39:53 AM - connecting
1/26/2012 10:39:54 AM - Info bpbrm(pid=5288) starting bpbkar32 on client         
1/26/2012 10:39:54 AM - connected; connect time: 00:00:01
1/26/2012 10:39:55 AM - Info bpbkar32(pid=3124) Backup started           
1/26/2012 10:39:55 AM - Info bptm(pid=960) start            
1/26/2012 10:39:55 AM - Info bptm(pid=960) using 65536 data buffer size        
1/26/2012 10:39:55 AM - Info bptm(pid=960) setting receive network buffer to 263168 bytes      
1/26/2012 10:39:55 AM - Info bptm(pid=960) using 30 data buffers         
1/26/2012 10:39:55 AM - Info bptm(pid=960) start backup           
1/26/2012 10:39:55 AM - Info bptm(pid=960) Waiting for mount of media id 0014L5 (copy 1) on server NBU-Server.
1/26/2012 10:39:55 AM - mounting 0014L5
1/26/2012 10:40:32 AM - Info bptm(pid=960) media id 0014L5 mounted on drive index 2, drivepath {3,0,4,0}, drivename IBM.ULT3580-TD5.002, copy 1
1/26/2012 10:40:32 AM - mounted; mount time: 00:00:37
1/26/2012 10:40:32 AM - positioning 0014L5 to file 281
1/26/2012 10:41:30 AM - positioned 0014L5; position time: 00:00:58
1/26/2012 10:41:30 AM - begin writing
1/26/2012 10:55:28 AM - Info bptm(pid=960) waited for full buffer 31786 times, delayed 40142 times    
1/26/2012 10:55:28 AM - Info bpbkar32(pid=3124) bpbkar waited 10922 times for empty buffer, delayed 11000 times.   
1/26/2012 10:55:28 AM - Error bpbrm(pid=5288) db_FLISTsend failed: network write failed (44)       
1/26/2012 11:00:28 AM - Error bpbrm(pid=5288) could not send server status message       
1/26/2012 11:00:28 AM - end writing; write time: 00:18:58
1/26/2012 11:00:33 AM - Info bpbkar32(pid=3124) done. status: 44: network write failed       
network write failed(44)

[Zahid_Haseeb]

Log folder size is around 10GB.

( I feel that in future I have to move the Catalog to another partition. Any way to shrink/compact/defragment the Catalog )

[Mark_Solutions]

I would clear down your logs to free up the disk space and try the catalog backup again.

If that works then I would look at re-locating your catalog files (the db folder can be relocated relatively easy) or as an alternative you can relocate your logs elsewhere to save them using your space up

Give it a try and I will get you the links for moving your db folder and the logs - but lets see if it works first

[Zahid_Haseeb]

See the below logs. It seems that the space was out of space. But now my backup is running successful with less space.

For example

When my backup was failing that time the space was around 12GB and now the space is around 9GB and backup is running fine.

See the below logs for reference:

log entry in bpdbm

08:55:27.560 [6420.1388] <2> get_adaptable_string: (4) network read() error:  No buffer space available.

[Marianne]

Network buffer space and disk space is not the same...

Please do not leave logging level at 5 if you are not troubleshooting a specific problem.

Level 5 logs grow large, make your system slow, consume system resources, etc... ONLY increase logging level while troubleshooting specific problems when you have an open case with Symantec. Drop down logging levels to 0 as soon as all necessary logs have been collected.
Level 0 logs are sufficient to troubleshoot day-to-day issues.

Please also check at Windows level for disk fragmentation.

[Zahid_Haseeb]

This couldnt be the problem ?

08:55:27.560 [6420.1388] <2> get_adaptable_string: (4) network read() error:  No buffer space available.

[Marianne]

Yes, that could be causing the problem.

I was trying to say that it points to a network problem, not disk space.

[Zahid_Haseeb]

What sort of that buffer could be ?

(4) network read() error:  No buffer space available

[mph999]

I also pointed out it could be a network problem, but my advice was discounted ...

To answer the latest questions ...

The send /receive buffers on the NIC ... 

OK, lets straighten something out ...

If we suggest something, or ask for detail that seems not relevent, please provide it anyway - we are not asking for the fun of of it ...

Just becuase you can see no reason for the network being the issue, does not mean it is not the issue, until it has been PROVED 100%.

A little story to demonstrate ...

 I had a case recently, NBU would not mount a tape in the drive, hung when mounting.

Now, I am the first to point out that tape issues are usually not NBU ... but ...

Robtest could load a tape

The operating system could access that tape, and, write to it using tar ... etc ...

Only fault, was NBU not mounting the tape ...

So, even I had to admit it was looking like NBU was the fault ....

Nope, turns out, it was the firmware on the drive(s).

So, you see, no matter what the issue does, or does not look like, it can be the most unexpected thing that casues the problem.

Martin

[Marianne]

GOOGLE found some info for us.

Firstly a Symantec TechNote with a different status code but the same network buffer error.
The TN is very old, so I cannot tell if it will be applicable to W2008.

http://www.symantec.com/docs/TECH55906

The error is caused when the master server ran out of network buffer space.

Resolution:
Check the boot.ini on the master and verify that the /3GB switch is not used.  What the /3GB switch does is allocate 1GB of address space to the Windows operating system and 3GB of address space to user mode processes.  This allows Windows to better accommodate demanding applications such as Exchange and SQL Servers.

By only allowing the operating system to allocate 1GB of address space, a limit is also placed on the amount of network buffers your operating system can use.  Remove the /3GB entry from your boot.ini and the "No buffer space available" error from the operating system should not reappear.

Second Google find:

http://docs.dal.net/docs/connection.html#6

6 · [10055] No buffer space available

Scenario: Joe wanted to call Mary, but his hands were already full.

This means mIRC is having a problem creating a new a network socket; it cannot use your Internet connection to connect to an IRC server. If you are using a lot of other network applications at the same time, you might get this error. Close some other applications and/or reset your Internet connection to fix this problem. This error also indicates a shortage of resources on your system. It can occur if you're trying to run too many applications (of any kind) simultaneously on your machine. If this tends to occur after running certain applications for a while, it might be a symptom of an application that doesn't return system resources (like memory) properly. It may also indicate you are not closing the applications properly. If it persists, exit Windows or reboot your machine to remedy the problem. You can monitor available memory with Windows Explorer's "Help/About..." command.

Third Google find:

http://msdn.microsoft.com/en-us/library/windows/desktop/ms740668%28v=vs.85%29.aspx

Windows Sockets Error Codes

 

WSAENOBUFS

10055

No buffer space available.

An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.

*******************************************************************

So, it seems that you have a resource problem at OS level?

Fellow Connect expert AAlmroth has written an excellent article regarding performance tuning on Windows servers: https://www-secure.symantec.com/connect/articles/tuning-windows-2003-and-2008-symantec-netbackup

Maybe this article will provide some useful info?

[Mark_Solutions · Accepted solution]

Also check out the post I did earlier in the thread to tune your network  - did you do these and reboot the server?:

1. Add the following registry key to the Master Server:

HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\

DWORD – TcpTimedWaitDelay  - Decimal Value of 30

2. From a "run as administrator" from command line

 Netsh int ipv4 set dynamicport tcp start=10000 num=50000

 This gives it 60000 connections, the default is 16383

Also, what NET_BUFFER_SZ values are you using (if any - \ntebackup\bin\)?

[Zahid_Haseeb]

mph999 first of all I would like to pay bundle of thanks in the regard of your kins help and I do value it. If at some place if you feel that your suggesstions are being ignored so sorry for that.

Mark and Marianne let me do this and share the result