New NDMP backup fails with setsockopt error 132, SIZE_DATA_BUFFERS_NDMP has no effect!

[DennisGauge]

I'm setting up a new NDMP backup and it fails with:

05/01/2017 12:51:04 - Error ndmpagent (pid=25513) setsockopt error 132 (No buffer space available)

According to this knowledgebase article:

https://www.veritas.com/support/en_US/article.000035688

I need to set up SIZE_DATA_BUFFERS_NDMP with a smaller size like 262144.

I did that, but the backups still use 524288 for the buffer size, and still fail!

[DennisGauge]

Here is some additional information on the situation:

I am backing up to DATA DOMAIN, so apparently SIZE_DATA_BUFFERS_NDMP does not apply.

SIZE_DATA_BUFFERS_DISK is the important attribute. On this environment it is set to 524288.

If I change SIZE_DATA_BUFFERS_DISK to 262144, the NDMP backup runs fine, but I am afraid this will adversely affect backup perforance elsewhere, and my ability to restore.

[Tousif]

Hello,

"If I change SIZE_DATA_BUFFERS_DISK to 262144, the NDMP backup runs fine, but I am afraid this will adversely affect backup performance elsewhere, and my ability to restore."

If you do the changes in SIZE_DATA_BUFFERS_DISK may effect other backup performance.

Article URL:http://www.veritas.com/docs/000028359

Suggestion:

Configured dedicated Media server for NDMP backup and restore, so changes will effective for that media server and NDMP backup/restore.

If you have tape library then please go with local NDMP for better performance.

Regards,

[DennisGauge]

Dedicated media server is not an option. We have to pull teeth to keep the ones we have somewhat up to date. They will not provide more media servers, crying about budget.

Tape is not an option. We've worked for three years to get off tape. We are not going back on tape.

This same data buffer size works with remote NDMP backups in two other backup environments within the company. All are the same exact version of Netbackup 7.7.3.

[Nicolai]

I wouldn't worry to much using a 256K disk buffer size. The performance drop, if any at all, can likely be disregarded.

Disk does not take a performances hit on SIZE_DATA_BUFFERS_DISK as tape devices do. Disk with their random access pattern can resume writes much faster then tape that has to do "write-stop-reposition-write" for chunk of missing data.

Please check SIZE_DATA_BUFFERS_NDMP file is WITHOUT any file extensions.

[DennisGauge]

This is getting frustrating. I finally found a media server that will run the backup without changing the parameters, but when it gets done, now I get this:

05/02/2017 08:47:19 - Info ndmpagent (pid=12840) vnx0544nfs: server_archive: emctar vol 1, 71085 files, 0 bytes read, 340913637306 bytes written

05/02/2017 08:47:20 - Info ndmpagent (pid=12840) NDMP backup successfully completed, path = /lavastorm

05/02/2017 08:47:21 - Error bpbrm (pid=10968) db_flist_complete failed: events out of sequence - image inconsistency (229) vnx0544nfs LAVD0P30.lav00p30_ndmp 0 1493727784

05/02/2017 08:47:31 - Error bptm (pid=8116) get_string() failed, Broken pipe (32), premature end of file encountered

05/02/2017 08:47:31 - Info bptm (pid=8116) EXITING with status 42 <----------

05/02/2017 08:47:32 - Critical bpbrm (pid=10968) unexpected termination of client vnx0544nfs

05/02/2017 08:47:32 - Info ndmpagent (pid=0) done. status: 42: network read failed

05/02/2017 08:52:25 - end writing; write time: 0:29:16

network read failed  (42)  

 Searching on the errors, all I can find is info on tape backups...

[Nicolai]

do you have accelerator enabled in the policy ?

[DennisGauge]

Accelerator is NOT enabled. Should it be?

[Marianne]

No, accellerator does not need to be enabled.

You now have a different issue and we know nothing about the media server or the policy config. 

The error seems to indicate a comms/timeout problem between master and (new) media server.

So, troubleshooting needs to start on the media server.
Which OS? 
Which NBU version and patch level? 
How long does the backup run before the backup fails? 
We can see from the Activity Monitor snippet that the backup has actually completed but that the media server fails to send catalog info to the master.

Do you have bpbrm log on the media server (level 3 at least)?
And bpdbm on master server? 

[Tousif]

Hello,

Sharing my finding:

Error:
Error bpbrm (pid=10968) db_flist_complete failed: events out of sequence - image inconsistency (229) vnx0544nfs LAVD0P30.lav00p30_ndmp 0 1493727784

Cause:
1. Network
2. Below mentioned added in backup selection
Set "TYPE=VBB" in NetBackup policy directives

If above mentioned not added in backup selection then share current backup selection.


1) replace with SET TYPE=TAR

2) Increase the Network timeout
Reference Article: https://www.veritas.com/support/en_US/article.TECH21246

Is this EMC NAS box?

If you do not find my suggestion applicable to your issue then please provide NBU environment detail:

Master/Media Operating System:

Master/Media NBU version:

NAS Filer Model:

Logs with debug logging:

ndmpagent

bpbrm

ndmp

complete failed job detail status.

Regards,

[Marianne]

Adding NDMP_PROGRESS_TIMEOUT file on the new media server seems like a good idea.