Duplications issue with 191 error code

Question

Hello dear veritas community vox, long time we were to read your topics, greatfull thanks !We have some duplications failing with OST plugin error, break of the network exchange fail over a precise time (especialy recently discovered)Duplication is through IP (WAN). Netbackup side we have :&lt;16&gt;:bptm:ddp_filecopy_status() failed, start_offset[0], Err: 5009-filecopy operation failed (nfs: I/O error)Datadomain side :&nbsp;DDErrNo = 5009 (I/O error)
DDErrNo = 5057 (File handle is stale)All duplications are failing over :&nbsp;time for duplication network side break (131 minutes) (2h11 30 seconds)Vendor purpose to set up a time out value for NFS of 600000, plugin was updated on the media server which achieve the duplication through the WAN (which is one of 10 media servers for this master).For future purpose we had seen also some codes different of the master code 191 :&nbsp;84 : media write error190 / 191 : bellow the children code and plugin errorError&nbsp;bpduplicate&nbsp;backup&nbsp;id&nbsp;optimized&nbsp;duplication&nbsp;failed,&nbsp;client&nbsp;process&nbsp;aborted&nbsp;(50).
Error&nbsp;bpduplicate&nbsp;Duplicate&nbsp;of&nbsp;backupid&nbsp;failed,&nbsp;client&nbsp;process&nbsp;aborted&nbsp;(50).
Error&nbsp;bpduplicate&nbsp;Status&nbsp;=&nbsp;no&nbsp;images&nbsp;were&nbsp;successfully&nbsp;processed.
2060046:&nbsp;plugin&nbsp;errorVendor suggest it could be a network break caused by IPS or security enhanced feature.Best regards and thanks in advance for all your answers.

nicolai · Answer

the tcp_keepalive_time will only make a difference&nbsp;if there is a firewall between the two data domains.
tcp_keepalive ensure to send a idle frame to destination when the idle time has passed to prevent a fireall in closing idle connections.&nbsp;
You should ask the network admin what is configured in the firewall to find the right&nbsp;tcp_keepalive_time

davidmoline · Answer

Hey Benjamin
Altering the tcp parameters on the NetBackup side will have little effect of the data transfer between the Data Domain devices. You are trying to solve the problem in the wrong spot.
As&nbsp;RiaanBadenhorst&nbsp;has stated, this is really a problem for EMC to resolve. This may require altering tcp parameters on the DataDomain system along the lines of what&nbsp;Nicolai&nbsp;has suggested.
Given the traffic is traversing a WAN link, there will almost certainly be some form of gateway routers and/or firewalls involved. You will need to discuss with your network/security admins on the configuration of these to determine if one of these devices is interrupting the session.

genericus · Answer

I had this issue with my DD990 - but only VERY sporadically.&nbsp;It would happen only once every few months, and I could not cause it to happen. My duplications would run to the end, then fail. If that is what you are seeing, try this to verify:
Once the duplication fails with 191, cancel the SLP process after noting the image details.
Try manually duplicating, either from catalog or command line.
I found that even manual duplications would not work, somehow the DD as it deduped the image, or through normal cleaning/system maintenance made the image not stable.
I had to rerun the backup to get an image that I could duplicate.
This issue went away with my next upgrade/patch of my DD - you might try that if you are having this very sporadic issue.
&nbsp;
Since I could not replicate the process, I was not able to get any joy from DD support.

riaanbadenhorst · Answer

Those DD errors need to be resolved by EMC and your network team. NetBackup only tells the DD to replicate/duplicate and it waits for the DD to say when it's done. In your case its not getting done, its complaining about NFS I/O. Once that is resolved, you'll see no issue in NetBackup.

davidmoline · Answer

Thinking about this some more - the error you have shown as reported by the DD is around a stale file handle. If you look at this in a general sense stale NFS file handles have this cause: "A filehandle becomes stale whenever the file or directory referenced by the handle is removed by another host, while your client still holds an active reference to the object. A typical example occurs when the current directory of a process, running on your client, is removed on the server (either by a process running on the server or on another client)".
Has or is anything happening on either of the DD's involved to change the underlying storage/volume presented to the other side via NFS? Is there some kind of housekeeping task that runs at the same time as the backups that may change the file structure.&nbsp;
Something else to investigate anyway....

Forum Discussion

Duplications issue with 191 error code

Related Content

DLM issues - Duplicated backup sets

duplicate issue

Backup Exec 15 - GRT duplicate (Exchange issue)

Drive Error - Duplicate key or index

Duplication options