SLP Duplications Error with Status 84

[David_Bowlby]

Netbackup Version:  7.5.0.6

Storage Servers:  DD860 DataDomain

Datadomain OS:  5.4.0.8-404909

OST Plugin Version:  2.6.1.0

Issue:  During SLP Operations Receive a status 84.  Some of the duplications are successfull, most are not.  During these operations i recieve thousands of image clean-up jobs also.  We turned on replication of two cifs shares during the day before this started, i have since removed the replication pair.   Should i delete all the awaiting duplications and start from scratch?  Detailed status Below. 

4/21/2014 3:31:10 PM - Info bpdm(pid=19468) started            
4/21/2014 3:31:10 PM - Info Duplicate(pid=24400) Initiating optimized duplication from @aaaaD to @aaaaB      
4/21/2014 3:31:11 PM - started process bpdm (19468)
4/21/2014 3:31:11 PM - Info bpdm(pid=19468) requesting nbjm for media         
4/21/2014 3:31:17 PM - begin writing
4/21/2014 3:31:17 PM - Critical bpdm(pid=19468) sts_copy_extent failed: error 2060046 plugin error       
4/21/2014 3:31:17 PM - Critical bpdm(pid=19468) image copy failed: error 2060046: plugin error      
4/21/2014 3:31:17 PM - Error bpdm(pid=19468) cannot copy image from disk, bytesCopied = 18446744073709551615     
4/21/2014 3:31:23 PM - Error bpduplicate(pid=24400) host okcnbkp02.loves.com backup id vprosphrp03_1397883954 optimized duplication failed, media write error (84). 
4/21/2014 3:31:23 PM - Error bpduplicate(pid=24400) db_IMAGE() failed: images are in process (1519)      
4/21/2014 3:31:23 PM - Error bpduplicate(pid=24400) Duplicate of backupid vprosphrp03_1397883954 failed, media write error (84).    
4/21/2014 3:31:23 PM - Error bpduplicate(pid=24400) Status = no images were successfully processed.      
4/21/2014 3:31:23 PM - end Duplicate; elapsed time: 00:02:40
media write error(84)

Image Cleanup

4/21/2014 3:32:35 PM - begin 
4/21/2014 3:32:35 PM - Info nbdelete(pid=17020) Another nbdelete process is being run on media @aaaaB    
4/21/2014 3:32:35 PM - end ; elapsed time: 00:00:00
the requested operation was successfully completed(0)

Things i have attempted to correct this issue: 

1. Contacted EMC lowered the concurrent streams to 90 per their recommendation.  This is down from 115

2. Suspended all SLP's and activate 1 at a time.  I still get tons of 84 codes, but for the most part i can get the duplications completed.  

*attached bpdm, dptm, and more detailed status's to this post*

Any assistance will be appreciated. 

[David_Bowlby]

Found these in the logs.  thought it might be helpful. 

From the DDR – netbackup info log

“WARNING: ddboost-<okcnbkp02-43259>: ddboost_api ERROR: ddp_filecopy_start() failed, Err: 5005-nfs filecopy start failed (nfs: No space left on device)

04/21 18:53:28.805 (tid 0x2aaf69ffd400): ddboost-<okcnbkp02-43268>: JOB END OPTDUP_SOURCE ip=172.21.0.201 pid=12764 cd=0

04/21 18:53:28.805 (tid 0x2aaf69ffd400): DDBOOST 172.21.0.201 (eth1a) Jobs IMAGE_WRITE=1 IMAGE_READ=0 OPTDUP_SOURCE=59 OPTDUP_TARGET=31 IMAGE_INCLUDE=0

04/21 18:53:28.805 (tid 0x2aaaab30ea60): WARNING: ddboost-<okcnbkp02-43259>: plugin ERROR: F:\Program files\Veritas\NetBackup\bin\\ost-plugins\libstspiDataDomain.dll:stspi_copy_extent STS_EPLUGIN [DDErrNo = 5005 (no room left)]”

From netbackup  bpdm log

“15:30:20.222 [18312.9608] <4> copy_data: begin copying backup id psaphanacrp01.loves.com_1397937609 (duplicate-optimized), copy 1, fragment 1

15:30:20.222 [18312.9608] <16> 2093578:bptm:18312:okcnbkp02.loves.com: [4788:2588] ddp_filecopy_start() failed, Err: 5005-nfs filecopy start failed (nfs: No space left on device)

15:30:20.222 [18312.9608] <16> 2093578:bptm:18312:okcnbkp02.loves.com: F:\Program files\Veritas\NetBackup\bin\\ost-plugins\libstspiDataDomain.dll:stspi_copy_extent STS_EPLUGIN [DDErrNo = 5005 (no room left)]

15:30:20.222 [18312.9608] <32> do_copy_extent: sts_copy_extent failed: error 2060046

15:30:20.222 [18312.9608] <32> copy_data: image copy failed: error 2060046:

15:30:20.222 [18312.9608] <16> copy_data: cannot copy image from disk, bytesCopied = 18446744073709551615

15:30:20.222 [18312.9608] <2> validate_image_list: validating val_img_list

15:30:20.222 [18312.9608] <2> delete_validate_image_list: deleting val_img_list”

[SymTerry]

So your getting: DDErrNo = 5005 (no room left) and nfs: No space left on device

Is your DD full because the plugin is getting push back with space issues.

[David_Bowlby]

I have two datadomains both have 100 tb of space.  Neither is over 40% full. I run the cleaning every other day.  Backups are successful this only happens with slp duplications.    

I have been trying to figure out what that error means by no room left.  

[David_Bowlby · Accepted solution]

Setting the DiskPool Max I/O Streams to 90 and worked my way up to 120 fixed this issue.