Forum Discussion

David_Bowlby's avatar
David_Bowlby
Level 4
11 years ago

SLP Duplications Error with Status 84

Netbackup Version:  7.5.0.6

Storage Servers:  DD860 DataDomain

Datadomain OS:  5.4.0.8-404909

OST Plugin Version:  2.6.1.0

Issue:  During SLP Operations Receive a status 84.  Some of the duplications are successfull, most are not.  During these operations i recieve thousands of image clean-up jobs also.  We turned on replication of two cifs shares during the day before this started, i have since removed the replication pair.   Should i delete all the awaiting duplications and start from scratch?  Detailed status Below. 

4/21/2014 3:31:10 PM - Info bpdm(pid=19468) started            
4/21/2014 3:31:10 PM - Info Duplicate(pid=24400) Initiating optimized duplication from @aaaaD to @aaaaB      
4/21/2014 3:31:11 PM - started process bpdm (19468)
4/21/2014 3:31:11 PM - Info bpdm(pid=19468) requesting nbjm for media         
4/21/2014 3:31:17 PM - begin writing
4/21/2014 3:31:17 PM - Critical bpdm(pid=19468) sts_copy_extent failed: error 2060046 plugin error       
4/21/2014 3:31:17 PM - Critical bpdm(pid=19468) image copy failed: error 2060046: plugin error      
4/21/2014 3:31:17 PM - Error bpdm(pid=19468) cannot copy image from disk, bytesCopied = 18446744073709551615     
4/21/2014 3:31:23 PM - Error bpduplicate(pid=24400) host okcnbkp02.loves.com backup id vprosphrp03_1397883954 optimized duplication failed, media write error (84). 
4/21/2014 3:31:23 PM - Error bpduplicate(pid=24400) db_IMAGE() failed: images are in process (1519)      
4/21/2014 3:31:23 PM - Error bpduplicate(pid=24400) Duplicate of backupid vprosphrp03_1397883954 failed, media write error (84).    
4/21/2014 3:31:23 PM - Error bpduplicate(pid=24400) Status = no images were successfully processed.      
4/21/2014 3:31:23 PM - end Duplicate; elapsed time: 00:02:40
media write error(84)

 

Image Cleanup

4/21/2014 3:32:35 PM - begin 
4/21/2014 3:32:35 PM - Info nbdelete(pid=17020) Another nbdelete process is being run on media @aaaaB    
4/21/2014 3:32:35 PM - end ; elapsed time: 00:00:00
the requested operation was successfully completed(0)

 

Things i have attempted to correct this issue: 

1. Contacted EMC lowered the concurrent streams to 90 per their recommendation.  This is down from 115

2. Suspended all SLP's and activate 1 at a time.  I still get tons of 84 codes, but for the most part i can get the duplications completed.  

 

*attached bpdm, dptm, and more detailed status's to this post*

 

Any assistance will be appreciated. 

7.5
Backing Up
Backup and Recovery
Best Practice
Error messages
Managing Backup Devices
Monitoring
NetBackup
Performance
Troubleshooting
  • David_Bowlby's avatar
    David_Bowlby
    11 years ago

    Setting the DiskPool Max I/O Streams to 90 and worked my way up to 120 fixed this issue.   

  • David_Bowlby's avatar
    David_Bowlby
    Level 4
    11 years ago

    Found these in the logs.  thought it might be helpful. 

     

    From the DDR – netbackup info log

    “WARNING: ddboost-<okcnbkp02-43259>: ddboost_api ERROR: ddp_filecopy_start() failed, Err: 5005-nfs filecopy start failed (nfs: No space left on device)

    04/21 18:53:28.805 (tid 0x2aaf69ffd400): ddboost-<okcnbkp02-43268>: JOB END OPTDUP_SOURCE ip=172.21.0.201 pid=12764 cd=0

    04/21 18:53:28.805 (tid 0x2aaf69ffd400): DDBOOST 172.21.0.201 (eth1a) Jobs IMAGE_WRITE=1 IMAGE_READ=0 OPTDUP_SOURCE=59 OPTDUP_TARGET=31 IMAGE_INCLUDE=0

    04/21 18:53:28.805 (tid 0x2aaaab30ea60): WARNING: ddboost-<okcnbkp02-43259>: plugin ERROR: F:\Program files\Veritas\NetBackup\bin\\ost-plugins\libstspiDataDomain.dll:stspi_copy_extent STS_EPLUGIN [DDErrNo = 5005 (no room left)]”

     

    From netbackup  bpdm log

    “15:30:20.222 [18312.9608] <4> copy_data: begin copying backup id psaphanacrp01.loves.com_1397937609 (duplicate-optimized), copy 1, fragment 1

    15:30:20.222 [18312.9608] <16> 2093578:bptm:18312:okcnbkp02.loves.com: [4788:2588] ddp_filecopy_start() failed, Err: 5005-nfs filecopy start failed (nfs: No space left on device)

    15:30:20.222 [18312.9608] <16> 2093578:bptm:18312:okcnbkp02.loves.com: F:\Program files\Veritas\NetBackup\bin\\ost-plugins\libstspiDataDomain.dll:stspi_copy_extent STS_EPLUGIN [DDErrNo = 5005 (no room left)]

    15:30:20.222 [18312.9608] <32> do_copy_extent: sts_copy_extent failed: error 2060046

    15:30:20.222 [18312.9608] <32> copy_data: image copy failed: error 2060046:

    15:30:20.222 [18312.9608] <16> copy_data: cannot copy image from disk, bytesCopied = 18446744073709551615

    15:30:20.222 [18312.9608] <2> validate_image_list: validating val_img_list

    15:30:20.222 [18312.9608] <2> delete_validate_image_list: deleting val_img_list”

  • SymTerry's avatar
    SymTerry
    Level 6
    11 years ago

    So your getting: DDErrNo = 5005 (no room left) and nfs: No space left on device

    Is your DD full because the plugin is getting push back with space issues.

  • David_Bowlby's avatar
    David_Bowlby
    Level 4
    11 years ago

    I have two datadomains both have 100 tb of space.  Neither is over 40% full. I run the cleaning every other day.  Backups are successful this only happens with slp duplications.    

     

    I have been trying to figure out what that error means by no room left.  

  • David_Bowlby's avatar
    David_Bowlby
    Level 4
    11 years ago

    Setting the DiskPool Max I/O Streams to 90 and worked my way up to 120 fixed this issue.