06-03-2015 12:54 AM
Dear all,
Duplication Status 150 and 191 (Could not get raw segment - no such object. Refer to the replication logs on the source storage server for more information. V-454-105)
We are experiencing duplication status 150 for duplications to from disk to disk and 191 from disk to tape. We believe that the issue occurres due to big workload at the media server at 02:00 am (both backups and duplications jobs are trying to access the disk pool and read/write data). We have procceed at the following tune up's but the issue still occures:
Old LIFECYCLE_PARAMETERS
DUPLICATION_SESSION_INTERVAL_MINUTES = 5
MAX_MINUTES_TIL_FORCE_SMALL_DUPLICATION_JOB = 30
MAX_GB_SIZE_PER_DUPLICATION_JOB = 200
MIN_GB_SIZE_PER_DUPLICATION_JOB = 16
New LIFECYCLE_PARAMETERS
DUPLICATION_SESSION_INTERVAL_MINUTES = 30
MAX_MINUTES_TIL_FORCE_SMALL_DUPLICATION_JOB = 60
MAX_GB_SIZE_PER_DUPLICATION_JOB = 400
MIN_GB_SIZE_PER_DUPLICATION_JOB = 80
The logs when the issue occurs:
Detailed Status from failed duplication:
9 Μαϊ 2015 2:15:35 πμ - Info bpdm (pid=2545) Creating OPT_DUP_END marker
9 Μαϊ 2015 2:23:35 πμ - Critical bpdm (pid=2545) Storage Server Error: (Storage server: PureDisk:bs-kifissos) async_get_job_status: Replication started but failed to complete successfully: __forward_SO: Could not get raw segment a7e4f1f4ff09cf55be3caee38aad60a0, len 16777216: no such object. Refer to the replication logs on the source storage server for more information. V-454-105
9 Μαϊ 2015 2:23:35 πμ - Critical bpdm (pid=2545) sts_copy_extent failed: error 2060014 operation aborted
9 Μαϊ 2015 2:23:42 πμ - Error bpduplicate (pid=19933) host bs-pireos backup id r6-osn6-kif_1431126003 optimized duplication failed, termination requested by administrator (150).
9 Μαϊ 2015 2:23:42 πμ - Info bs-pireos (pid=2545) StorageServer=PureDisk:bs-kifissos; Report=PDDO Stats for (bs-kifissos): scanned: 2 KB, CR sent: 0 KB, CR sent over FC: 0 KB, dedup: 100.0%
9 Μαϊ 2015 2:23:42 πμ - Error bpduplicate (pid=19933) db_IMAGE() failed: Images are in process. Please, run the nbstlutil command to get a list of those images if you need to cancel or expire them. (1519)
9 Μαϊ 2015 2:23:42 πμ - Error bpduplicate (pid=19933) Duplicate of backupid r6-osn6-kif_1431126003 failed, termination requested by administrator (150).
Replication logs on the source storage server (bs-kifissos):
May 09 02:16:31 INFO [140159293536000]: sessionStartAgent: Server is Version 7.0006.0013.112, Protocol Version 6.6.1
May 09 02:16:52 INFO [140159294064384]: sessionStartAgent: Server is Version 7.0006.0013.112, Protocol Version 6.6.1
May 09 02:17:54 ERR [140159294064384]: 2: __forward_SO: couldn't get raw segment ssize a7e4f1f4ff09cf55be3caee38aad60a0 len 16777216
May 09 02:17:54 ERR [140159294064384]: -1: __forward_SO: Could not get raw segment a7e4f1f4ff09cf55be3caee38aad60a0, len 16777216: no such object
May 09 02:17:54 ERR [140159294064384]: 2: __replicate_SOs_of_DO:replication failed so fingerprint a7e4f1f4ff09cf55be3caee38aad60a0 (16777216 bytes)
May 09 02:17:54 ERR [140159294064384]: 2: __process_batch: replication failed batch send failed at DO 5ec160006a8a7722bd5068dc4da99166
May 09 02:17:54 ERR [140159294064384]: 2: CRReplicate: replication failed batch send failed at DO 5ec160006a8a7722bd5068dc4da99166
May 09 02:17:54 ERR [140159294064384]: 2: CRReplicate of batch failed: no such object
May 09 02:17:54 ERR [140159294064384]: 2: forward fingerprints failed: no such object (2)
May 09 02:17:59 INFO [140159294064384]: WSRequestExt: submitting
Thank you in regards.
Solved! Go to Solution.
06-03-2015 07:54 AM
I do not have personal experience with the other 2 touch files, but if Symantec experts suggest to create them, then best to do so.
This TN says to restart nbrmms (NetBackup Remote Manager and Monitor Service) to implement these settings: http://www.symantec.com/docs/TECH156490
There is no work-around.
06-03-2015 01:12 AM
What kind of performance tuning was done on the media server itself? (Apart from what you have listed)
How much memory does the media server have and what is the size of MSDP pool?
Does /usr/openv/netbackup/db/config/DPS_PROXYDEFAULTRECVTMO file exist?
If so, what value in this file?
(I have seen recommendation of 800).
More tuning parameters in these docs:
The NetBackup Backup Planning and Performance Tuning Guide, Release 7.5 and Release 7.6
http://www.symantec.com/docs/DOC7449
NetBackup 7.6 Best Practices: Optimizing Performance
06-03-2015 04:26 AM
Dear Marianne,
We have only applied the tune ups at the media server as described at my previous post.
Thank you for link with the NetBackup 7.6 Best Practices: Optimizing Performance.
06-03-2015 07:54 AM
I do not have personal experience with the other 2 touch files, but if Symantec experts suggest to create them, then best to do so.
This TN says to restart nbrmms (NetBackup Remote Manager and Monitor Service) to implement these settings: http://www.symantec.com/docs/TECH156490
There is no work-around.