cancel
Showing results for 
Search instead for 
Did you mean: 

NetBackup 7.5.0.7 Duplication Status 150 and 191

m_karampasis
Level 4
Partner Accredited

Dear all,

Duplication Status 150 and 191 (Could not get raw segment -  no such object. Refer to the replication logs on the source storage server for more information. V-454-105)

We are experiencing duplication status 150  for duplications to from disk to disk and 191 from disk to tape. We believe that the issue occurres due to big workload at the media server at 02:00 am (both backups and duplications jobs are trying to access the disk pool and read/write data). We have procceed at the following tune up's but the issue still occures:

  1. Maximum I/O streams on the Disk Pool = 60
  2. Changes to the LIFECYCLE_PARAMETERS file 
  3. Changes the Media server connect timeouts to 60 seconds from 30 which were the default value. 

          

Old LIFECYCLE_PARAMETERS

DUPLICATION_SESSION_INTERVAL_MINUTES = 5

MAX_MINUTES_TIL_FORCE_SMALL_DUPLICATION_JOB = 30

MAX_GB_SIZE_PER_DUPLICATION_JOB = 200

MIN_GB_SIZE_PER_DUPLICATION_JOB = 16

 

New LIFECYCLE_PARAMETERS

DUPLICATION_SESSION_INTERVAL_MINUTES = 30

MAX_MINUTES_TIL_FORCE_SMALL_DUPLICATION_JOB = 60

MAX_GB_SIZE_PER_DUPLICATION_JOB = 400

MIN_GB_SIZE_PER_DUPLICATION_JOB = 80


 

The logs when the issue occurs:

 

Detailed Status from failed duplication:

 

9 Μαϊ 2015 2:15:35 πμ - Info bpdm (pid=2545) Creating OPT_DUP_END marker

9 Μαϊ 2015 2:23:35 πμ - Critical bpdm (pid=2545) Storage Server Error: (Storage server: PureDisk:bs-kifissos) async_get_job_status: Replication started but failed to complete successfully:  __forward_SO: Could not get raw segment a7e4f1f4ff09cf55be3caee38aad60a0, len 16777216: no such object. Refer to the replication logs on the source storage server for more information. V-454-105

9 Μαϊ 2015 2:23:35 πμ - Critical bpdm (pid=2545) sts_copy_extent failed: error 2060014 operation aborted

9 Μαϊ 2015 2:23:42 πμ - Error bpduplicate (pid=19933) host bs-pireos backup id r6-osn6-kif_1431126003 optimized duplication failed, termination requested by administrator (150).

9 Μαϊ 2015 2:23:42 πμ - Info bs-pireos (pid=2545) StorageServer=PureDisk:bs-kifissos; Report=PDDO Stats for (bs-kifissos): scanned: 2 KB, CR sent: 0 KB, CR sent over FC: 0 KB, dedup: 100.0%

9 Μαϊ 2015 2:23:42 πμ - Error bpduplicate (pid=19933) db_IMAGE() failed: Images are in process. Please, run the nbstlutil command to get a list of those images if you need to cancel or expire them. (1519)

9 Μαϊ 2015 2:23:42 πμ - Error bpduplicate (pid=19933) Duplicate of backupid r6-osn6-kif_1431126003 failed, termination requested by administrator (150).

 

Replication logs on the source storage server (bs-kifissos):

 

May 09 02:16:31 INFO [140159293536000]: sessionStartAgent: Server is Version 7.0006.0013.112, Protocol Version 6.6.1

May 09 02:16:52 INFO [140159294064384]: sessionStartAgent: Server is Version 7.0006.0013.112, Protocol Version 6.6.1

May 09 02:17:54 ERR [140159294064384]: 2: __forward_SO: couldn't get raw segment ssize a7e4f1f4ff09cf55be3caee38aad60a0 len 16777216

May 09 02:17:54 ERR [140159294064384]: -1:  __forward_SO: Could not get raw segment a7e4f1f4ff09cf55be3caee38aad60a0, len 16777216: no such object

May 09 02:17:54 ERR [140159294064384]: 2: __replicate_SOs_of_DO:replication failed so fingerprint a7e4f1f4ff09cf55be3caee38aad60a0 (16777216 bytes)

May 09 02:17:54 ERR [140159294064384]: 2: __process_batch: replication failed batch send failed at DO 5ec160006a8a7722bd5068dc4da99166

May 09 02:17:54 ERR [140159294064384]: 2: CRReplicate: replication failed batch send failed at DO 5ec160006a8a7722bd5068dc4da99166

May 09 02:17:54 ERR [140159294064384]: 2: CRReplicate of batch failed: no such object

May 09 02:17:54 ERR [140159294064384]: 2: forward fingerprints failed: no such object (2)

May 09 02:17:59 INFO [140159294064384]: WSRequestExt: submitting

 

Thank you in regards.

1 ACCEPTED SOLUTION

Accepted Solutions

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

I do not have personal experience with the other 2 touch files, but if Symantec experts suggest to create them, then best to do so.

This TN says to restart  nbrmms (NetBackup Remote Manager and Monitor Service) to implement these settings:  http://www.symantec.com/docs/TECH156490 

There is no work-around.

View solution in original post

3 REPLIES 3

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

What kind of performance tuning was done on the media server itself? (Apart from what you have listed)

How much memory does the media server have and what is the size of MSDP pool?

Does /usr/openv/netbackup/db/config/DPS_PROXYDEFAULTRECVTMO file exist?
If so, what value in this file? 
(I have seen recommendation of 800).

More tuning parameters in these docs:

The NetBackup Backup Planning and Performance Tuning Guide, Release 7.5 and Release 7.6  
 http://www.symantec.com/docs/DOC7449 

NetBackup 7.6 Best Practices: Optimizing Performance

 

m_karampasis
Level 4
Partner Accredited

Dear Marianne,

 

We have only applied the tune ups at the media server as described at my previous post.

  • The RAM of the media is 16GB and the disk pool is 9 TB.
  • DPS_PROXYDEFAULTRECVTMO file do not exist at the media server and we are going to create it. Do you suggest also to create DPS_PROXYNOEXPIRE & DPS_PROXYDEFAULTSENDTMO? Is there a way to overcome the restart of the services after creating those files?

Thank you for link with the NetBackup 7.6 Best Practices: Optimizing Performance.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

I do not have personal experience with the other 2 touch files, but if Symantec experts suggest to create them, then best to do so.

This TN says to restart  nbrmms (NetBackup Remote Manager and Monitor Service) to implement these settings:  http://www.symantec.com/docs/TECH156490 

There is no work-around.