cancel
Showing results for 
Search instead for 
Did you mean: 

SLP Duplication failing with Error Code 174

shashi_pratap
Level 3
Accredited

Hi ,

I have a SLP duplication job failure with error code 174.

Below are the job details :-

9/23/2012 6:52:12 AM - Info bpdm(pid=30527) started           
9/23/2012 6:52:12 AM - started process bpdm (30527)
9/23/2012 6:52:12 AM - Info bpdm(pid=30527) requesting nbjm for media        
9/23/2012 6:53:12 AM - begin writing
9/23/2012 6:54:16 AM - end writing; write time: 00:01:04
9/23/2012 6:54:40 AM - Critical bpdm(pid=30527) get image properties failed: error 2060001: one or more invalid arguments 
9/23/2012 6:54:40 AM - Critical bpdm(pid=30527) Invalid image copy: error 2060001       
9/23/2012 6:54:43 AM - Critical bpdm(pid=30527) get image properties failed: error 2060001: one or more invalid arguments 
9/23/2012 6:54:56 AM - begin writing
9/23/2012 6:54:57 AM - end writing; write time: 00:00:01
9/23/2012 6:55:02 AM - Critical bpdm(pid=30527) sts_close_handle failed: 2060022 software error       
9/23/2012 6:55:06 AM - Critical bpdm(pid=30527) get image properties failed: error 2060001: one or more invalid arguments 
9/23/2012 6:55:06 AM - Critical bpdm(pid=30527) Invalid image copy for bk-fsd-01.cbk.gov.kw_1348142403_C2_F1_R35: error 2060001     
9/23/2012 6:55:22 AM - Info main-media-2(pid=30527) StorageServer=PureDisk:main-5020; Report=PDDO Stats for (main-5020): scanned: 4 KB, CR sent: 0 KB, CR sent over FC: 0 KB, dedup: 100.0%
9/23/2012 6:55:48 AM - Info main-media-2(pid=30527) StorageServer=PureDisk:dr-5020; Report=PDDO Stats for (dr-5020): scanned: 17454161 KB, CR sent: 2 KB, CR sent over FC: 0 KB, dedup: 100.0%
9/23/2012 7:50:03 AM - begin Duplicate
9/23/2012 7:50:03 AM - Info Duplicate(pid=29335) Initiating optimized duplication from @aaaal to @aaaae     
9/23/2012 7:50:03 AM - requesting resource LCM_NBU-APP-Main-media-2-STU
9/23/2012 7:50:03 AM - granted resource LCM_NBU-APP-Main-media-2-STU
9/23/2012 7:50:03 AM - started process RUNCMD (29335)
9/23/2012 7:50:03 AM - requesting resource NBU-APP-Main-media-2-STU
9/23/2012 7:50:03 AM - reserving resource @aaaal
9/23/2012 7:50:03 AM - reserved resource @aaaal
9/23/2012 7:50:03 AM - granted resource MediaID=@aaaae;DiskVolume=PureDiskVolume;DiskPool=Main-5020-DiskPool;Path=PureDiskVolume;StorageServ...
9/23/2012 7:50:03 AM - granted resource NBU-APP-Main-media-2-STU
9/23/2012 7:50:03 AM - requesting resource @aaaal
9/23/2012 7:50:03 AM - granted resource MediaID=@aaaal;DiskVolume=PureDiskVolume;DiskPool=Dr-5020-DiskPool;Path=PureDiskVolume;StorageServer...
9/23/2012 7:50:04 AM - ended process 0 (29335)
9/23/2012 7:53:39 AM - Error bpduplicate(pid=29335) host main-media-2 backup id bk-fsd-01.cbk.gov.kw_1348142403 optimized duplication failed, media manager - system error occurred (174).
9/23/2012 7:53:39 AM - Error bpduplicate(pid=29335) Duplicate of backupid bk-fsd-01.cbk.gov.kw_1348142403 failed, media manager - system error occurred (174).
9/23/2012 7:53:39 AM - Error bpduplicate(pid=29335) Status = no images were successfully processed.     
9/23/2012 7:53:39 AM - end Duplicate; elapsed time: 00:03:36
media manager - system error occurred(174).

 

The problem is that duplication has been successful for rest of the clients , and there was another client for which the duplication was failing with same error code,but it got successful eventually automatic retrials by SLP itslef , however this particular ID still keeps failing.

I also tried a new backup for this particular client , but again the SLP duplication was failing with the same error code.

 

Is it a client related issue or something worng with SLP.

Please suggest.

 

 

15 REPLIES 15

sksujeet
Level 6
Partner Accredited Certified

Please provide the NBU version

As per the detailed status it shows

9/23/2012 7:53:39 AM - Error bpduplicate(pid=29335) host main-media-2 backup id bk-fsd-01.cbk.gov.kw_1348142403 optimized duplication failed, media manager - system error occurred (174).
 

http://www.symantec.com/docs/TECH143365
 

If this issue is being experienced please contact Technical support and reference Etrack 2176297 and this document.  There are binaries files that can be provided that will resolve this issue.  These binaries are required to be installed on all media servers that have credentials for the storage servers. 

http://www.symantec.com/docs/TECH179483



 

shashi_pratap
Level 3
Accredited

Hi Sazz ,

Thanks for the response.

The NBU version is 7.5.0.1 and backup is taking place through 2 NBU 5220 Appliance media servers whivh have which have Appliance 5020 PureDisk pools configured ,onto which the backup is going eventually.

The SLP duplication job which is failing is actually optimized dupliaction jobs between two 5020 Appliance diskpools which are configured through means of two 5220 Appliance media servers.

Hope it will help.

sksujeet
Level 6
Partner Accredited Certified

In the catalog section search with the client bk-fsd-01.cbk.gov.kw and check if you are able to find the backup id bk-fsd-01.cbk.gov.kw_1348142403.

Also does any of the cleanup job failed with the above backup id.

This error occurs if a deduplication backup job fails after the job writes part of the backup to the MediaServer Deduplication Pool. NetBackup starts an image cleanup job, but that job fails because the data necessary to complete the image clean-up was not written to the Media Server Deduplication Pool.

Usually Deduplication queue processing cleans up the image objects, so you do not need to take corrective action.

How long this image is been failing? Have you ever had this image successfully completed before?

shashi_pratap
Level 3
Accredited

Hi Sazz,
I am able to find the backup-id bk-fsd-01.cbk.gov.kw_1348142403 in the catalog,but only the primary copy since the duplication is failing.
This image is failing since Thursday ,

Clean up job failed , it didn't failed due to this particular backup ID.

Moreover, the SLP duplication is also failing for this client for yesterday's backup too " bk-fsd-01.cbk.gov.kw_1348401604" and
for another client for which the SLP duplication was failing initially "bk-ev-01.cbk.gov.kw_1348146002" , but it got successful over the weekend.,
but now the SLP duplication for the backup of yesterday "bk-ev-01.cbk.gov.kw_1348405203"is failing again

I have attached the logs for 2 previously incomplete Image Cleanup Jobs.

Please suggest.

shashi_pratap
Level 3
Accredited

Hi ,

The primary copies get duplicated immediately via SLP and both copies are being retained for 2 weeks for daily backups.

I tried manually duplicating the particular backup ID bk-fsd-01.cbk.gov.kw_1348142403 , but it too failed with the same error 174.

This is the 1st time duplication error is encountered for this client,and we have only this client in this policy.

The other backup ID which was failing initially bk-ev-01.cbk.gov.kw_1348146002

9/20/2012 5:09:43 PM - Critical bpdm(pid=8878) get image properties failed: error 2060001: one or more invalid arguments 
9/20/2012 5:09:43 PM - Critical bpdm(pid=8878) Invalid image copy: error 2060001       
9/20/2012 5:09:47 PM - Critical bpdm(pid=8878) get image properties failed: error 2060001: one or more invalid arguments 
9/20/2012 5:10:01 PM - begin writing
9/20/2012 5:10:02 PM - end writing; write time: 00:00:01
9/20/2012 5:10:08 PM - Critical bpdm(pid=8878) sts_close_handle failed: 2060022 software error       
9/20/2012 5:10:12 PM - Critical bpdm(pid=8878) get image properties failed: error 2060001: one or more invalid arguments 
9/20/2012 5:10:12 PM - Critical bpdm(pid=8878) Invalid image copy for bk-ev-01.cbk.gov.kw_1348146002_C2_F1_R2: error 2060001     
9/20/2012 5:10:28 PM - Info main-media-2(pid=8878) StorageServer=PureDisk:main-5020; Report=PDDO Stats for (main-5020): scanned: 10 KB, CR sent: 0 KB, CR sent over FC: 0 KB, dedup: 100.0%
9/20/2012 5:10:28 PM - Info main-media-2(pid=8878) StorageServer=PureDisk:dr-5020; Report=PDDO Stats for (dr-5020): scanned: 36273935 KB, CR sent: 4 KB, CR sent over FC: 0 KB, dedup: 100.0%
9/20/2012 6:02:41 PM - begin Duplicate
9/20/2012 6:02:41 PM - Info Duplicate(pid=16106) Initiating optimized duplication from @aaaal to @aaaae     
9/20/2012 6:02:41 PM - requesting resource LCM_NBU-APP-Main-media-2-STU
9/20/2012 6:02:41 PM - granted resource LCM_NBU-APP-Main-media-2-STU
9/20/2012 6:02:41 PM - started process RUNCMD (16106)
9/20/2012 6:02:41 PM - requesting resource NBU-APP-Main-media-2-STU
9/20/2012 6:02:41 PM - reserving resource @aaaal
9/20/2012 6:02:41 PM - reserved resource @aaaal
9/20/2012 6:02:41 PM - granted resource MediaID=@aaaae;DiskVolume=PureDiskVolume;DiskPool=Main-5020-DiskPool;Path=PureDiskVolume;StorageServ...
9/20/2012 6:02:41 PM - granted resource NBU-APP-Main-media-2-STU
9/20/2012 6:02:41 PM - requesting resource @aaaal
9/20/2012 6:02:41 PM - granted resource MediaID=@aaaal;DiskVolume=PureDiskVolume;DiskPool=Dr-5020-DiskPool;Path=PureDiskVolume;StorageServer...
9/20/2012 6:02:42 PM - ended process 0 (16106)
9/20/2012 6:08:20 PM - Error bpduplicate(pid=16106) host main-media-2 backup id bk-ev-01.cbk.gov.kw_1348146002 optimized duplication failed, media manager - system error occurred (174).
9/20/2012 6:08:20 PM - Error bpduplicate(pid=16106) Duplicate of backupid bk-ev-01.cbk.gov.kw_1348146002 failed, media manager - system error occurred (174).
9/20/2012 6:08:20 PM - Error bpduplicate(pid=16106) Status = no images were successfully processed.     
9/20/2012 6:08:20 PM - end Duplicate; elapsed time: 00:05:39
media manager - system error occurred(174)

but it got completed successfully later.

9/20/2012 8:25:47 PM - Info bpdm(pid=24038) started           
9/20/2012 8:25:47 PM - started process bpdm (24038)
9/20/2012 8:25:47 PM - Info bpdm(pid=24038) reading backup image         
9/20/2012 8:25:47 PM - Info bpdm(pid=24038) using 30 data buffers        
9/20/2012 8:25:47 PM - Info bpdm(pid=24038) spawning a child process        
9/20/2012 8:25:47 PM - Info bpbrm(pid=24038) child pid: 24044         
9/20/2012 8:25:47 PM - Info bpdm(pid=24038) requesting nbjm for media        
9/20/2012 8:25:59 PM - begin reading
9/20/2012 9:20:26 PM - end reading; read time: 00:54:27
9/20/2012 9:20:29 PM - begin reading
9/20/2012 9:55:40 PM - end reading; read time: 00:35:11
9/20/2012 9:55:43 PM - begin reading
9/20/2012 10:22:28 PM - end reading; read time: 00:26:45
9/20/2012 10:22:31 PM - begin reading
9/20/2012 10:59:03 PM - end reading; read time: 00:36:32
9/20/2012 10:59:06 PM - begin reading
9/20/2012 11:43:13 PM - end reading; read time: 00:44:07
9/20/2012 11:43:17 PM - begin reading
9/21/2012 12:08:42 AM - Info bptm(pid=25746) waited for full buffer 88182 times, delayed 1039227 times   
9/21/2012 12:08:51 AM - Info bptm(pid=25746) EXITING with status 0 <----------       
9/21/2012 12:08:51 AM - Info main-media-2(pid=25746) StorageServer=PureDisk:main-5020; Report=PDDO Stats for (main-5020): scanned: 36273931 KB, CR sent: 12061 KB, CR sent over FC: 12061 KB, dedup: 100.0%, cache hits: 342839 (100.0%)
9/21/2012 12:24:51 AM - end reading; read time: 00:41:34
9/21/2012 12:24:54 AM - begin reading
9/21/2012 1:06:34 AM - end reading; read time: 00:41:40
9/21/2012 1:06:34 AM - Info bpdm(pid=24038) completed reading backup image        
9/21/2012 1:06:34 AM - Info bpdm(pid=24038) EXITING with status 0        
9/21/2012 1:06:34 AM - Info main-nb-mast(pid=24038) StorageServer=PureDisk:dr-5020; Report=PDDO Stats for (dr-5020): read: 36273932 KB, CR received: 36292624 KB, CR received over FC: 0 KB, dedup: 0.0%
9/21/2012 1:06:43 AM - end Duplicate; elapsed time: 04:43:14
the requested operation was successfully completed(0)

Mark_Solutions
Level 6
Partner Accredited Certified

if this is strarting to look intermittent then try adjusting the dps proxy timeout on Master and Media Servers (appliances).  To do that just create the flat file at:

/usr/openv/netbackup/db/config/DPS_PROXYDEFAULTRECVTMO

touch it with a value of 800 in it (or create it with vi)

So do this on the appliances and also on your Master Server (equivalent path if it is Windows but make sure there is no extention on the file)

This needs at least a service restart to take effect and if available reboot everything - you shouldn't need to do anything with the 5020 appliances, just the Master and 5200 media servers

See if this helps and let me know

shashi_pratap
Level 3
Accredited

Hi Mark ,

Sorry for the delayed response.

The problem is that we will need to forward request for downtime for restarting both the master n media server services , which is unlikely given the hectic backup activity.

We are still trying for other options ,

stranegly the duplication is failing for only 2 clients consecutively ,

and when I chnaged the storage units for these clients , so that the duplication goes otherway round ,

then the SLP duplication completed successfully for one of the clients , but it was still failing for the other 1.

Anything we should be looking from the client end ?

Please suggest.

Mark_Solutions
Level 6
Partner Accredited Certified

Ok - another quick check please

Find one of the images involved (bk-ev-01.cbk.gov.kw_1348146002) in the catalog section of the admin console for the copy 2 - right click and select verify

See if that actually works or fails

If if fails tell me if the client bk-ev-01.cbk.gov.kw appears in more than one policy and if so if it has the same case on both policies (bk-ev-01.cbk.gov.kw and BK-ev-01.cbk.gov.kw for example)

shashi_pratap
Level 3
Accredited

Hi Mark ,

 

I initiated the verify task for copy 2 of bk-ev-01.cbk.gov.kw_1348146002

and it completed successfully.

The client bk-ev-01.cbk.gov.kw is there only in 1 policy

Mark_Solutions
Level 6
Partner Accredited Certified

OK thanks

If you have put the DPS touch file in place then you can just try restarting nbrmms to get the changes to register - i am told that this does not have an affect on any running backups etc

For the media servers (can't remember what you master is but equivalent for that -or via activity monitor)


1.) At command line of appliance enter:  ps -ef | grep nbrmms
2.) From last command, enter pid as parameter to kill command:    kill <nbrmms pid number>
3.) Restart nbrmms:  /usr/openv/netbackup/bin/nbrmms
 

shashi_pratap
Level 3
Accredited

Hi Mark ,

Thanks for that info,

I created the file on the master server under /usr/openv/db/config with the value 800 ,

then killed and started the nbrmms service again.

But on the media sever , this file already existed with the vaule 800.

I have started the backup again.

Will update you shortly.

shashi_pratap
Level 3
Accredited

Hi Mark ,

The SLP duplication failed again with error 174.

However , for the same client , I changed the storage unit , took a backup and the duplication was also successful.the duplication went just the opposite of what it is failing now, I mean the primary source and destination storage units were swapped.

 

Mark_Solutions
Level 6
Partner Accredited Certified

Is it just one image or any that you do in that one direction?

shashi_pratap
Level 3
Accredited

For the client bk-fsd-01.cbk.gov.kw , for earlier images which failed during duplication as well , when I tried the backup opposite way , the duplication was also successfull ,

for bk-ev-01.cbk.gov.kw , the duplication failed while trying the reverse method too.

Mark_Solutions
Level 6
Partner Accredited Certified

Guess you need Symantec involved unless they already are - sounds like a host name resolution error but that really shouldnt be the case with a duplication!

Can't think what else it could be - I have to assume everything else is setup correctly as the rest of it works and it is just these two clients!

Will keep it in mind to see if anything pops into my grey matter!