Exchange 2013 image duplication from tape to disk failed error 191

[mjalil]

Dear All,

when i attempt to start process of restoration for exchange 2013 DAG of which duplicating images from tape to disk,

as first step is doing initial import for the tape is completed successfully,

then when i initiate duplicating found images to disk i got the below error from the job:

07/08/2015 13:24:01 - begin Duplicate
07/08/2015 13:24:02 - requesting resource bmmi-exch-disk
07/08/2015 13:24:02 - requesting resource VA0002
07/08/2015 13:24:02 - reserving resource VA0002
07/08/2015 13:24:03 - Error bpduplicate (pid=29043) Could not lock backupid: EXDAG.BMMIHQ.ROOT.AD_1420552923, copy 3; unable to process request
07/08/2015 13:24:03 - Error bpduplicate (pid=29043) Duplicate of backupid EXDAG.BMMIHQ.ROOT.AD_1420552923 failed, unable to process request (228).
07/08/2015 13:24:03 - resource VA0002 reserved
07/08/2015 13:24:03 - granted resource  MediaID=@aaaac;Path=/Exchg_bkp;MediaServer=bmmi-nbusrv
07/08/2015 13:24:03 - granted resource  bmmi-exch-disk
07/08/2015 13:24:03 - granted resource  VA0002
07/08/2015 13:24:03 - granted resource  HP.ULTRIUM5-SCSI.001
07/08/2015 13:24:03 - Error bpduplicate (pid=29043) Could not lock backupid: EXDAG.BMMIHQ.ROOT.AD_1420639321, copy 1; unable to process request
07/08/2015 13:24:03 - Error bpduplicate (pid=29043) Duplicate of backupid EXDAG.BMMIHQ.ROOT.AD_1420639321 failed, unable to process request (228).
07/08/2015 13:24:03 - Error bpduplicate (pid=29043) Could not lock backupid: EXDAG.BMMIHQ.ROOT.AD_1420639323, copy 1; unable to process request
07/08/2015 13:24:03 - Error bpduplicate (pid=29043) Duplicate of backupid EXDAG.BMMIHQ.ROOT.AD_1420639323 failed, unable to process request (228).
07/08/2015 13:24:03 - Error bpduplicate (pid=29043) Could not lock backupid: EXDAG.BMMIHQ.ROOT.AD_1420639322, copy 1; unable to process request
07/08/2015 13:24:03 - Error bpduplicate (pid=29043) Duplicate of backupid EXDAG.BMMIHQ.ROOT.AD_1420639322 failed, unable to process request (228).
07/08/2015 13:24:03 - Error bpduplicate (pid=29043) Status = no images were successfully processed.
07/08/2015 13:24:04 - end Duplicate; elapsed time 0:00:03
no images were successfully processed  (191)

i have tried from many tapes with same error of locking backupid, as will when i initiate verifing the same images it through this error:

07/08/2015 09:21:09 - Error bpverify (pid=17011) Expected filename /Microsoft Exchange Database Availability Groups/ in database, found no more files.
07/08/2015 09:21:09 - Error bpverify (pid=17011) Expected filename /Microsoft Exchange Database Availability Groups/EXDAG/ in database, found no more files.
07/08/2015 09:21:09 - Error bpverify (pid=17011) Expected filename /Microsoft Exchange Database Availability Groups/EXDAG/Microsoft Information Store/ in database, found no more files.
07/08/2015 09:21:09 - Error bpverify (pid=17011) Expected filename /Microsoft Exchange Database Availability Groups/EXDAG/Microsoft Information Store/STD Database/ in database, found no more files.
07/08/2015 10:43:07 - Error bpverify (pid=17011) Expected filename /Microsoft Exchange Database Availability Groups/EXDAG/Microsoft Information Store/STD Database/Database in database, found no more files.
07/08/2015 10:43:07 - Error bpverify (pid=17011) Expected filename /Microsoft Exchange Database Availability Groups/EXDAG/Microsoft Information Store/STD Database/Database/STD Database.edb in database, found no more files.
07/08/2015 10:43:08 - Error bpverify (pid=17011) Expected filename /Microsoft Exchange Database Availability Groups/EXDAG/Microsoft Information Store/STD Database/DLLs in database, found no more files.
07/08/2015 10:43:08 - Error bpverify (pid=17011) Expected filename /Microsoft Exchange Database Availability Groups/EXDAG/Microsoft Information Store/STD Database/DLLs/ese.dll in database, found no more files.
07/08/2015 10:44:13 - Error bpverify (pid=17011) Expected filename /Microsoft Exchange Database Availability Groups/EXDAG/Microsoft Information Store/STD Database/Logs_1420552830 in database, found no more files.
07/08/2015 10:44:13 - Error bpverify (pid=17011) Expected filename /Microsoft Exchange Database Availability Groups/EXDAG/Microsoft Information Store/STD Database/Logs_1420552830/E040004098A.log in database, found no more files.
07/08/2015 10:44:13 - Error bpverify (pid=17011) At least 10 database compare errors occurred, not logging any more.

this is what i found in admin logs:

13:03:49.270 [27752] <2> bpduplicate: Duplicating policy EXCHANGE schedule EXCH_BAK backup id EXDAG.BMMIHQ.ROOT.AD_14205
52923 copy 2 created on 01/06/2015 17:02:03 on source media id VA0002
13:03:49.363 [27752] <4> bpduplicate: EMM interface already initialized (using cached master name bmmi-nbusrv).
13:03:49.400 [27752] <2> bpduplicate: version of write_host bmmi-nbusrv is 761200
13:03:49.400 [27752] <2> bpduplicate: locking EXDAG.BMMIHQ.ROOT.AD_1420552923 on lock_fd -1 copy 3 called at line <3205>
13:03:49.400 [27752] <2> db_lock_copy_byte: Sending LOCK_COPY
13:03:49.655 [27752] <16> bpduplicate: Could not lock backupid: EXDAG.BMMIHQ.ROOT.AD_1420552923, copy 3; unable to proce
ss request
13:03:49.656 [27752] <16> bpduplicate: Duplicate of backupid EXDAG.BMMIHQ.ROOT.AD_1420552923 failed, unable to process r
equest (228).

13:03:49.656 [27752] <2> bpduplicate: unlocking source EXDAG.BMMIHQ.ROOT.AD_1420552923 copy 2 called at line <3898> with
 lock_fd <-1>
13:03:49.656 [27752] <2> bpduplicate: Duplicating policy EXCHANGE schedule EXCH_BAK backup id EXDAG.BMMIHQ.ROOT.AD_14206
39321 copy 2 created on 01/07/2015 17:02:01 on source media id VA0002
13:03:49.674 [27752] <2> bpduplicate: locking EXDAG.BMMIHQ.ROOT.AD_1420639321 on lock_fd -1 copy 1 called at line <3205>
13:03:49.674 [27752] <2> db_lock_copy_byte: Sending LOCK_COPY
13:03:49.701 [27752] <16> bpduplicate: Could not lock backupid: EXDAG.BMMIHQ.ROOT.AD_1420639321, copy 1; unable to proce
ss request
13:03:49.701 [27752] <16> bpduplicate: Duplicate of backupid EXDAG.BMMIHQ.ROOT.AD_1420639321 failed, unable to process r
equest (228).

13:03:49.701 [27752] <2> bpduplicate: unlocking source EXDAG.BMMIHQ.ROOT.AD_1420639321 copy 2 called at line <3898> with
 lock_fd <-1>

i have gone to many forum discussions but didnt find similar issue, wierdly the backup is running smooth and staging from disk to tape gets completed successfully,

the version of netbackup is 7.6.1.2 on solaris 10

[Marianne]

bpverify results seems to indicate that expected data is not on tape.

Any chance that this tape may have been overwritten somehow?

[mnolan]

I am guessing that this is a UNIX master server?
If so, check /usr/openv/netbackup/db/images for the client name EXDAG.BMMIHQ.ROOT.AD.. 

Is it showing both uppercase and lower case? If both, dive down into the folder that would contain 1420639321 and see which has the info for it. If the info is contained in the lower case one, you may need to move the contents of one to the other and then link them.

[mjalil]

Dear Marianne,

even verify does give me an error, i have tried multiple tapes with same results of error

[mjalil]

Dear mnolan,

i have found the folders but they are empty from any files..

[mnolan]

Do you store them elsewhere, are they linked somewhere else? Otherwise this could be the cause?

[sdo]

Are you able to verify any other backup images - i.e. something other then the MS Exchange backups?  Are you able to duplicate any other backup images?  Or are they all unreadable?

Were the tapes encrypted by the source environment?  If you use Reports / Images on Media, you should be able to add a column which will show whether the image was encrypted or not.

[mjalil]

no, there no other location i moved the files to, if they are not there, am i able to recreate them?

[mjalil]

no, there no other location i moved the files to, if they are not there, am i able to recreate them?

[mjalil]

any other type of backups am able to verify, duplicate and restore, only MS Exchange

i dont encrypt the backups as not required in our business as tapes stored in safe location