Netbackup 7.5 Restore Failure error 2820

[stwali005]

All,

Our Vault Duplication Job completed successfully to tape. We run month end test restores for several servers, using a touch file to force NB to use copy 3 and use tapes to run the restores.

So far every restore has been successful except for 1 server. (LNCVIM)

error VMware policy restore error (2820)

with an error in policy of NBU status: 95, EMM status: Media does not exist

i checked the Vault logs and the LNCVIM duplicated successfully.

If you run a restore of of the primary copy - ie our datadomain - the job restores successfully - selecting 25th July as our month end date.

If you do the same from tape - copy 3 - it failes. Some exctra note is it is a VmWare Policy type we are using.

I have tested selecting one files, folders and the whole D drive and it still fails.

I kicked off another duplication and verifyed the duplication in the catalog as successful - it obviously then wrote the data to a different tape and this again is failing to restore with the above same error message.

Can anybody advise if they have seen this before, or if they require any more information to assist in resolving this?

Many thanks

[RiaanBadenhorst]

To test, make the tape primary copy and see if it works. Can then work back from there to figure out what is wrong.

[RamNagalla]

NBU status: 95, EMM status: Media does not exist

please post detail status of the failed restore job

also below command output for the media that is trying to use for restore..

nbemmcmd -listmedia -mediaid <string>

[Marianne]

I would like to see if NBU sees 3 valid copies for this image.

Please show image output for this client:

bpimagelist -client LNCVIM -d 07/25/2014 -e 07/26/2014 -L

[stwali005]

Nagalla, please find the details you requested below:

Command output:

C:\Windows\system32>nbemmcmd -listmedia -mediaid 000034

NBEMMCMD, Version: 7.5.0.5

====================================================================

Media GUID:                     d002e732-1ab8-4eac-acd2-3d56903a2ce2

Media ID:                       000034

Partner:                        -

Media Type:                     HCART2

Volume Group:                   000_00001_TLD

Application:                    Netbackup

Media Flags:                    1

Description:                    Scratch Media Label

Barcode:                        000034

Partner Barcode:                --------

Last Write Host:                ln1nb01.domain.com

Created:                        12/14/2012 11:02

Time Assigned:                  08/14/2014 11:09

First Mount:                    12/31/2012 03:50

Last Mount:                     08/14/2014 11:10

Volume Expiration:              -

Data Expiration:                01/27/2015 22:36

Last Written:                   08/14/2014 11:09

Last Read:                      -

Robot Type:                     TLD

Robot Control Host:             ln1nb01.domain.com

Robot Number:                   1

Slot:                           4

Side/Face:                      -

Cleanings Remaining:            -

Number of Mounts:               2

Maximum Mounts Allowed:         0

Media Status:

Kilobytes:                      14164174

Images:                         3

Valid Images:                   3

Retention Period:               6

Number of Restores:             0

Optical Header Size Bytes:      1024

Optical Sector Size Bytes:      0

Optical Partition Size Bytes:   0

Last Header Offset:             55337

Adamm Guid:                     00000000-0000-0000-0000-000000000000

Rsm Guid:                       00000000-0000-0000-0000-000000000000

Origin Host:                    NONE

Master Host:                    lncvnbmgt01.domain.com

Server Group:                   NO_SHARING_GROUP

Upgrade Conflicts Flag:

Pool Number:                    7

Volume Pool:                    ENCR_Monthly_Vault

Previous Pool Name:             Scratch

Vault Flags:                    -

Vault Container:                -

Vault Name:                     -

Vault Slot:                     -

Session ID:                     -

Date Vaulted:                   -

Return Date:                    -

Media on Hold:                  0

====================================================================

Command completed successfully.

Error code details

(Attempt 1)

14/08/2014 13:07:06 - begin Restore
14/08/2014 13:07:08 - media 000034 required
14/08/2014 13:07:08 - restoring image lncvimpr01_1406327802
14/08/2014 13:07:08 - Info bprd(pid=1480) Restoring from copy 4 of image created 07/25/14 23:36:42   
14/08/2014 13:07:12 - Info bptm(pid=6576) start           
14/08/2014 13:07:12 - started process bptm (6576)
14/08/2014 13:07:12 - Info bpdm(pid=6576) reading backup image         
14/08/2014 13:07:12 - Info bptm(pid=6576) using 64 data buffers        
14/08/2014 13:07:12 - Info bptm(pid=6576) spawning a child process        
14/08/2014 13:07:12 - Info bptm(pid=6576) child pid: 5264         
14/08/2014 13:07:12 - Info bptm(pid=5264) start           
14/08/2014 13:07:12 - started process bptm (5264)
14/08/2014 13:07:14 - end Restore; elapsed time: 00:00:08
Restore failed due to failure to read the header from the backup image(2841)

(Attempt 2)

14/08/2014 13:07:14 - begin Restore
14/08/2014 13:07:14 - requesting resource @AAAAD
14/08/2014 13:07:14 - Error nbjm(pid=8696) NBU status: 95, EMM status: Media does not exist   
14/08/2014 13:07:14 - Error nbjm(pid=8696) NBU status: 95, EMM status: Media does not exist   
14/08/2014 13:07:15 - Info bprd(pid=1480) Retrying restore from copy 4 of image created 07/25/14 23:36:42  
14/08/2014 13:07:15 - Info bptm(pid=6576) EXITING with status 95 <----------       
14/08/2014 13:07:18 - Info bptm(pid=8608) start           
14/08/2014 13:07:18 - started process bptm (8608)
14/08/2014 13:07:18 - Info bpdm(pid=8608) reading backup image         
14/08/2014 13:07:18 - Info bptm(pid=8608) using 64 data buffers        
14/08/2014 13:07:18 - Info bptm(pid=8608) spawning a child process        
14/08/2014 13:07:18 - Info bptm(pid=8608) child pid: 9212         
14/08/2014 13:07:19 - Info bptm(pid=9212) start           
14/08/2014 13:07:19 - started process bptm (9212)
14/08/2014 13:07:21 - Info bptm(pid=8608) EXITING with status 95 <----------       
14/08/2014 13:07:21 - end Restore; elapsed time: 00:00:07
VMware policy restore error(2820)
14/08/2014 13:07:21 - requesting resource @AAAAD
14/08/2014 13:07:21 - Error nbjm(pid=8696) NBU status: 95, EMM status: Media does not exist   
14/08/2014 13:07:21 - Error nbjm(pid=8696) NBU status: 95, EMM status: Media does not exist

[stwali005]

C:\Windows\system32>bpimagelist -client LNCVIMPR01 -d 07/25/2014 -e 07/26/2014 -L

Client:            lncvimpr01

Backup ID:         lncvimpr01_1406327802

Policy:            LNC_VMWare

Policy Type:       VMware (40)

Proxy Client:      (none specified)

Creator:           root

Name1:             (none specified)

Sched Label:       Monthly_Backup

Schedule Type:     FULL (0)

Retention Level:   6 months (6)

Backup Time:            25/07/2014 23:36:42 (1406327802)

Elapsed Time:      590 second(s)

Expiration Time:        27/01/2015 22:36:42 (1422398202)

Compressed:        no

Client Encrypted:  no

Kilobytes:         14164173

Number of Files:   62994

Number of Copies:  4

Number of Fragments:   10

Histogram:         0 0 0 0 0 0 0 0 0 0

DB Compressed:     no

Files File Name:   LNC_VMWare_1406327802_FULL.f

Previous Backup Files File Name:   (none specified)

Parent Backup Image File Name:   (none specified)

SW Version:        (none specified)

Options:           0x0

MPX:               0

TIR Info:          0

TIR Expiration:         01/01/1970 00:00:00 (0)

Keyword:           (none specified)

Ext Security Info: no

File Restore Raw:  yes

Image Dump Level:  0

File System Only:  no

Object Descriptor: (none specified)

Previous BI Time:       25/07/2014 23:36:42 (1406327802)

BI Full Time:           25/07/2014 23:36:42 (1406327802)

Request Pid:       0

Backup Status:     0

Stream Number:     105

Backup Copy:       Standard (0)

Files File size:     3036026

PFI type:     0

IMAGE_ATTRIBUTE:     0

Primary Copy:      1

Image Type:        0  (Regular)

Job ID:            1552374

Num Resumes:       3

Resume Expiration:      01/01/1970 00:00:00 (0)

Data Classification:    (none specified)

Data_Classification_ID: (none specified)

Storage Lifecycle Policy:    Monthly_Backup_LNC_LN1

Storage Lifecycle Policy Version:    6

STL_Completed:      3

Remote Expiration Time:      01/01/1970 00:00:00 (0)

Origin Master Server:  (none specified)

Origin Master GUID:    (none specified)

Snap Time:           25/07/2014 23:35:40 (1406327740)

IR Enabled:      no

Client Character Set:     0

Image On Hold:     0

Indexing Status:   0

Copy number:       1

 Fragment:         TIR (-1)

 Kilobytes:        7637

 Remainder:        455

 Media Type:       Disk (0)

 Density:          qscsi (0)

 File Num:         0

 ID:               @aaaad

 Host:             lncnb01.domain.com

 Block Size:       262144

 Offset:           0

 Media Date:            01/01/1970 00:00:00 (0)

 Dev Written On:   -1

 Flags:            0x0

 Media Descriptor:        1;DataDomain;lncemcdd01.domain.com;lncemcdd01-dp;lncemcdd01-lsu;0

 checkpoint:       0

 resume num:       0

 Key tag:          *NULL*

 STL tag:          *NULL*

 Copy on hold:     0

Copy number:       1

 Fragment:         1

 Kilobytes:        14156536

 Remainder:        0

 Media Type:       Disk (0)

 Density:          qscsi (0)

 File Num:         0

 ID:               @aaaad

 Host:             lncnb01.domain.com

 Block Size:       262144

 Offset:           0

 Media Date:            01/01/1970 00:00:00 (0)

 Dev Written On:   -1

 Flags:            0x0

 Media Descriptor:        1;DataDomain;lncemcdd01.domain.com;lncemcdd01-dp;lncemcdd01-lsu;0

 Expiration Time:       27/01/2015 22:36:42 (1422398202)

 MPX:              0

 retention_lvl:    6 months (6)

 Try to Keep Time:       27/01/2015 22:36:42 (1422398202)

 Copy Creation Time:       25/07/2014 23:46:32 (1406328392)

 Data Format:      Tar

 checkpoint:       0

 resume num:       0

 Copy on hold:     0

Copy number:       2

 Fragment:         TIR (-1)

 Kilobytes:        7637

 Remainder:        455

 Media Type:       Disk (0)

 Density:          qscsi (0)

 File Num:         0

 ID:               @aaaab

 Host:             ln1nb02.domain.com

 Block Size:       262144

 Offset:           0

 Media Date:            01/01/1970 00:00:00 (0)

 Dev Written On:   -1

 Flags:            0x0

 Media Descriptor:        1;DataDomain;ln1emcdd01.domain.com;ln1emcdd01-dp;ln1emcdd01-lsu;0

 checkpoint:       0

 resume num:       1

 Key tag:          *NULL*

 STL tag:          *NULL*

 Copy on hold:     0

Copy number:       2

 Fragment:         1

 Kilobytes:        14156536

 Remainder:        0

 Media Type:       Disk (0)

 Density:          qscsi (0)

 File Num:         0

 ID:               @aaaab

 Host:             ln1nb02.domain.com

 Block Size:       262144

 Offset:           0

 Media Date:            01/01/1970 00:00:00 (0)

 Dev Written On:   -1

 Flags:            0x0

 Media Descriptor:        1;DataDomain;ln1emcdd01.domain.com;ln1emcdd01-dp;ln1emcdd01-lsu;0

 Expiration Time:       27/01/2015 22:36:42 (1422398202)

 MPX:              0

 retention_lvl:    6 months (6)

 Try to Keep Time:       27/01/2015 22:36:42 (1422398202)

 Copy Creation Time:       25/07/2014 23:53:57 (1406328837)

 Data Format:      Tar

 checkpoint:       0

 resume num:       1

 Copy on hold:     0

Copy number:       3

 Fragment:         TIR (-1)

 Kilobytes:        7638

 Remainder:        0

 Media Type:       Media Manager (2)

 Density:          hcart2 (14)

 File Num:         292

 ID:               000450

 Host:             ln1nb01.domain.com

 Block Size:       262144

 Offset:           1446184

 Media Date:            27/07/2014 04:01:14 (1406430074)

 Dev Written On:   2

 Flags:            0x40  (Tape Encrypted)

 Media Descriptor:        ?

 checkpoint:       0

 resume num:       0

 Key tag:          2a2e3da16e31eaa8231b2f52d4e905ed8e6279252f57b029d27802e07edb8cf8

 STL tag:          *NULL*

 Copy on hold:     0

Copy number:       3

 Fragment:         1

 Kilobytes:        10240000

 Remainder:        0

 Media Type:       Media Manager (2)

 Density:          hcart2 (14)

 File Num:         290

 ID:               000450

 Host:             ln1nb01.domain.com

 Block Size:       262144

 Offset:           1390881

 Media Date:            27/07/2014 04:01:14 (1406430074)

 Dev Written On:   2

 Flags:            0x40  (Tape Encrypted)

 Media Descriptor:        ?

 Expiration Time:       22/07/2024 23:36:42 (1721687802)

 MPX:              0

 retention_lvl:    10 years (10)

 Try to Keep Time:       01/01/1970 00:00:00 (0)

 Copy Creation Time:       27/07/2014 06:52:16 (1406440336)

 Data Format:      Tar

 checkpoint:       0

 resume num:       0

 Copy on hold:     0

Copy number:       3

 Fragment:         2

 Kilobytes:        3916536

 Remainder:        0

 Media Type:       Media Manager (2)

 Density:          hcart2 (14)

 File Num:         291

 ID:               000450

 Host:             ln1nb01.domain.com

 Block Size:       262144

 Offset:           1430883

 Media Date:            27/07/2014 04:01:14 (1406430074)

 Dev Written On:   2

 Flags:            0x40  (Tape Encrypted)

 Media Descriptor:        ?

 checkpoint:       0

 resume num:       0

 Copy on hold:     0

Copy number:       4

 Fragment:         TIR (-1)

 Kilobytes:        7638

 Remainder:        0

 Media Type:       Media Manager (2)

 Density:          hcart2 (14)

 File Num:         3

 ID:               000034

 Host:             ln1nb01.domain.com

 Block Size:       262144

 Offset:           55305

 Media Date:            14/08/2014 11:09:31 (1408010971)

 Dev Written On:   0

 Flags:            0x40  (Tape Encrypted)

 Media Descriptor:        ?

 checkpoint:       0

 resume num:       0

 Key tag:          2a2e3da16e31eaa8231b2f52d4e905ed8e6279252f57b029d27802e07edb8cf8

 STL tag:          *NULL*

 Copy on hold:     0

Copy number:       4

 Fragment:         1

 Kilobytes:        10240000

 Remainder:        0

 Media Type:       Media Manager (2)

 Density:          hcart2 (14)

 File Num:         1

 ID:               000034

 Host:             ln1nb01.domain.com

 Block Size:       262144

 Offset:           2

 Media Date:            14/08/2014 11:09:31 (1408010971)

 Dev Written On:   0

 Flags:            0x40  (Tape Encrypted)

 Media Descriptor:        ?

 Expiration Time:       27/01/2015 22:36:42 (1422398202)

 MPX:              0

 retention_lvl:    6 months (6)

 Try to Keep Time:       01/01/1970 00:00:00 (0)

 Copy Creation Time:       14/08/2014 11:14:16 (1408011256)

 Data Format:      Tar

 checkpoint:       0

 resume num:       0

 Copy on hold:     0

Copy number:       4

 Fragment:         2

 Kilobytes:        3916536

 Remainder:        0

 Media Type:       Media Manager (2)

 Density:          hcart2 (14)

 File Num:         2

 ID:               000034

 Host:             ln1nb01.domain.com

 Block Size:       262144

 Offset:           40004

 Media Date:            14/08/2014 11:09:31 (1408010971)

 Dev Written On:   0

 Flags:            0x40  (Tape Encrypted)

 Media Descriptor:        ?

 checkpoint:       0

 resume num:       0

 Copy on hold:     0

[stwali005]

Marianne,

Just to clarify on the above output. There are 4 copies, because when we could not restore from tape 000450, we then ran another duplication job that ran successfully.

We then changed the touch file to use Copy 4 to try the restore and even though you can see it runs from copy 4 and tries to use the new tape 000034, it fails with the exact same error.

[RiaanBadenhorst · Accepted solution]

Restore failed due to failure to read the header from the backup image(2841)

The tape is encrypted, is the server (and drive) doing the restore abel to read the encrypted media?

[Marianne]

I don't understand... 

Attempt 1 
 identifies copy 4 location as  media 000034 and fails with this error:
Restore failed due to failure to read the header from the backup image(2841)

Attempt 2
identifies copy 4 as Disk (@AAAAD) that apparently no longer exists?
14/08/2014 13:07:14 - requesting resource @AAAAD
14/08/2014 13:07:14 - Error nbjm(pid=8696) NBU status: 95, EMM status: Media does not exist   
 

I also battle to understand how 2 different attempts have exactly the same bprd and bptm PIDs?

Attempt 1 
14/08/2014 13:07:08 - Info bprd(pid=1480) Restoring from copy 4 of image created 07/25/14 23:36:42   
14/08/2014 13:07:12 - Info bptm(pid=6576) start     

Attempt 2
4/08/2014 13:07:15 - Info bprd(pid=1480) Retrying restore from copy 4 of image created 07/25/14 23:36:42  
14/08/2014 13:07:15 - Info bptm(pid=6576) EXITING with status 95 <----------       

[watsons]

The 1st attempt seems to read from copy 4 (mediaID 000034), but it fails with error 2841. 

Probably 2nd attempt kicks off but this time it didn't try from tape 000034 but the disk @aaaad (copy 1).

I would suggest you assign copy4 to be primary copy and retry, rather than creating the touch file (was it ALT_RESTORE_COPY_NUMBER ?) to point it to copy4, as I am not sure how reliable that touch file can work. 

Also see the error 2841 explanation: http://www.symantec.com/docs/HOWTO91934