Exchange DAG restore - cannot find the backup image

Have you configured Master Server Properties... tab for    Distributed Application Restore Mapping ?

FS01 is the master server, it's also where I'm running the restore from.

Yes, I've got all the exchange servers in against the DAG (both FQDN and short name) in distributed application mapping restore:

I've also got the DAG name and all the seperate exchange servers in both the master and media additional servers entries.

In C:\Program Files\Veritas\NetBackup\db\client I have folders for all the exchange servers (FQDN names) and the old DAG name (short name) but not the new DAG, is that to be expected?

Yes - don't worry about the lack of \db\client\NEWDAG name just yet -  they come from the bpclient command BTW.

Anyway...

...are the CAS servers in the Distributed Application Restore Mapping too?  I think you already said the MS Exchange DAG nodes are also CAS nodes?  But is that a supported config?  I though CAS servers cannot be MS Exchange servers - but I'm probably wrong/confused/un-enlightened about this.

.

Can you do a bpimagelist -U for the VM backup images that go hand-in-hand with the MS Exchange image... I think you need to list the two images from 25th, these two:

hw-core-ex01.hw-group.co.uk_1451076257

hw-core-ex02.hw-group.co.uk_1451075478

Can we just take a moment to double check versions:

1) ESXi version and update and patch level (need the uber detail - sorry)

2) MS Exchange 2013 roll-up / CU / patch number?

3) Please confirm that original MS Exchange has not had *any* revisions to MS Exchange Server between the backup time, and now?

4) The original backup was taken using NetBackup Client v7.6.1.1 ?

5) Were all and I mean all MS Exchange servers *and* CAS nodes *all* running the same version of NetBackup Client at the time of the original backup?

6) And the client you are wanting to restore to is running NetBackup Client v7.6.1.2 ?

I know it seems a drag - but we need to rule some things out.

Thank you.

.

7) The restore target MS Exchange Server is running *exactly* the same version of MS Exchange as the original sourec backup MS Exchange Server DAG and CAS group?

Thanks for sticking with this!

Yep, all CAS (and mailbox) servers are in DARM, in Exchange 2013 the model has changed from seperate CAS/mailbox servers to servers that have both the CAS and mailbox roles on the same server.

Images lists:

C:\Program Files\Veritas\NetBackup\bin\admincmd>bpimagelist -L -backupid ex01.example.co.uk_1451076257

Client:            ex01.example.co.uk
Backup ID:         ex01.example.co.uk_1451076257
Policy:            EX-NEW
Policy Type:       VMware (40)
Proxy Client:      (none specified)
Creator:           root
Name1:             (none specified)
Sched Label:       03-Yearly
Schedule Type:     FULL (0)
Retention Level:   infinity (9)
Backup Time:            25/12/2015 20:44:17 (1451076257)
Elapsed Time:      21146 second(s)
Expiration Time:             INFINITY       (2147483647)
Maximum Expiration Time:             INFINITY       (2147483647)
Compressed:        no
Client Encrypted:  no
Kilobytes:         947823147
Number of Files:   644428
Number of Copies:  1
Number of Fragments:   2
Histogram:         0 0 0 0 0 0 0 0 0 0
DB Compressed:     no
Files File Name:   EX-NEW_1451076257_FULL.f
Previous Backup Files File Name:   (none specified)
Parent Backup Image File Name:   (none specified)
SW Version:        (none specified)
Options:           0x0
MPX:               1
TIR Info:          0
TIR Expiration:         01/01/1970 00:00:00 (0)
Keyword:           (none specified)
Ext Security Info: no
File Restore Raw:  yes
Image Dump Level:  0
File System Only:  no
Object Descriptor: (none specified)
Previous BI Time:       25/12/2015 20:44:17 (1451076257)
BI Full Time:           25/12/2015 20:44:17 (1451076257)
Request Pid:       0
Backup Status:     0
Stream Number:     5
Backup Copy:       Standard (0)
Files File size:     89700476
PFI type:     0
IMAGE_ATTRIBUTE:     0
Primary Copy:      1
Image Type:        0  (Regular)
Job ID:            7566
Num Resumes:       0
Resume Expiration:      01/01/1970 00:00:00 (0)
Data Classification:    (none specified)
Data_Classification_ID: (none specified)
Storage Lifecycle Policy:    (none specified)
Storage Lifecycle Policy Version:    0
STL_Completed:      0
Remote Expiration Time:      01/01/1970 00:00:00 (0)
Origin Master Server:  (none specified)
Origin Master GUID:    (none specified)
Snap Time:           25/12/2015 20:43:27 (1451076207)
IR Enabled:      no
Client Character Set:     0
Image On Hold:     0
Indexing Status:   0
Copy number:       1
 Fragment:         TIR (-1)
 Kilobytes:        87667
 Remainder:        0
 Media Type:       Media Manager (2)
 Density:          hcart (6)
 File Num:         4
 ID:               000257
 Host:             fs02.example.co.uk
 Block Size:       65536
 Offset:           18309863
 Media Date:            25/12/2015 20:02:03 (1451073723)
 Dev Written On:   0
 Flags:            0x0
 Media Descriptor:        ?
 checkpoint:       0
 resume num:       0
 Key tag:          *NULL*
 STL tag:          *NULL*
 Copy on hold:     0
Copy number:       1
 Fragment:         1
 Kilobytes:        947735480
 Remainder:        0
 Media Type:       Media Manager (2)
 Density:          hcart (6)
 File Num:         3
 ID:               000257
 Host:             fs02.example.co.uk
 Block Size:       65536
 Offset:           3501494
 Media Date:            25/12/2015 20:02:03 (1451073723)
 Dev Written On:   0
 Flags:            0x0
 Media Descriptor:        ?
 Expiration Time:            INFINITY       (2147483647)
 MPX:              0
 retention_lvl:    infinity (9)
 Try to Keep Time:       01/01/1970 00:00:00 (0)
 Copy Creation Time:       26/12/2015 02:36:43 (1451097403)
 Data Format:      Tar
 checkpoint:       0
 resume num:       0
 Copy on hold:     0

C:\Program Files\Veritas\NetBackup\bin\admincmd>bpimagelist -L -backupid ex02.example.co.uk_1451075478

Client:            ex02.example.co.uk
Backup ID:         ex02.example.co.uk_1451075478
Policy:            EX-NEW
Policy Type:       VMware (40)
Proxy Client:      (none specified)
Creator:           root
Name1:             (none specified)
Sched Label:       03-Yearly
Schedule Type:     FULL (0)
Retention Level:   infinity (9)
Backup Time:            25/12/2015 20:31:18 (1451075478)
Elapsed Time:      23154 second(s)
Expiration Time:             INFINITY       (2147483647)
Maximum Expiration Time:             INFINITY       (2147483647)
Compressed:        no
Client Encrypted:  no
Kilobytes:         958727130
Number of Files:   695121
Number of Copies:  1
Number of Fragments:   2
Histogram:         0 0 0 0 0 0 0 0 0 0
DB Compressed:     no
Files File Name:   EX-NEW_1451075478_FULL.f
Previous Backup Files File Name:   (none specified)
Parent Backup Image File Name:   (none specified)
SW Version:        (none specified)
Options:           0x0
MPX:               1
TIR Info:          0
TIR Expiration:         01/01/1970 00:00:00 (0)
Keyword:           (none specified)
Ext Security Info: no
File Restore Raw:  yes
Image Dump Level:  0
File System Only:  no
Object Descriptor: (none specified)
Previous BI Time:       25/12/2015 20:31:18 (1451075478)
BI Full Time:           25/12/2015 20:31:18 (1451075478)
Request Pid:       0
Backup Status:     0
Stream Number:     6
Backup Copy:       Standard (0)
Files File size:     96616190
PFI type:     0
IMAGE_ATTRIBUTE:     0
Primary Copy:      1
Image Type:        0  (Regular)
Job ID:            7565
Num Resumes:       0
Resume Expiration:      01/01/1970 00:00:00 (0)
Data Classification:    (none specified)
Data_Classification_ID: (none specified)
Storage Lifecycle Policy:    (none specified)
Storage Lifecycle Policy Version:    0
STL_Completed:      0
Remote Expiration Time:      01/01/1970 00:00:00 (0)
Origin Master Server:  (none specified)
Origin Master GUID:    (none specified)
Snap Time:           25/12/2015 20:30:22 (1451075422)
IR Enabled:      no
Client Character Set:     0
Image On Hold:     0
Indexing Status:   0
Copy number:       1
 Fragment:         TIR (-1)
 Kilobytes:        94418
 Remainder:        0
 Media Type:       Media Manager (2)
 Density:          hcart (6)
 File Num:         4
 ID:               000241
 Host:             fs02.example.co.uk
 Block Size:       65536
 Offset:           15852144
 Media Date:            25/12/2015 20:00:13 (1451073613)
 Dev Written On:   1
 Flags:            0x0
 Media Descriptor:        ?
 checkpoint:       0
 resume num:       0
 Key tag:          *NULL*
 STL tag:          *NULL*
 Copy on hold:     0
Copy number:       1
 Fragment:         1
 Kilobytes:        958632712
 Remainder:        0
 Media Type:       Media Manager (2)
 Density:          hcart (6)
 File Num:         3
 ID:               000241
 Host:             fs02.example.co.uk
 Block Size:       65536
 Offset:           873505
 Media Date:            25/12/2015 20:00:13 (1451073613)
 Dev Written On:   1
 Flags:            0x0
 Media Descriptor:        ?
 Expiration Time:            INFINITY       (2147483647)
 MPX:              0
 retention_lvl:    infinity (9)
 Try to Keep Time:       01/01/1970 00:00:00 (0)
 Copy Creation Time:       26/12/2015 02:57:12 (1451098632)
 Data Format:      Tar
 checkpoint:       0
 resume num:       0
 Copy on hold:     0

I've added all the exchange clients and the dag to both master and media additional servers (and rebooted), same error in bpbrm:

13:46:26.961 [2588.2592] <2> ConnectionCache::connectAndCache: Acquiring new connection for host fs01.example.co.uk, query type 293
13:46:26.961 [2588.2592] <2> vnet_pbxConnect: pbxConnectEx Succeeded
13:46:26.961 [2588.2592] <2> logconnections: BPDBM CONNECT FROM 192.168.252.122.49545 TO 192.168.252.121.1556 fd = 612
13:46:27.055 [2588.2592] <2> db_startrequest: protocol error 1
13:46:27.055 [2588.2592] <16> db_begin: db_startrequest() failed: operation requested by an invalid server
13:46:27.055 [2588.2592] <2> db_receiveIMAGEdependees: db_begin() failed: operation requested by an invalid server
13:46:27.055 [2588.2592] <16> restore_files_from_fs_view: cannot find the backup image that the view 10680 depends on
13:46:27.055 [2588.2592] <2> ConnectionCache::connectAndCache: Acquiring new connection for host fs01.example.co.uk, query type 1
13:46:27.055 [2588.2592] <2> vnet_pbxConnect: pbxConnectEx Succeeded

The fact that the error mentions images makes me think it's tied to the current issue which appears to be a disconnect between the vmware images and the MS-Exchange-Server images.

As far as the vmware backups go, I've got them but they're against the individual exchange servers rather than the DAG:

C:\Program Files\Veritas\NetBackup\bin\admincmd>bpimagelist -idonly -d 01/01/1970 00:00:00 -pt VMware -client ex01.example.co.uk
Time:      22/01/2016 22:48:58   ID: ex01.example.co.uk_1453502938   FULL (0)
Time:      15/01/2016 23:24:13   ID: ex01.example.co.uk_1452900253   FULL (0)
Time:      25/12/2015 20:44:17   ID: ex01.example.co.uk_1451076257   FULL (0)

C:\Program Files\Veritas\NetBackup\bin\admincmd>bpimagelist -idonly -d 01/01/1970 00:00:00 -pt VMware -client ex02.example.co.uk
Time:      30/01/2016 09:49:07   ID: ex02.example.co.uk_1454147347   FULL (0)
Time:      22/01/2016 23:19:23   ID: ex02.example.co.uk_1453504763   FULL (0)
Time:      15/01/2016 22:50:10   ID: ex02.example.co.uk_1452898210   FULL (0)
Time:      25/12/2015 20:31:18   ID: ex02.example.co.uk_1451075478   FULL (0)

1) ESXi version and update and patch level (need the uber detail - sorry)

ESXi hosts are all "VMware ESXi, 5.5.0, 2302651", vSphere is 5.5.0.2646482

2) MS Exchange 2013 roll-up / CU / patch number?

They're all on 2013 CU11 (build 15.0.1156.6) as we need a fix released in CU11, do you think that's the cause?

3) Please confirm that original MS Exchange has not had *any* revisions to MS Exchange Server between the backup time, and now?

The exchange servers have all gone from CU8 to CU11 since the backup was taken.

4) The original backup was taken using NetBackup Client v7.6.1.1 ?

Yes

5) Were all and I mean all MS Exchange servers *and* CAS nodes *all* running the same version of NetBackup Client at the time of the original backup?

Yes, they were also running v7.6.1.1, we've just moved to v7.6.1.2 across all Netbackup serves and exchange servers.

6) And the client you are wanting to restore to is running NetBackup Client v7.6.1.2 ?

Yes

7) The restore target MS Exchange Server is running *exactly* the same version of MS Exchange as the original sourec backup MS Exchange Server DAG and CAS group?

No, the source server was CU8, the target server is on CU11 currently. I know there's official support for CU8 in 7.6.1.1 and CU10 in 7.5.0.7 but we needed a fix in CU11.

I can try downgrading the target server to CU8 over the weekend and seeing if that resolves but it doesn't feel to me like it's related, it doesn't even get as far as talking to the server before it dies.

Ok - here's a thought that I share when I can... do Microsoft support the restore of MS Exchange across different CU updates?  If Microsoft don't then Veritas can't.  NetBackup is entirely in the hands of the MS Exchange APIs when it comes to backup and restore.  My thought is that meta-data and schemas can change within MS Exchange, even with a seemingly innocuous CU update - which might render backup and restore across even minor version revisions not possible.  All conjecture, and it's not helping.

If it were me, and I could, then I would ensure that we're dealing with the same versions on source and target.  But we can't always have we want.  And we have not proven that MS the Exchange APIs are rejecting the restore due to different versions - so it's probably unlikely (for now).

(I / we still need to look through the image lists for something - hopefully soon I will)

We need to look in more detail at the bpbrm log.  What logging level did you have on the bpbrm log?  Could you lift the bpbrm logging to 5, and then see what it says?