Exchange DAG restore - cannot find the backup image

[a1expi]

-Exchange 2013 DAG on Windows 2012 R2

-Netbackup 7.6.1.1 (Just upgraded to 7.6.1.2)

-VMware backups to tape (no disk)

-VSS provider installed, exchange admin guide followed

I'm having a really odd error when trying to do a restore of a database from a DAG backup (I'm aware I can't do GRT as i'm on tape).

The policy is set to backup the DAG exactly as described in the admin guide, the backup (and the ASC part) completes without any issue.

The first oddity is that when I come to do a restore I choose the DAG as the source client, I see the backups listed but I can't preview any media, the box to show the tape numbers is blank.

If I change the source client to be the underlying Exchange VM I can see the same backup points (But with the drives/folders rather than Information Store) and selecting something lets preview media work as normal.

When I come to restore a database the restore starts, it works out which server to restore to and then errors with:

04/02/2016 22:27:46 - Info bprd(pid=5788) Searched ( 14) files of (189) files for Restore Job ID 8214.xxx
04/02/2016 22:27:46 - Info bprd(pid=5788) Restoring from copy 1 of image created 12/25/15 20:00:15 from policy EX-NEW
04/02/2016 22:27:56 - Info tar32(pid=3452) Restore started           
04/02/2016 22:27:57 - Error bpbrm(pid=4332) cannot find the backup image that the view 10680 depends on  
04/02/2016 22:27:57 - Info tar32(pid=3452) done. status: 175: not all requested files were restored    
04/02/2016 22:27:57 - Error bpbrm(pid=4332) client restore EXIT STATUS 175: not all requested files were restored 

I have distributed application restore mapping set for the DAG, both with FQDN and short names for component host.

Does anybody have any idea about what's going on?

It's almost like it's struggling to link the images taken of the VM to the MS-Exchange-Server restore type.

[Mark_Solutions]

Where are you running the restore from (Master or client)

Have you setup a RDB for the restore?

Are the NBU service accounts on the exchange clients setup (client, legacy client, legacy network)

[a1expi]

Hi Mark, thanks for taking a look.

I've tried running the restore from both master and one of the clients, same result.

Yes the RDB is setup and ready for the restore.

The services are def setup now but they might not have been when the backup was taken, will this make a difference?

Full log:

05/02/2016 10:50:24 - begin Restore
05/02/2016 10:50:24 - Info bprd(pid=4580) Found (189) files in (1) images for Restore Job ID 8217.xxx  
05/02/2016 10:50:24 - restoring image DAG01.example.co.uk_1451073615
05/02/2016 10:50:30 - Info bprd(pid=4580) Searched ( 14) files of (189) files for Restore Job ID 8217.xxx
05/02/2016 10:50:30 - Info bprd(pid=4580) Restoring from copy 1 of image created 12/25/15 20:00:15 from policy HW-CORE-EX-NEW
05/02/2016 10:50:36 - Info bpbrm(pid=4624) EX07 is the host to restore to      
05/02/2016 10:50:48 - Info tar32(pid=18784) Restore started           
05/02/2016 10:50:48 - Error bpbrm(pid=4624) cannot find the backup image that the view 10680 depends on  
05/02/2016 10:50:49 - Info tar32(pid=18784) done. status: 175: not all requested files were restored    
05/02/2016 10:50:49 - Error bpbrm(pid=4624) client restore EXIT STATUS 175: not all requested files were restored  
05/02/2016 10:50:49 - restored image DAG01.example.co.uk_1451073615 - (not all requested files were restored(175)); restore time 0:00:25
05/02/2016 10:51:01 - end Restore; elapsed time: 0:00:37
MS-Exchange-Server policy restore error (2810)

[sdo]

Yes, the NetBackup Client 'services' which now use a service account logon would need to be restarted in order for teh restore.

If they weren't setup to use service accounts before, then can we be sure that the backup was valid?

.

This error looks interesting:

cannot find the backup image that the view 10680 depends on  

.

Can you show us the bpimagelist -idonly for the MS Exchange Policy Type and the client names involved in the DAG set-up?

i.e.

bpimagelist -idonly -d 01/01/1970 00:00:00 -pt MS-Exchange-Server -client blah

...and adjust the date so we don't go back in time too far...

...and change blah to be the client names...

...and run this command for each "client name" you see in each of the jobs (parent and child) for the backup jobs from which you want to restore... i.e. there could be several different names - and you need to enter the names in the command above - one per call to bpimagelist - in exactly the same spelling and upper and lowercase as shown in the original backup job.

[a1expi]

The services have definitely been restarted before the restore (all Netbackup/Exchange servers were restarted overnight and the problem persists this morning).

Output of bpimagelist:

C:\Program Files\Veritas\NetBackup\bin\admincmd>bpimagelist -idonly -d 01/01/1970 00:00:00 -pt MS-Exchange-Server -client DAG01.example.co.uk

Time:      30/01/2016 09:15:43   ID: DAG01.example.co.uk_1454145343   FULL (0)
Time:      22/01/2016 20:00:15   ID: DAG01.example.co.uk_1453492815   FULL (0)
Time:      15/01/2016 20:00:15   ID: DAG01.example.co.uk_1452888015   FULL (0)
Time:      15/01/2016 20:00:13   ID: DAG01.example.co.uk_1452888013   FULL (0)
Time:      25/12/2015 20:00:18   ID: DAG01.example.co.uk_1451073618   FULL (0)
Time:      25/12/2015 20:00:15   ID: DAG01.example.co.uk_1451073615   FULL (0)

The image from 25/12/2015 is the one I'm testing the restore from.

The  client listed in the parent job (Under "Client" on job overview) is DAG01.example.co.uk, the client listed in the child job (in the same place) is EX07.

There are no images brought back when searching for images for the client "EX07" with bpimagelist.

EX07 is the target server for the restore and holds the RDB database. The original server that held the mailbox was EX01.

For reference I've got 4 Exchange 2013 servers (EX01,EX02,EX07 and EX08) all are both CAS and Mailbox servers.

Full log from parent job:

05/02/2016 10:50:09 - begin Restore
05/02/2016 10:50:09 - Info bprd(pid=1740) Found (189) files in (1) images for Restore Job ID 8216.xxx  
05/02/2016 10:50:15 - Info bprd(pid=1740) starting BPRESOLVER           
05/02/2016 10:50:17 - Info bprd(pid=1740) BPRESOLVER has executed on server (EX08)       
05/02/2016 10:51:01 - Error bprd(pid=1740) Cannot process target restore path        
05/02/2016 10:51:02 - end Restore; elapsed time: 0:00:53
the restore failed to recover the requested files (5)

Full log from child job:

05/02/2016 10:50:24 - begin Restore
05/02/2016 10:50:24 - Info bprd(pid=4580) Found (189) files in (1) images for Restore Job ID 8217.xxx  
05/02/2016 10:50:24 - restoring image DAG01.example.co.uk_1451073615
05/02/2016 10:50:30 - Info bprd(pid=4580) Searched ( 14) files of (189) files for Restore Job ID 8217.xxx
05/02/2016 10:50:30 - Info bprd(pid=4580) Restoring from copy 1 of image created 12/25/15 20:00:15 from policy EX-NEW
05/02/2016 10:50:36 - Info bpbrm(pid=4624) EX07 is the host to restore to      
05/02/2016 10:50:48 - Info tar32(pid=18784) Restore started           
05/02/2016 10:50:48 - Error bpbrm(pid=4624) cannot find the backup image that the view 10680 depends on  
05/02/2016 10:50:49 - Info tar32(pid=18784) done. status: 175: not all requested files were restored    
05/02/2016 10:50:49 - Error bpbrm(pid=4624) client restore EXIT STATUS 175: not all requested files were restored  
05/02/2016 10:50:49 - restored image DAG01.example.co.uk_1451073615 - (not all requested files were restored(175)); restore time 0:00:25
05/02/2016 10:51:01 - end Restore; elapsed time: 0:00:37
MS-Exchange-Server policy restore error (2810)

[sdo]

I think you/we need to check the bpbrm log on the client:

Error bpbrm(pid=4624) cannot find the backup image that the view 10680 depends on

Are you able to inspect the brbrm log?

[a1expi]

There's nothing in brbrm on the client (EX07) or the master server (FS01, 192.168.252.121) but this is the snip from the log on the media server (FS02, 192.168.252.122):

10:50:48.625 [4624.3712] <2> bpbrm read_media_msg: read from media manager: COMPOSITE_FS_RESTORE Start filelist
10:50:48.641 [4624.3712] <2> bpbrm read_media_msg: read from media manager: COMPOSITE_FS_RESTORE File /C/Program Files/Veritas/NetBackup/online_util/fi_cntl/bpfis.fim.ex01.example.co.uk_1451073616.1.*
10:50:48.641 [4624.3712] <2> bpbrm read_media_msg: read from media manager: COMPOSITE_FS_RESTORE End filelist NONE
10:50:48.641 [4624.3712] <2> ConnectionCache::connectAndCache: Acquiring new connection for host fs01.example.co.uk, query type 293
10:50:48.657 [4624.3712] <2> vnet_pbxConnect: pbxConnectEx Succeeded
10:50:48.657 [4624.3712] <2> logconnections: BPDBM CONNECT FROM 192.168.252.122.51801 TO 192.168.252.121.1556 fd = 632
10:50:48.766 [4624.3712] <2> db_startrequest: protocol error 1
10:50:48.766 [4624.3712] <16> db_begin: db_startrequest() failed: operation requested by an invalid server
10:50:48.766 [4624.3712] <2> db_receiveIMAGEdependees: db_begin() failed: operation requested by an invalid server
10:50:48.766 [4624.3712] <16> restore_files_from_fs_view: cannot find the backup image that the view 10680 depends on
10:50:48.766 [4624.3712] <2> ConnectionCache::connectAndCache: Acquiring new connection for host fs01.example.co.uk, query type 1
10:50:48.782 [4624.3712] <2> vnet_pbxConnect: pbxConnectEx Succeeded
10:50:48.782 [4624.3712] <2> logconnections: BPDBM CONNECT FROM 192.168.252.122.51802 TO 192.168.252.121.1556 fd = 632
10:50:48.891 [4624.3712] <2> db_end: Need to collect reply
10:50:48.907 [4624.3712] <2> restore_files_from_fs_view: COMPOSITE_FS_STATUS 37
10:50:48.907 [4624.3712] <2> restore_files_from_fs_view: wrote COMPOSITE RESTORE STATUS on client socket
10:50:48.907 [4624.3712] <2> bpbrm read_media_msg: read from media manager: LOG_MESSAGE 2 4 INF - BPPFI EXITING WITH STATUS = 175
10:50:48.907 [4624.3712] <2> bpbrm write_msg_to_progress_file: (8217.001) INF - BPPFI EXITING WITH STATUS = 175

10:50:48.907 [4624.3712] <2> bpbrm read_media_msg: read from media manager: EXIT DAG01.example.co.uk_1451073615 175
10:50:48.907 [4624.3712] <2> pfi_media_msg_handler: (DAG01.example.co.uk_1451073615) PFI completion status 175: not all requested files were restored
10:50:48.907 [4624.3712] <2> bpbrm remove_client_rename_file: removing rename file /usr/openv/netbackup/.rename.4624 on client EX07
10:50:48.907 [4624.3712] <2> ConnectionCache::connectAndCache: Acquiring new connection for host fs01.example.co.uk, query type 223
10:50:48.938 [4624.3712] <2> vnet_pbxConnect: pbxConnectEx Succeeded

All the clients (EX01/EX02/EX07/EX08) have the master server and media server configured in the client.

All the firewalls are turned off on all servers.

[Mark_Solutions]

I have a horrible feeling that the error relates to the application state capture image and backupimage itself.

When a VMware backup runs it runs both jobs and each gets recorded as an image which is why you see two backups for each date in your output.

Why it cannot tie the two together i am not sure but it may be worth looking at the full output for each of the images to see if anything is of note (such as one being primary and one secondary etc:

bpimagelist -L -backupid DAG01.example.co.uk_1451073615

bpimagelist -L -backupid DAG01.example.co.uk_1451073618

[a1expi]

Output of the two commands below.

The only warnings I've had in the backups have been if for some reason there are no active mailboxes on that specific server when it's backed up. If that happens there's a warning from the Symantec VSS writer that there's no active databases on that particular host (as expected).

In this specific backup, half the databases were on EX01 and half on EX02 so I thought this was why there was two VMware backups taken (one of EX01 and one of EX02) and there are two restore points for Exchange data for that date (each with half the databases in).

The exchange data images appear to be there, the VMware images of the underlying Exchange VMs seem to be there, but it's like it's struggling to tie the two togther.

I assume the MS-Exchange-Server images are "virtual", generated based on the one-pass VMware backup?

C:\Program Files\Veritas\NetBackup\bin\admincmd>bpimagelist -L -backupid DAG01.example.co.uk_1451073615

Client:            DAG01.example.co.uk
Backup ID:         DAG01.example.co.uk_1451073615
Policy:            EX-NEW
Policy Type:       Ms-Exchange-Server (16)
Proxy Client:      (none specified)
Creator:           root
Name1:             (none specified)
Sched Label:       03-Yearly
Schedule Type:     FULL (0)
Retention Level:   infinity (9)
Backup Time:            25/12/2015 20:00:15 (1451073615)
Elapsed Time:      681 second(s)
Expiration Time:             INFINITY       (2147483647)
Maximum Expiration Time:        01/01/1970 00:00:00 (0)
Compressed:        no
Client Encrypted:  no
Kilobytes:         0
Number of Files:   189
Number of Copies:  0
Number of Fragments:   0
Histogram:         0 0 0 0 0 0 0 0 0 0
DB Compressed:     no
Files File Name:   EX-NEW_1451073615_FULL_10680.f
Previous Backup Files File Name:   (none specified)
Parent Backup Image File Name:   (none specified)
SW Version:        (none specified)
Options:           0x0
MPX:               4
TIR Info:          0
TIR Expiration:         01/01/1970 00:00:00 (0)
Keyword:           (none specified)
Ext Security Info: no
File Restore Raw:  no
Image Dump Level:  0
File System Only:  no
Object Descriptor: (none specified)
Previous BI Time:       01/01/1970 00:00:00 (0)
BI Full Time:           01/01/1970 00:00:00 (0)
Request Pid:       0
Backup Status:     0
Stream Number:     0
Backup Copy:       Standard (0)
Files File size:     67750
PFI type:     0
IMAGE_ATTRIBUTE:     16
Primary Copy:      0
Image Type:        0  (Regular)
Job ID:            7555
Num Resumes:       0
Resume Expiration:      01/01/1970 00:00:00 (0)
Data Classification:    (none specified)
Data_Classification_ID: (none specified)
Storage Lifecycle Policy:    (none specified)
Storage Lifecycle Policy Version:    0
STL_Completed:      0
Remote Expiration Time:      01/01/1970 00:00:00 (0)
Origin Master Server:  (none specified)
Origin Master GUID:    (none specified)
Snap Time:           25/12/2015 20:00:15 (1451073615)
IR Enabled:      no
Client Character Set:     0
Image On Hold:     0
Indexing Status:   0

C:\Program Files\Veritas\NetBackup\bin\admincmd>bpimagelist -L -backupid DAG01.example.co.uk_1451073618

Client:            DAG01.example.co.uk
Backup ID:         DAG01.example.co.uk_1451073618
Policy:            EX-NEW
Policy Type:       Ms-Exchange-Server (16)
Proxy Client:      (none specified)
Creator:           root
Name1:             (none specified)
Sched Label:       03-Yearly
Schedule Type:     FULL (0)
Retention Level:   infinity (9)
Backup Time:            25/12/2015 20:00:18 (1451073618)
Elapsed Time:      660 second(s)
Expiration Time:             INFINITY       (2147483647)
Maximum Expiration Time:        01/01/1970 00:00:00 (0)
Compressed:        no
Client Encrypted:  no
Kilobytes:         0
Number of Files:   180
Number of Copies:  0
Number of Fragments:   0
Histogram:         0 0 0 0 0 0 0 0 0 0
DB Compressed:     no
Files File Name:   EX-NEW_1451073618_FULL_10681.f
Previous Backup Files File Name:   (none specified)
Parent Backup Image File Name:   (none specified)
SW Version:        (none specified)
Options:           0x0
MPX:               4
TIR Info:          0
TIR Expiration:         01/01/1970 00:00:00 (0)
Keyword:           (none specified)
Ext Security Info: no
File Restore Raw:  no
Image Dump Level:  0
File System Only:  no
Object Descriptor: (none specified)
Previous BI Time:       01/01/1970 00:00:00 (0)
BI Full Time:           01/01/1970 00:00:00 (0)
Request Pid:       0
Backup Status:     0
Stream Number:     0
Backup Copy:       Standard (0)
Files File size:     64598
PFI type:     0
IMAGE_ATTRIBUTE:     16
Primary Copy:      0
Image Type:        0  (Regular)
Job ID:            7556
Num Resumes:       0
Resume Expiration:      01/01/1970 00:00:00 (0)
Data Classification:    (none specified)
Data_Classification_ID: (none specified)
Storage Lifecycle Policy:    (none specified)
Storage Lifecycle Policy Version:    0
STL_Completed:      0
Remote Expiration Time:      01/01/1970 00:00:00 (0)
Origin Master Server:  (none specified)
Origin Master GUID:    (none specified)
Snap Time:           25/12/2015 20:00:18 (1451073618)
IR Enabled:      no
Client Character Set:     0
Image On Hold:     0
Indexing Status:   0

[Mark_Solutions]

Both of those say ms-exchange server backup type .. both ave a number of files but no KB .. which indicate they are both application backups

There doesnt seem to be a vmware backup with KB there .. perhaps that is what is missing?

The unautorised client in your earlier log i have seen before .. is the distributed application part hasnt worked then as a workaround add the exchnage dag and mailbox servers to the master and media servers "additional servers" list in their host properties

[Mark_Solutions]

So you have DAG01, EX01 and EX02 .. so why is FS01 making the restroe request? ... this looks like it is the unauthorised client?

[sdo]

Have you configured Master Server Properties... tab for    Distributed Application Restore Mapping ?

[a1expi]

FS01 is the master server, it's also where I'm running the restore from.

Yes, I've got all the exchange servers in against the DAG (both FQDN and short name) in distributed application mapping restore:

I've also got the DAG name and all the seperate exchange servers in both the master and media additional servers entries.

In C:\Program Files\Veritas\NetBackup\db\client I have folders for all the exchange servers (FQDN names) and the old DAG name (short name) but not the new DAG, is that to be expected?

[sdo]

Yes - don't worry about the lack of \db\client\NEWDAG name just yet -  they come from the bpclient command BTW.

Anyway...

...are the CAS servers in the Distributed Application Restore Mapping too?  I think you already said the MS Exchange DAG nodes are also CAS nodes?  But is that a supported config?  I though CAS servers cannot be MS Exchange servers - but I'm probably wrong/confused/un-enlightened about this.

.

Can you do a bpimagelist -U for the VM backup images that go hand-in-hand with the MS Exchange image... I think you need to list the two images from 25th, these two:

hw-core-ex01.hw-group.co.uk_1451076257

hw-core-ex02.hw-group.co.uk_1451075478

[sdo]

Can we just take a moment to double check versions:

1) ESXi version and update and patch level (need the uber detail - sorry)

2) MS Exchange 2013 roll-up / CU / patch number?

3) Please confirm that original MS Exchange has not had *any* revisions to MS Exchange Server between the backup time, and now?

4) The original backup was taken using NetBackup Client v7.6.1.1 ?

5) Were all and I mean all MS Exchange servers *and* CAS nodes *all* running the same version of NetBackup Client at the time of the original backup?

6) And the client you are wanting to restore to is running NetBackup Client v7.6.1.2 ?

I know it seems a drag - but we need to rule some things out.

Thank you.

.

7) The restore target MS Exchange Server is running *exactly* the same version of MS Exchange as the original sourec backup MS Exchange Server DAG and CAS group?

[a1expi]

Thanks for sticking with this!

Yep, all CAS (and mailbox) servers are in DARM, in Exchange 2013 the model has changed from seperate CAS/mailbox servers to servers that have both the CAS and mailbox roles on the same server.

Images lists:

C:\Program Files\Veritas\NetBackup\bin\admincmd>bpimagelist -L -backupid ex01.example.co.uk_1451076257

Client:            ex01.example.co.uk
Backup ID:         ex01.example.co.uk_1451076257
Policy:            EX-NEW
Policy Type:       VMware (40)
Proxy Client:      (none specified)
Creator:           root
Name1:             (none specified)
Sched Label:       03-Yearly
Schedule Type:     FULL (0)
Retention Level:   infinity (9)
Backup Time:            25/12/2015 20:44:17 (1451076257)
Elapsed Time:      21146 second(s)
Expiration Time:             INFINITY       (2147483647)
Maximum Expiration Time:             INFINITY       (2147483647)
Compressed:        no
Client Encrypted:  no
Kilobytes:         947823147
Number of Files:   644428
Number of Copies:  1
Number of Fragments:   2
Histogram:         0 0 0 0 0 0 0 0 0 0
DB Compressed:     no
Files File Name:   EX-NEW_1451076257_FULL.f
Previous Backup Files File Name:   (none specified)
Parent Backup Image File Name:   (none specified)
SW Version:        (none specified)
Options:           0x0
MPX:               1
TIR Info:          0
TIR Expiration:         01/01/1970 00:00:00 (0)
Keyword:           (none specified)
Ext Security Info: no
File Restore Raw:  yes
Image Dump Level:  0
File System Only:  no
Object Descriptor: (none specified)
Previous BI Time:       25/12/2015 20:44:17 (1451076257)
BI Full Time:           25/12/2015 20:44:17 (1451076257)
Request Pid:       0
Backup Status:     0
Stream Number:     5
Backup Copy:       Standard (0)
Files File size:     89700476
PFI type:     0
IMAGE_ATTRIBUTE:     0
Primary Copy:      1
Image Type:        0  (Regular)
Job ID:            7566
Num Resumes:       0
Resume Expiration:      01/01/1970 00:00:00 (0)
Data Classification:    (none specified)
Data_Classification_ID: (none specified)
Storage Lifecycle Policy:    (none specified)
Storage Lifecycle Policy Version:    0
STL_Completed:      0
Remote Expiration Time:      01/01/1970 00:00:00 (0)
Origin Master Server:  (none specified)
Origin Master GUID:    (none specified)
Snap Time:           25/12/2015 20:43:27 (1451076207)
IR Enabled:      no
Client Character Set:     0
Image On Hold:     0
Indexing Status:   0
Copy number:       1
 Fragment:         TIR (-1)
 Kilobytes:        87667
 Remainder:        0
 Media Type:       Media Manager (2)
 Density:          hcart (6)
 File Num:         4
 ID:               000257
 Host:             fs02.example.co.uk
 Block Size:       65536
 Offset:           18309863
 Media Date:            25/12/2015 20:02:03 (1451073723)
 Dev Written On:   0
 Flags:            0x0
 Media Descriptor:        ?
 checkpoint:       0
 resume num:       0
 Key tag:          *NULL*
 STL tag:          *NULL*
 Copy on hold:     0
Copy number:       1
 Fragment:         1
 Kilobytes:        947735480
 Remainder:        0
 Media Type:       Media Manager (2)
 Density:          hcart (6)
 File Num:         3
 ID:               000257
 Host:             fs02.example.co.uk
 Block Size:       65536
 Offset:           3501494
 Media Date:            25/12/2015 20:02:03 (1451073723)
 Dev Written On:   0
 Flags:            0x0
 Media Descriptor:        ?
 Expiration Time:            INFINITY       (2147483647)
 MPX:              0
 retention_lvl:    infinity (9)
 Try to Keep Time:       01/01/1970 00:00:00 (0)
 Copy Creation Time:       26/12/2015 02:36:43 (1451097403)
 Data Format:      Tar
 checkpoint:       0
 resume num:       0
 Copy on hold:     0

C:\Program Files\Veritas\NetBackup\bin\admincmd>bpimagelist -L -backupid ex02.example.co.uk_1451075478

Client:            ex02.example.co.uk
Backup ID:         ex02.example.co.uk_1451075478
Policy:            EX-NEW
Policy Type:       VMware (40)
Proxy Client:      (none specified)
Creator:           root
Name1:             (none specified)
Sched Label:       03-Yearly
Schedule Type:     FULL (0)
Retention Level:   infinity (9)
Backup Time:            25/12/2015 20:31:18 (1451075478)
Elapsed Time:      23154 second(s)
Expiration Time:             INFINITY       (2147483647)
Maximum Expiration Time:             INFINITY       (2147483647)
Compressed:        no
Client Encrypted:  no
Kilobytes:         958727130
Number of Files:   695121
Number of Copies:  1
Number of Fragments:   2
Histogram:         0 0 0 0 0 0 0 0 0 0
DB Compressed:     no
Files File Name:   EX-NEW_1451075478_FULL.f
Previous Backup Files File Name:   (none specified)
Parent Backup Image File Name:   (none specified)
SW Version:        (none specified)
Options:           0x0
MPX:               1
TIR Info:          0
TIR Expiration:         01/01/1970 00:00:00 (0)
Keyword:           (none specified)
Ext Security Info: no
File Restore Raw:  yes
Image Dump Level:  0
File System Only:  no
Object Descriptor: (none specified)
Previous BI Time:       25/12/2015 20:31:18 (1451075478)
BI Full Time:           25/12/2015 20:31:18 (1451075478)
Request Pid:       0
Backup Status:     0
Stream Number:     6
Backup Copy:       Standard (0)
Files File size:     96616190
PFI type:     0
IMAGE_ATTRIBUTE:     0
Primary Copy:      1
Image Type:        0  (Regular)
Job ID:            7565
Num Resumes:       0
Resume Expiration:      01/01/1970 00:00:00 (0)
Data Classification:    (none specified)
Data_Classification_ID: (none specified)
Storage Lifecycle Policy:    (none specified)
Storage Lifecycle Policy Version:    0
STL_Completed:      0
Remote Expiration Time:      01/01/1970 00:00:00 (0)
Origin Master Server:  (none specified)
Origin Master GUID:    (none specified)
Snap Time:           25/12/2015 20:30:22 (1451075422)
IR Enabled:      no
Client Character Set:     0
Image On Hold:     0
Indexing Status:   0
Copy number:       1
 Fragment:         TIR (-1)
 Kilobytes:        94418
 Remainder:        0
 Media Type:       Media Manager (2)
 Density:          hcart (6)
 File Num:         4
 ID:               000241
 Host:             fs02.example.co.uk
 Block Size:       65536
 Offset:           15852144
 Media Date:            25/12/2015 20:00:13 (1451073613)
 Dev Written On:   1
 Flags:            0x0
 Media Descriptor:        ?
 checkpoint:       0
 resume num:       0
 Key tag:          *NULL*
 STL tag:          *NULL*
 Copy on hold:     0
Copy number:       1
 Fragment:         1
 Kilobytes:        958632712
 Remainder:        0
 Media Type:       Media Manager (2)
 Density:          hcart (6)
 File Num:         3
 ID:               000241
 Host:             fs02.example.co.uk
 Block Size:       65536
 Offset:           873505
 Media Date:            25/12/2015 20:00:13 (1451073613)
 Dev Written On:   1
 Flags:            0x0
 Media Descriptor:        ?
 Expiration Time:            INFINITY       (2147483647)
 MPX:              0
 retention_lvl:    infinity (9)
 Try to Keep Time:       01/01/1970 00:00:00 (0)
 Copy Creation Time:       26/12/2015 02:57:12 (1451098632)
 Data Format:      Tar
 checkpoint:       0
 resume num:       0
 Copy on hold:     0

[a1expi]

I've added all the exchange clients and the dag to both master and media additional servers (and rebooted), same error in bpbrm:

13:46:26.961 [2588.2592] <2> ConnectionCache::connectAndCache: Acquiring new connection for host fs01.example.co.uk, query type 293
13:46:26.961 [2588.2592] <2> vnet_pbxConnect: pbxConnectEx Succeeded
13:46:26.961 [2588.2592] <2> logconnections: BPDBM CONNECT FROM 192.168.252.122.49545 TO 192.168.252.121.1556 fd = 612
13:46:27.055 [2588.2592] <2> db_startrequest: protocol error 1
13:46:27.055 [2588.2592] <16> db_begin: db_startrequest() failed: operation requested by an invalid server
13:46:27.055 [2588.2592] <2> db_receiveIMAGEdependees: db_begin() failed: operation requested by an invalid server
13:46:27.055 [2588.2592] <16> restore_files_from_fs_view: cannot find the backup image that the view 10680 depends on
13:46:27.055 [2588.2592] <2> ConnectionCache::connectAndCache: Acquiring new connection for host fs01.example.co.uk, query type 1
13:46:27.055 [2588.2592] <2> vnet_pbxConnect: pbxConnectEx Succeeded

The fact that the error mentions images makes me think it's tied to the current issue which appears to be a disconnect between the vmware images and the MS-Exchange-Server images.

As far as the vmware backups go, I've got them but they're against the individual exchange servers rather than the DAG:

C:\Program Files\Veritas\NetBackup\bin\admincmd>bpimagelist -idonly -d 01/01/1970 00:00:00 -pt VMware -client ex01.example.co.uk
Time:      22/01/2016 22:48:58   ID: ex01.example.co.uk_1453502938   FULL (0)
Time:      15/01/2016 23:24:13   ID: ex01.example.co.uk_1452900253   FULL (0)
Time:      25/12/2015 20:44:17   ID: ex01.example.co.uk_1451076257   FULL (0)

C:\Program Files\Veritas\NetBackup\bin\admincmd>bpimagelist -idonly -d 01/01/1970 00:00:00 -pt VMware -client ex02.example.co.uk
Time:      30/01/2016 09:49:07   ID: ex02.example.co.uk_1454147347   FULL (0)
Time:      22/01/2016 23:19:23   ID: ex02.example.co.uk_1453504763   FULL (0)
Time:      15/01/2016 22:50:10   ID: ex02.example.co.uk_1452898210   FULL (0)
Time:      25/12/2015 20:31:18   ID: ex02.example.co.uk_1451075478   FULL (0)

[a1expi]

1) ESXi version and update and patch level (need the uber detail - sorry)

ESXi hosts are all "VMware ESXi, 5.5.0, 2302651", vSphere is 5.5.0.2646482

2) MS Exchange 2013 roll-up / CU / patch number?

They're all on 2013 CU11 (build 15.0.1156.6) as we need a fix released in CU11, do you think that's the cause?

3) Please confirm that original MS Exchange has not had *any* revisions to MS Exchange Server between the backup time, and now?

The exchange servers have all gone from CU8 to CU11 since the backup was taken.

4) The original backup was taken using NetBackup Client v7.6.1.1 ?

Yes

5) Were all and I mean all MS Exchange servers *and* CAS nodes *all* running the same version of NetBackup Client at the time of the original backup?

Yes, they were also running v7.6.1.1, we've just moved to v7.6.1.2 across all Netbackup serves and exchange servers.

6) And the client you are wanting to restore to is running NetBackup Client v7.6.1.2 ?

Yes

7) The restore target MS Exchange Server is running *exactly* the same version of MS Exchange as the original sourec backup MS Exchange Server DAG and CAS group?

No, the source server was CU8, the target server is on CU11 currently. I know there's official support for CU8 in 7.6.1.1 and CU10 in 7.5.0.7 but we needed a fix in CU11.

I can try downgrading the target server to CU8 over the weekend and seeing if that resolves but it doesn't feel to me like it's related, it doesn't even get as far as talking to the server before it dies.

[sdo]

Ok - here's a thought that I share when I can... do Microsoft support the restore of MS Exchange across different CU updates?  If Microsoft don't then Veritas can't.  NetBackup is entirely in the hands of the MS Exchange APIs when it comes to backup and restore.  My thought is that meta-data and schemas can change within MS Exchange, even with a seemingly innocuous CU update - which might render backup and restore across even minor version revisions not possible.  All conjecture, and it's not helping.

If it were me, and I could, then I would ensure that we're dealing with the same versions on source and target.  But we can't always have we want.  And we have not proven that MS the Exchange APIs are rejecting the restore due to different versions - so it's probably unlikely (for now).

(I / we still need to look through the image lists for something - hopefully soon I will)

We need to look in more detail at the bpbrm log.  What logging level did you have on the bpbrm log?  Could you lift the bpbrm logging to 5, and then see what it says?

[a1expi · Accepted solution]

Chasing the thread about fs01 being an illegal server I've eventually stumbled on the solution.

What I eventually did was add the media server (fs02) to both the master and media server additional server list:

Master server (fs01) additional servers:

fs02

All exchange servers

DAG Name

Media server (fs02) additional servers:

fs02 (Itself, if this isn't in the list the backups fail!)

All exchange servers

DAG name