12-15-2016 10:00 AM
I have a strange issue regarding our MS Exchange backups when it comes to the Granular processing on a certain storage type.
To start off we are running:
AIX 6.1 with Netbackup 7.7.2
NB Media :
3 x AIX 6.1 with Netbackup 7.7.2 (Not used for Exchange backups)
3 x MS Windows 2012 R2 with Netbackup 7.7.2
OST Plugin for all media Servers is 3.1.5
1 x MSDP Pure disk Volume (GRT Backups work when going here)
1 x HP StoreOnce 6500 running software 3.13 and using Catalyst Stores (GRT Backups fail on this storage unit)
To cut a long story short, we want to move the storatge unit our exchange backups go to from the MSDP to the Open Storage platform.
The backup seems to fail on the GRT segment of the backup on the Open Storage, the database backs up fine.
As far as i can see the compatibility lists state that our StoreOnce should be on software version 3.1 and Netbackup on 7.6 or above which we meet.
The detailed Status Log when backup up to MSDB says: (No error just to reiterate)
12/15/2016 12:47:02 - Info bpbrm (pid=7944) DB_BACKUP_STATUS is 0
12/15/2016 12:48:01 - Info bpbrm (pid=7944) from client server-dag-01.domain.com: TRV - Starting granular backup processing for (/Microsoft Exchange Database Availability Groups/server-dag-01\Microsoft Information Store:\server-MBX-DB2\). This may take a while...
12/15/2016 12:48:04 - end writing; write time: 0:36:29
12/15/2016 12:48:36 - Info bpbrm (pid=7944) from client server-dag-01.domain.com: TRV - Successfully finished granular backup processing for (/Microsoft Exchange Database Availability Groups/server-dag-01\Microsoft Information Store:\server-MBX-DB2\).
12/15/2016 12:48:36 - Info bpbrm (pid=7944) from client server-dag-01.domain.com: TRV - Exchange granular database changes were backed up successfully.
12/15/2016 12:48:36 - Info bptm (pid=11284) waited for full buffer 88039 times, delayed 139391 times
12/15/2016 12:48:39 - Info bptm (pid=11284) EXITING with status 0 <----------
12/15/2016 12:48:39 - Info resnetb3 (pid=11284) StorageServer=PureDisk:mediaserver2; Report=PDDO Stats (multi-threaded stream used) for (mediaserver2): scanned: 37852441 KB, CR sent: 88611 KB, CR sent over FC: 0 KB, dedup: 99.8%, cache hits: 2209 (0.7%), rebased: 28 (0.0%)
12/15/2016 12:48:40 - Info bpbrm (pid=7944) validating image for client server-dag-01.domain.com
12/15/2016 12:48:41 - Info bpbkar32 (pid=45212) done. status: 0: the requested operation was successfully completed
the requested operation was successfully completed (0)
But as soon as we switch over to the StoreOnce:
12/15/2016 17:11:01 - Info bptm (pid=12956) start
12/15/2016 17:47:24 - end writing; write time: 0:37:00
12/15/2016 17:47:31 - Info bpbrm (pid=13832) DB_BACKUP_STATUS is 0
12/15/2016 17:47:50 - Info bpbrm (pid=13832) from client server-dag-01.domain.com: TRV - Starting granular backup processing for (/Microsoft Exchange Database Availability Groups/server-dag-01\Microsoft Information Store:\server-MBX-DB2\). This may take a while...
12/15/2016 17:47:54 - Info bpbrm (pid=13832) from client server-dag-01.domain.com: TRV - Granular processing failed!
12/15/2016 17:47:54 - Error bpbrm (pid=13832) from client server-dag-01.domain.com: ERR - Error encountered while attempting to get additional files for Microsoft Information Store:\server-MBX-DB2\Logs_1481821614\
12/15/2016 17:47:54 - Error bpbrm (pid=13832) from client server-dag-01.domain.com: ERR - Exchange granular restore from this image may not work.
12/15/2016 17:47:54 - Info bpbrm (pid=13832) from client server-dag-01.domain.com: TRV - Exchange granular database changes were backed up successfully.
12/15/2016 17:47:56 - Info bptm (pid=4620) waited for full buffer 70906 times, delayed 126450 times
12/15/2016 17:48:00 - Info bptm (pid=4620) EXITING with status 0 <----------
12/15/2016 17:48:00 - Info mediaserver2 (pid=4620) StorageServer=hp-StoreOnceCatalyst:STORAGEBKUPTIER3SS1; Report=scanned: 38748558336 Bytes, CR sent: 337319494 Bytes, dedup: 99.12%
12/15/2016 17:48:00 - Info bpbrm (pid=13832) validating image for client server-dag-01.domain.com
12/15/2016 17:48:02 - Info bpbkar32 (pid=51732) done. status: 1: the requested operation was partially successful
The requested operation was partially successful (1)
The only difference between these backups is the storage unit. it is the same policy with the same settings. We are just dropping the menu down and selecting a different Storage unit. To help with our investigation we have locked this test down to just 1 of our 6 media servers.
Any help on this issue is greatly appreciated.
12-15-2016 10:06 AM
As it won't let me edit my post, we are running on Exchange 2010 and have a 6 node DAG configuration.
12-16-2016 08:11 AM
Do you have a verbose bpbkar log?
12-16-2016 09:01 AM
Thanks for your responce.
I have set the verbose level on the client to 5. I will report back with the log files after the backup has completed tomorrow.
12-16-2016 11:35 AM
I have attached the bpbkar logs.
I can confirm that the temp files in C:\Program Files\Veritas\NetBackup\Temp are being created. but for some reason it dosnt like to do anything with them!
12-16-2016 12:31 PM
This is a level 0 log, not very informative:
ov_log::OVInit: GENERAL Log Level (Effective): 0
ov_log::OVInit: TCP Log Level (Effective): 0
OVLog: BPBKAR NetBackup Backup/Archive 7.7GA [Jan 11 2016]
OVLog: Copyright © 1993 - 2016 Symantec Corporation, All Right
WinMain: BPBKAR_VERBOSE Debug log level: Actual=0, Effective=0
Here's the failure. Need max level log to know why:
_nbfs_close_link_file(): ERR - Attempt to link files failed, error = 1006
In light of this error, please also provide the nbfsd logs from both the client and the media server (especially the media server, and make sure it's at max level).
If you are changing media servers, do you have NFS enabled on the new server?
01-04-2017 05:33 AM
I hope you have had a good new year, unfortunately it’s time for me to get back into this issue!
To answer your questions:
All media servers have been backing up the exchange environment for 3+ years, we are only changing the storage unit in which it goes to.
We have switched form a MSDP that was attached to a media server to an Open Storage platform (HP StoreOnce 6500).
If I change the backup to run to the MSDP it completes successfully, but as the MSDP storage is now going out of support we really need to move the backups over to the StoreOnce.
I will collect he log files for you and post them as soon as I have them.
01-04-2017 06:18 AM
Given that it works to MSDP but not to your OST plug-in, I need to see a bpbkar log at max logging levels (set both levels), and both nbfsd and bptm logs from the media server, also at maximum level.
It's likely that all we'll learn from the bptm log is more evidence that points to the HP plug-in. That would need you to open coordinated support cases with Veritas and HP. Let's start with the logs and see what we find.
Please clarify. In the discussion you have indicated that the GRT phase of backup has been failing. Your thread title indicates GRT recovery. Are we focusing on the backup? (Without GRT backup, there is no GRT recovery.)
01-04-2017 06:38 AM
Yes its just the GRT phaze, so the backup itself will work successfully (error code 1) to the StoreOnce but without the ability to restore individual emails.
And yes sorry for the confusion, it is at the backup stage GRT is failing.