Exchange GRT restore issue - NetBackup 7.5.0.6
Hi Folks,
Hoping someone can help shed some light on this one:
Master Server: NetBackup 7.5.0.6 (Windows 2008 R2 SP1 64bit)
Media Server: NetBackup 5230 Appliance Version 2.5.3
Exchange: Exchange 2007 SP3 (Windows 2008 R2 SP1 64bit)
I have a VMware policy configured using application protection for Exchange. This completes successfully, but when trying to do a GRT restore I can expand the Information Store and Storage group with no problems. Attempting to expand the Database however eventually times out and gives the dreaded "ERROR: database system error".
I have been over & over the requirements for Exchange GRT and am sure I have everything correct. To prove it, I created a standard MS-Exchange policy with the "Perform snapshot backups" and "Enable granular recovery" options turned on. This backs up fine (although via LAN - I want be going via SAN and VMware), and I am able to expand all items and go down to individual messages. This seems to indicate I have everything set correctly for GRT restores.
Does anyone have any ideas as to why this might be? I've been banging my head on this one for a little while now. I'd much rather use the VMware policy type with Exchange protection so I get the SAN performance, but also want granular recovery.
Many thanks in advance,
Steve
Hi Folks,
Just thought I'd put in an update about what my particular issue was in this case - very interesting. Was working it through with support when we found it. Moderators - you can lock this thread afterwards if you like :) Thx, Steve
We found the answer to this one by accident. One of my network engineers came to me about a week ago and said he had some suspicious traffic between two devices (the appliance and Exchange server for this case). He asked if I would check it out for him.
The data for this NetBackup environment traverses a Fortigate firewall/security appliance with IPS/IDS. I had already checked all possibilities while setting up and trying to troubleshoot the GRT browse issue - so I was aware this firewall was in place, and we had already confirmed (multiple times) that all required ports were open. What I wasn't aware of was that the IDS/IPS was scanning all traffic for vulnerabilities, and where detected engaging an active response (ie: drop traffic). The Fortigate was not set up to send automated alerts when something was detected, so it was only when our network engineers logged in to the device for a check that this was discovered.
The vulnerability as detected by the Fortigate device was: http://www.fortiguard.com/encyclopedia/vulnerability/#id=34667
The Microsoft write-up of this issue from the Windows side is: http://technet.microsoft.com/en-us/security/bulletin/ms13-014
Once we were aware of this, a rule was put in place to ignore traffic between the appliance and Exchange server - GRT browsing then started working. I had some additional issues in getting the restore to actually complete, but that was a minor issue in which the client was attempting to contact the appliance using a short DNS name. A minor tweak and having it use the FQDN resolved that issue and it now works end to end :) We had a similar issue with our production site (although this was throwing a File I/O error rather than Database error) - the same ignore rule was applied at the production site and GRT browsing worked there as well.
I reviewed the Microsoft KB article, and although I could not see the specific patch installed in either Exchange server, judging by the release date I would suspect that this fix has already been applied in a later security rollup update. I reviewed CVE vulnerabilities for NetBackup software and appliances and could find nothing relating to this. Also of interest is that during the initial part of this issue I did have Symantec Endpoint Security installed on the Exchange server, and this had reported nothing.
The previous engineer on this case before yourself pointed out in the NCFLBC logs that we were seeing issues with renaming files. I believe this is where the DOS attack that the Fortigate detected was coming from, as if you read the detail of the Microsoft KB it talks about attempting to rename a file on a read-only file system causing the issue described.
A couple of other points of interest: During an Active Directory GRT browse, and also during a traditional Exchange Snapshot (non-VMware) GRT browse, this issue was not evident. It was only via a VMware Exchange GRT browse that an active response was triggered on the Fortigate appliance. I'm not sure if you want to pass this info on to any of your other teams, as I guess it is quite possible any IDS/IPS system scanning and responding to this vulnerability could create the same scenario.
