EV\NBU conflict hanging file server during restore

[Vikes_2]

Hello,

We have been beating our heads against the wall trying to figure this one out and cannot. Currently we are working with backline support and still not finding what is causing this so I thought I would toss out the info and maybe someone else has seen this, but not reported it to support?

Issue:

When restoring file system data from NBU to the file server the file server hangs and needs to be rebooted, obviously an issue when there are several thousand users attached to the file server. We have noticed that after the server comes back there are evps.tmp files in the restore directory, so something was recalling the items as they were being restored, we just have not figured out what yet and the event logs show no errors associated to this.

Steps:

Of course we thought SEP so we stopped the SEP services and that did not help, and there is a weird PID that shows up in the dtrace whose id is in the millions. We also think that it might be associated to placeholders that are for files smaller than 4k, we do have a ton of them but they seem to function without issue otherwise. Also, via a full memory dump we know that the evmf.sys driver is actually what is causing the OS to hang. We have also ran the EVUpdatePlaceholderFormat.exe and that did not help.

Background: 

We are running EV 8.0SP5 and have even upgraded the FSA Agent to 9.0sp2 just to rule that out. We are seeing this on both W2K3 and W2K8 servers and we are running NBU 7.1 media servers and agents. SEP is currently at 11.0.6100.  It seems that SEP also binds to the evmf.sys driver, even with the services stopped so our next plan is to uninstall SEP and try a restore to see if that helps.

Summary:

Something seems to be recalling files under 4k while they are being restored from NBU, causing the evmf.sys driver to hang the file server. The restore works fine if the EV services are stopped, which we use as a work around but we are hoping for a fix to this issue. Any ideas??

Thanks,

Travis

[Vikes_2]

I know this sounds alot like this:

http://www.symantec.com/business/support/index?page=content&id=TECH62001

But that was supposed to be fixed in 2007, we are running 8.0 SP5.

Travis

[JesusWept3]

In that technote it says they are thinking about updating the update placeholder utility, has that actually been ran? and to be fair if you also read the technote, you have this critical line here:

"There are no plans to address this issue by way of a patch or hotfix in the current or previous versions of the software at the present time. However, the issue is currently scheduled to be addressed in the next major revision of the product"

Meaning that it was planned to be fixed in EV8 with updates to EVUpdatePlaceholderFormat.exe, which if those fixes were made, you would still see the issue until you run EVUpdatePlaceholderFormat.exe

What i would suggest is getting backline to cross ref both the etracks stated, find how those etracks were closed, whether it was fixed, or new etracks were spawned or what not.

Also have them look at this:
https://www-secure.symantec.com/connect/forums/recovery-ev-fsa-shortcuts-nbu

That had some attention from Darren Locke who is the product manager for the File Server Archiving side of EV, and suggested that they could not reproduce the issue in house at symantec.

[Vikes_2]

Hey JW2,

First off, thanks for the reply! We did try the EVUpdatePlaceholderFormat.exe and still had the issue. I looked at your link and initially we had issues unloading the driver as it appeared that SEP was bound to it also. The easiest work around that we found is simply stopping the FSA services on the file server for  the restore from NBU then starting them back up after.  We know the hang is happening via the evmf.sys driver but it also appears that something is trying to recall placeholders why we are restoring them, so that is why we are going to test today looking at SEP. I will repost when if I get any more info in testing today.

Thanks!

Travis

[Vikes_2]

Also want to add, in one of our tests we would hang the server on every restore and it had a single evps.tmp file in the restore location every time. I then recalled that placeholder, took a new backup and restored without issue, I  don’t think it has anything to so with the number of files. It seems almost random, but appears to be tied to PHs under 4k. Going forward we will not leave PHs for files under 4k but we have tons of them now.

Regards,

Travis

[JesusWept3]

I take it procmon hasn't yielded any clues? Also who has the backline task? NBU support or EV?

[Vikes_2]

NBU had it a while back then for the last couple weeks EV. Nothing is giving any clues, they cannot reproduce the issue in the backline lab so who knows maybe something specific with our env? BTW uninstalling SEP did not help, so at least that removes that from the list :)

I will update the thread if I get any info.

Thanks,

Travis

[Vikes_2 · Accepted solution]

Ok it looks like we figured it out. It was related to the EVUpdatePlaceholderFormat.exe, fix. We tried the fix and it did not work, but we found a newer version with the /L option that deletes the suspect PHs and recreates them, that worked. So, backline support updated the tech article with the new version today so anyone grabbing it from the web would get the newer version.

Thanks for the input on this!

Travis

http://www.symantec.com/business/support/index?page=content&id=TECH52742