Backupmode not cleared in post_backup script after upgrade of EV from 8 to 9.0.1

[MatsHolm]

Have a very strange issue on one customer. We upgraded EV environment from 8 SP1 to 9 SP1 last week on two EV servers.
Both EV Servers on Windows 2008 R2.
Customer is using Netbackup to backup EV server and places them in backupmode with following string in a pre_backup script:

C:\Windows\SysWOW64\WindowsPowerShell\v1.0\PowerShell -psconsolefile "C:\Program Files (x86)\Enterprise Vault\EVShell.psc1" -command "& {set-vaultstorebackupmode 'EVSITExx' EVSERVERxx site; set-indexlocationbackupmode EVSERVERxx -EVSiteName EVSITExx}"

 Then after backup of index, storage and sql databases we then want to clear the backupmode and uses the following string in a post_backup script:

"C:\Windows\SysWOW64\WindowsPowerShell\v1.0\PowerShell" -psconsolefile "C:\Program Files (x86)\Enterprise Vault\EVShell.psc1" -command "& {clear-vaultstorebackupmode 'EVSITExx' EVSERVERxx site; clear-indexlocationbackupmode EVSERVERxx -EVSiteName EVSITExx}"

The two scripts worked fine before upgrade but now the pre_script sets EV in backupmode but the post_script can't clear the backupmode.
If we clear manually in VAC it is OK
if we run the post_script it manually by doubleclick it it clears the backupmode.

We have tried puttiing the command to clear backupmode on two different rows one for storage and one for index but it doesn't work.
We ran the the Transform-backup.ps1 script to generate powershell command and verified that everything was OK.
We also tried using a timeout in the script to make sure that powershell ran fine but we can't seem to find an answer to it.

Has anyone seen this before?

/Mats

[Mark_Shoger]

Of Netbackup are you using? EV9 requires that you be running NBU 7.0.1 which has a new enhanced agent for backing up EV9.

Cheers,

M

[FreKac2]

Since they are using the scripts there isn't really any requirements for the NBU version to be 7.0.1 as far as I know, if using the EV Agent then yes.

I also answered Mats's question on the TC forum, not sure if it's the same scenario that I've seen though.

But what I think happened in EV9 (maybe someone could verify this) since I noticed this on one of my customers.

Is that if setting the whole site to backupmode in both servers pre-script, the posts scripts on both servers has to clear the whole site from backupmode. Otherwise it will still be in backupmode.

In EV8 if you had e.g. two servers, if one of those servers finished and you cleared the whole site, then the whole site was turned to normal mode, wheter or not the second server was done or not. Which you could solve with e.g. checking in the script if the second server is done or not. If it's not done you just didn't run the clear command on that server. Once the second server is done it will also check if the first is done, if yes, then run the clear command.

But since as far as I know EV9 changed the behaviour and therefor you could try to remove the check, so wether or not the second server is done or not, run the clear command. It will still be in backupmode until the second server is done and also run the clear command.

This is what I saw at one of my customers when setting the scripts up in EV9. Took a while to figure out that the behaviour changed, unless it's a bug :)

/Fredrik

[GertjanA]

Interesting discussion

If using a triggerfile (but I assume you don't) be advised that that does not work when using powershell to set sitelevel backup mode.

When reading this thread, what I do not understand is that it (in your setup!) is possible to clear backupmode without being sure backup finished on all vault-servers. As you need to run the pre- and postscript on a server, would it not be better to put that specific server in backupmode?

Or, if there is a need to set the site in backup, schedule a cmd/bat file using task-scheduler, and don't rely on the netbackup to do it. Taking in account (ofcourse) the time it takes to backup the site, you cuold schedule it well in advance, and after expected finish time.

But, I am curious as to how this is going to be resolved (if resolved)

[FreKac2]

This is OT I guess but why wouldn't a trigger file work on a site level backupmode, if you in the post script create the trigger file before going out of backupmode ?

Back on topic.

The way I've reasoned about this in the past, is if you have multiple EV servers and use "share within group", can you be sure that the DB's are consistent if you have av DR-scenario. Since you are sharing the content of the Fingerprint DB between multiple servers and VS's. Or let me put it this way, if you put EV1 to backupmode will it prevent EV2 to add more content to the Fingerprint DB ?

If not then it's the same "issue" when going out of backupmode since that will let one server to add more content to the Fingerprint DB while the other is still running a backup of its VS. Maybe my resoning is wrong so correct me if that is the case.

In regard to your question if it's possible to clear backupmode without the backup being finished. That is what you have to look out for. So no you shouldn't clear backupmode before the backup is finished. But if a script clears the backupmode, EV is not going to care if the backup software is still running a backup, right ? Except that e.g. archivebit is not reset, may be issues with consistancy and so forth.

The thing I saw with EV9 though, at least I tried to explain above is that the behaviour changed from EV8 to EV9 in regard to how the setbackupmode and clearbackupmode is seen by EV. So if you do e.g. a site setbackupmode on multiple servers, all the servers need to run a site clearbackupmode before it will be brought into normal mode. That was not the case in EV8, then you had to check if all servers were done before running the site clear command.

The problem with using Task Scheduler is that you can't be sure when the backups will finish, so if you run backup/normal-mode scripts you have to "pad" the time quite a lot to make sure that the backups will be finished by that time, hence it's better to do that through a post-script.

The only way to get that interaction without the scripts is to use an EV Agent, which is limited to BE and NBU and that you've purchased that Agent. With NBU and the current agent I've pointed out that there is a "potential" consistancy issue with the Directory DB but that is a different issue and there is Etrack for it.

[tsingh]

Hi,

Just found this post, We are experiencing the same problem on one of our EV server.

Our setup is like this:

OS Win2k8 R2

EV: 9.01.

1 Site

1 EV Journal Server

2 EV Mailbox server

2 DB servers (the fingerprinting databases are separate based on location).

On one of the servers the pre and post scripts work with no issues.  On the server server pre script works but post does not clear the backup mode.  Further, we are not setting the entire site to backup mode, but only the specific VaultStores...

I called support and they are not able to figure it out.  They are blaming the issue on Microsoft and want me to re-install Powershell, which I think is not relavent since scirpts work manually...

Thanks for your help in advance.

[FreKac2]

You probably need to post the scripts.

What backup software do you use and what happens during the post script run ?

E.g. have you tried some echo entries in the post script just to see that the BU-software actually run the post script etc. ?