Backing up the open partition with EV10

[James_Slack]

Hi,

Our backup product doesn't use the archive bit, so we are using the "IgnoreArchiveBitTrigger.txt" file method.

This works fine for our closed partitions but this doesnt seem to work for the open partition.

When we run a backup it takes the ,txt and changes it to a .old file, so that side of it is fine. The partition is set to look for the trigger file and check every 60 minutes.

However, we have 500k and growing items "awaiting backup".

Having had a look what is awaiting backup, it is only items from our open partition.

Even if I do a backup, does it not work until the partition is closed or am I missing something.

Thanks - James

[AndrewB]

how are you creating the "IgnoreArchiveBitTrigger.txt" file? do you have a post-backup script doing it? just to clarify, the backup doesnt change the txt to .old, it's EV doing that.

if everything seems fine to you, you can get a better idea of what's going on if you run a DTRACE, create a new IgnoreArchiveBitTrigger.txt file in the root of your open partition and restart the storage service to trigger the check.

[LCT]

1. What backup software are you using? does not really matter but just out of interest

2. What service pack are you running for EV10?

3. Anything that changed recently that caused this to stop working?

4. Would you be able to post your EV Post backup script so we can check?

As Andy said, the it's not the backup that changes the file it's the EV Storage Service that does it. Storage Crawler and Storage File Watch processes that does the trolling for the DVS files etc. within the Vault Store partitions.

Do a dtrace of the preocess for a closed partition and a dtrace for an open partition using the method that Andy suggested and hopefully the comparisions of the dtrace will show why it is not working for you.

Also there is/was a problem with EV9.0.3

http://www.symantec.com/business/support/index?page=content&id=TECH180841

This technote says that it was addressed in EV9 SP4, it may not have been addressed in EV10 etc. you never know...but don't apply the workaround until you have studied your dtraces first.

The reson why I say or suggest this technote is because, when I create a new vault store partition, backup stopped working for with the problem addressed in the technote. I am using 9.03 at the moment but that is not to say that it won't be a problem in EV10 and SP1.

[v-2gahod]

Just make sure that the modified date for ignorearchivebit.txt.old is the recent one, try creating a new ignorearchivebittrigger.txt manually ad also change the scan time to monitor the progress something like 3 mins and then run the below mentioned query against your vaulstore database to check the progress:

select * from journalarchive where backupcomplete ='1'

select * from watchfile where itemsecured ='1'

If watch file table starts flushing  then might be an issue with the backup scripts.

http://www.symantec.com/business/support/index?page=content&id=TECH35610

[James_Slack]

v-2gahod :

The problem is not with the text files... they are created and changed to .old as expected. However, the list of files not backed up does not shrink.

Also, worth noting is the behaviour I have observied in this versino of EV. If you set the monitor process to say 5 minutes, when you come out of backup mode it will do 5 minutes worth of clearing the files awaiting archive then stop. Having it at the default of 60 minutes means you get 60 minutes worth of processing before it stops.

I know this is exactly how ours works as at one point we had 9m+ items and so I saw this happen many times. I was also suggested to set stores to small times, but they only processed a very small number each time, but having it at an hour does an hours worth. Maybe a bug, maybe design... but it does it for sure.

[James_Slack]

LCT,

1. What backup software are you using? does not really matter but just out of interest

You're right it doesn't matter as I have tried running the scripts (set backuo mode, delete .old, create .txt unset backup mode) manually and get the same issue. Closed partitions show that they have been backed up, open one doesn't. But in answer, Commvault.

2. What service pack are you running for EV10?

10.0.1.1171

3. Anything that changed recently that caused this to stop working?

This is a failrly new build, but the new partition is the same as our other partitions have been. Not sure if we had this issue with the previous open partitions.

4. Would you be able to post your EV Post backup script so we can check?

Can't copy and paste from server... but post backup does this.

Deletes the ignore....old files

Generates new ignore....txt files

clears backup mode from site.

(as I mentioned, the open partitions .txt gets converted to a .old file OK)

[James_Slack]

Andrew.

Do I need to stop and start storage as partr of my backup script? Or is that just a way to frig it during a manual test?

Thanks everyone for your replies.

[James_Slack]

Hi LCT.

Firstly thanks for the info on daily restarts. Something I don't have scheduled yet.

Next, looking at the last item secured / last scan I can see that is secured something yesterday at 14:56. However, I tried this again 2 hours later and so it didn't secure any items that time.

Also, I can see the last scan was run half hour ago.

I'm going to check the date/time stamp of some of the files that have not backed up, in case they are off in the future or something daft.

[LCT · Accepted solution]

James,

It does look like your script is OK. Don't need to restart storage service in the backup script from EV9 onwards. However, best practice you should restart your EV services on a daily basis to help it flush old cached stuff and clearing out 'messed up' connections to exchange and sql servers etc. Putting the Vault Store partition in and out of backup will also trigger the partition scan. i.e. changing the IgnoreArchiveBitTrigger.txt to .old.

Just out of interest, on the properties of your open partition - on the Backup tab - in the Status area, what does say in there. Has backup been detected since the partition was opened? If it has when was the last scan happened. This was the sympton I found when I discovered my issue.

*Edit - from EV8 Onwards

[James_Slack]

Well, scratch that idea. The file date and time is correct and over a week old now.

Guess I will try a stop / start of the services and then trigger the backup scripts again.

[LCT]

OK...so this part of the process seems to be normal as it does detect backup and scans items for your open partition. We can rule out the technote that I posted.

It does look like you will need to run the dtrace for Storageserver.exe, storagecrawler.exe, w3wp.exe, storagefilewatch.exe to see what is happening.

Set the processes in the dtrace, put a new IgnoreArchiveBitTrigger.txt in the root of the open partition, restart the storage service/or put in backup mode and take out again if you can't restart service during working hours, let it run for an hour or so. dtrace will be large so keep an eye on the size.

Possibly something funky with SQL that is not clearing down the watchfile table. You may need to enable SQL logging for this too. At which point you probably need to log a case with Tech Support.

[James_Slack]

It is processing now... gone from 880k down to 850k so far. Will monitor it and see how we go.

[James_Slack]

Sorted. <100k now and still dropping.

Just needed the service restart.

That didn't even occure to me a some processing had occured.

Thanks all. I will get it scheduled in and that should be everything.

Cheers - James

[LCT]

glad it's all working now James.

[James_Slack]

What is the best way to do a restart of Vault services now?

In the old days you used to have to stop and start the services in a sensible order. I have heard that you can just restart the Admin service now and it takes care of everything else in an order. Is that the case or or is it still best to do it one by one manually?

[GertjanA]

If you use a manual restart, just restart the Admin service.

If you want it scheduled, stop :

1 - Taskcontroller

2 - Shopping

3 - Storage

4 - Indexing

5 - Directory

 6 - Admin

Start:

1 - Admin

2 - Directory

3 - Indexing

4 - Storage

5 - Shopping

6 Taskcontroller

[LCT]

I would still use the old fashion way. Net Stop the EV services from bottom up i.e. start with the Enterprise Vault Task Controller ....Enterprise Vault Admin Service. Then of course Net Start it the reverse order...with EV Admin to Task Controller. This way if something goes wrong you can trace what is happening. Technically, once you have started the EV Admin service you can start the other services in any order you like.

You may need to give the EV Storage Service a little more time by adding SLEEP/TimeOut for a few minutes as this is the service that commonly cause problems when it comes to stopping and starting...This is because it has to terminate all the related processes

StorageArchive.exe
StorageCrawler.exe
StorageDelete.exe
StorageFileWatch.exe
StorageManagement.exe
StorageOnlineOpns.exe
StorageRestore.exe
StorageServer.exe

Every EV environment is different so do what is best suits your environment and what you are comfortable with ;)

I tend to do mine manually as part of my daily routine check, then I know what to do if a service failed to stop or start.

[James_Slack]

Thanks both... will get it scheduled in ASAP.

[AndrewB]

james, so what actually fixed the problem? restarting the storage service?