Netbackup - Enterprise Vault Agent Implementation

[SYMAJ]

I am implementing the EV agent with EV803 under Netbackup 7504.

What is the best practice for creating the policies in order to backup the EV environment ?  I see that not all directives can be mixed in the same policies and have therefore come up with the following:

1. Directory DB

2. Monitoring DB, Audit DB, Vault Store Databases, Fingerprint DB, Open Partitions, Indexes

3. Closed Partitions

I intend to run 1 and 2 daily on a Differential Incremental basis, and all 3 weekly on a Full basis.

Is this the correct way to do this ?  If the policies overlap will the EV environment be in the correct state for each of the backups (Backup mode) ?

Any advice on best practice appreciated.

AJ

[RLeon · Accepted solution]

Vault Store Databases, Fingerprint DB, Open Partitions, Indexes

This most important thing about EV backup is backing up these 4 together for each Vault Store, and you got that already. The OS system backup could also be scheduled together with the EV application level backup for each Vault Store, if you also need it protected.

The Monitoring DB and Audit DB could tag along in the same policy. If you have a multiple-building-blocks EV environment, then it is best to put these two plus the Index in a policy, and each of the Vault Stores in their own.

There are other DBs too but I guess you are not using them in your EV.

The rest looks fine. Apart from not letting the DirDB and Index+OpenPartition schedules overlap, the other policy schedules could overlap. You could put the Ready Partitions in the 3rd policy too, but note that it is also Vault Store based, so 1 such policy for each Vault Store.

It is better to run EV backups when it is least active so you will least likely to have consistency issuses. (Apparently it could still happen, regardless of the use of Backup mode)

[SYMAJ]

Thanks for that.  One point to clarify - are you saying that there should be a seperate policy for each vault store ?  I have 3 vault stores and currently have them all in the same policy.

AJ. 

[RLeon]

Yes.

All these directives

EV_OPEN_PARTITION
EV_CLOSED_PARTITIONS
EV_READY_PARTITIONS

Do not support more than one client on the policy client list. From the guide:

This directive does not support multiple policy clients in the client list.

This makes logical sense too. From a performance and operational perspective, having all Vault Stores in Backup Mode all at the same time is a bad idea.
This restriction encourages you to set different backup schedules for the Vault Stores, so that they are not backed up at the same time; even though NetBackup and EV should not have problems letting you do so.

[Mark_Solutions]

Take a good look at the NetBackup_AdminGuide_EntVault.pdf in your installation source.

There is a good section from about Page 100 onwards that describes all best practices to do this

Part of that says:

To ensure consistency when backing up a fingerprint database, you should add
all open partitions of the vault store group and the fingerprint database in the
same policy.

Other useful bits (but do take a read as there is a lot of good advice in the guide)

You should back up the closed partitions and the ready partitions of a Vault
Store in separate policies.

If the amount of data is small enough, you can combine the closed and the
ready partitions of multiple Vault Store’s in a single policy

You should protect index locations in an Enterprise Vault site with a separate
policy to maintain consistency.

Hope this helps

[SYMAJ]

Many thanks for the input:

To ensure consistency when backing up a fingerprint database, you should add
all open partitions of the vault store group and the fingerprint database in the
same policy.

I am currently doing this

Other useful bits (but do take a read as there is a lot of good advice in the guide)

You should back up the closed partitions and the ready partitions of a Vault
Store in separate policies.

Not backing up ready partitions (don't have any) but the closed partitions are in a seperate policy

If the amount of data is small enough, you can combine the closed and the
ready partitions of multiple Vault Store’s in a single policy

Not applicable

You should protect index locations in an Enterprise Vault site with a separate
policy to maintain consistency.

I have the index locations in the 'main' policy along with the databases and open partitions.

As I have 3 vault stores, then what I am reading from the above (and I will check into the manual as you pointed out) is that I should have a set of my three policies for each of the vaul stores ?  Seems overkill to me.  Also, I should split out the index location backups into a seperate policy.

As there is only one Fingerprint DB, one Audit DB, one Monitoring DB then these will only appear once.  The above is saying that the Fingerprint DB should be in the same policy as the Open Partitions (which I have three entries for - one for each vault store).  This is now confusing......

To summarise, I have the following policies:

1. DIR_DB

2. Monitoring DB, Audit DB, Fingerprint DB, Vault Store DB 1, Vault Store DB 2, Vault Store DB 3, Index Locations, Open Partitions Vault Store 1, Open Partitions Vault Store 2, Open Partitions Vault Store 3

3. Closed Partitions Vault Store 1, Closed Partitions Vault Store 2, Closed Partitions Vault Store 3

1 & 2 run daily differential, 1 & 2 & 3 run weekly Full.  1 starts at 1900, 2 starts at 2000, 3 starts at 2200.

Should I split the Vault Store elements out into additional policies, if so how do I keep the consistency with the Fingerprint DB ?? 

Confused......

AJ

[Mark_Solutions]

Been digging as all this rang bells and knew there was a doc somewhere - take a look at these:

http://www.symantec.com/docs/TECH126604

http://www.symantec.com/docs/TECH150237

First is EV8, second is EV9

Hope these help

[SYMAJ]

Excellent - many thanks.  The only thing to consider now is whether the 3 vault stores need to be handled seperately.  I don't see how I can do this and remain within the guidelines of the technote.....

AJ.

[Mark_Solutions]

I guess it is all down the things being looked at being within a group - never easy!

How much data do you have within EV?

If it is very large then you may well end up looking at using FlashBackup anyway as the EV Agent does not tend to be very fast! (just to complicate things!)

[RLeon]

http://www.symantec.com/business/support/index?page=content&id=TECH176623

Here is one for EV10...

None of the 3 Technotes seem to encourage the backup of individual Vault Store DBs (EV_VAULT_STORE_DB).

This is strange, as both the EV's own documents and the Nbu EV guide seem to agree that restoring them is one of the most important steps in EV recovery; and it should be done when you restore partitions.

Secondly, the fact that the Technotes put the EV_FINGERPRINT_DB together with a EV_OPEN_PARTITION suggests that this backup configuration is for a Vault Store Group that is associated with Vault Stores from a single server.
A Fingerprint DB can be associated with Vault Stores located on multiple servers through a Vault Store Group, and as we know, each Vault Store has its own Open Partition.
This would logically imply that the EV_FINGERPRINT_DB should not be bounded to a single EV_OPEN_PARTITION in a policy when a Vault Store Group has Vault Stores from different servers.

[RLeon]

Thanks for that.  One point to clarify - are you saying that there should be a seperate policy for each vault store ?  I have 3 vault stores and currently have them all in the same policy.

After rereading what I said, I kept associating a single client (server) to a single vault store, sorry for the confusion.
If all your Vault Stores belong to the same client, then yes you can put them in the same policy.
The one client per policy restriction still applies.

[SYMAJ]

OK - that sounds more sensical to my environment as everything is one the one server (client) - excepting the SQL elements.

One more question.....

If I run the policies manually - in a FULL backup mode - all works OK.  However, when I let it run a DIFF schedule (even after a full backup has completed successfully, albeit manually), I get a load of error 13's and the jobs fail.  I am puting this down to the fact that a SQL maintenance plan is still running (although I was told this had been stopped !).  Would this explain the error 13's ?  I haven't run a DIFF schedule manually so can't say if that would work or not, but I doubt it - I think it is related to the maintenance plan. 

Does this sound feasible ?

AJ.

[Will_Restore]

Yes, if something else backs up the DBs then EV Differential will fail with Status 13.   Don't ask me how I know this & how often I have to ask our DBAs to abide.

[SYMAJ]

Thanks for that - hoped this was the case.

I have now moved on and have just been advised we have status 2 errors from the FULL scheduled backup tonight !!  From what I can see status 2 is caused by the EV agent trying to put things into backup mode when it is already in backup mode (or hasn't authority to put it in backup mode).  I know authority isn't the issue as I can run each of the jobs individually with no issue.

This brings me back to the issue of scheduling various jobs for the EV backup.  As per above, I have the following:

1. DIR_DB

2. Monitoring DB, Audit DB, Fingerprint DB, Vault Store DB 1, Vault Store DB 2, Vault Store DB 3, Index Locations, Open Partitions Vault Store 1, Open Partitions Vault Store 2, Open Partitions Vault Store 3

3. Closed Partitions Vault Store 1, Closed Partitions Vault Store 2, Closed Partitions Vault Store 3

1 & 2 run daily differential, 1 & 2 & 3 run weekly Full. 1 starts at 1900, 2 starts at 2000, 3 starts at 2200.

This will mean that jobs can overlap, therefore the system could be trying to put EV into backup mode while it is already in backup mode from the previous job.

How do we best get around this, or is it simply a case of making sure the jobs don't overlap ?

AJ

[RLeon]

Directly from the Technotes:

Adjust the Start Times according to individual EV Site configurations and size of entities.
Configure the Start Windows to take into account the duration of the backups so that none
of the backups overlap each other.

Trial and error with the start times I guess.

Or you could also try to force some kind of relationships between policies so one completes before the next could start; you will need to look at playing around with the scripts. I.e., When jobA completes, a script triggers jobB to start, and so on. There are many threads about trying to do this. Here is one from a quick search:

https://www-secure.symantec.com/connect/forums/bpbackup-bpstartnotify-scripts#comment-5430421

[Marianne]

AJ, I believe your original query was answered:

What is the best practice for creating the policies

Please mark solution and start a new thread to troubleshoot failed backups.

[RLeon]

The following whitepaper confirms that the EV_OPEN_PARTITION directive automatically includes EV_VAULT_STORE_DB, making the latter redundant.
(This fact is in the Nbu EV guide, but under a section for EV7.5... Try to figure that one out.)

This validates the recommendations from the Technotes earlier.

http://www.symantec.com/business/support/index?page=content&id=TECH147148