Open file backups, VSS snapshots and bpstart_notify scripts

[RLeon]

Hi all,

This query is mainly for Windows, but may possibly apply to *nix platforms.

If a drive (say drive E:\), is to be VSS snapshotted and backed up, is this snapshot created BEFORE OR AFTER the bpstart_notify.bat script is run?
This is important because I may want to use bpstart_notify.bat to stop all I/Os to some opened files on E:\ before the snapshot should be taken.

Here is an example (which is really just my presumptions):
A custom program not supported by Netbackup, and has no VSS writer, is deployed on a Windows host.
It reads/writes some files on E:\ and never releases the file locks unless the program itself is shut down.

A script is created that does 2 things:
1.
Gracefully shuts down the program.
This is so that the file locks on E:\ are gone, and that the drive can be fully quiesced and snapshotted.
2.
After the shutdown (step 1. above), count 15 seconds and start the program again.
This means VSS is given a generous 15 seconds to take a snapshot of E:\
NOTE: This script has to be a non-blocking script. (IMPORTANT)

A line is placed in bpstart_notify.bat to call the script.

When the backup job starts, the following happens in strict order:
1. bpstart_notify.bat calls the script.
2. The script shuts down the program, which releases the file locks on E:\
2a. Since the script is a non-blocking script, Netbackup doesn't wait for it to finish.
3. The script at this point is still running a 15 seconds countdown, but AT THE SAME TIME, VSS takes a snapshot of the entire E:\, which should be successful seeing that no files are opened/locked.
4. The script completes its 15 seconds countdown and restarts the program.
5. Netbackup can take however long it wants to backup the snapshot. 1 hour? 2 hours? no problem! What's important is that the program is running and the backup doesn't interfere with it.

The above is what I presume how bpstart_notify.bat works in this situation.
So is this correct?
What I worry most is: What if the Windows VSS snapshot is taken BEFORE bpstart_notify.bat is run?
That would mean the snapshot would fail because there are still open/locked files on E:\.
If that happens, then the script that shuts down the program would be pointless because a failed snapshot attempt would already have happened before the file locks are released.

Thanks!

RLeon

[Mark_Solutions]

If you select "Perform Snapshot Backups" in the Policy Attributes tab then it run the snapshot before it runs the bpstart_notify script

The snapshot gets run immediately by the parent job and once complete it kicks off the child job which runs the bpstart_notify script

If you do not select "Perform Snapshot Backups" it will still use VSS during the backup if that is what you have selected in the Master Server Host Properties - Client Attributes tab - but not a true "snapshot"

You could however use the parent_start_notify.cmd script on the Master Server but any commands would then need to be directed remotely to the client - but this does not wait for it to complete.

I guess the only answer is not to select the Perform Snapsot Backups option - if just backing up a single drive it should be fine that was and VSS does still get used - but at least the bpstart_notify will work at the right time

Hope this helps

[RLeon]

Thanks for your input.

Can you elaborate on the seemingly contradictory points:

 1. "If you do not select "Perform Snapshot Backups" it will still use VSS"

 2. "....... but not a true "snapshot"

Also, do you mean I should use the parent_start_notify.cmd script instead? Do you mean I should put it in bin/ on the client? (because you said "on the Master Server")

I guess I'm also confused on why I should "not to select the Perform Snapsot Backups option" when I clearly want Windows to make a VSS snapshot of the drive.

And what do you mean by "but at least the bpstart_notify will work at the right time"..... Under what conditions?

Sorry for the many questions. Netbackup is making this topic very confusing for me.

Thanks for your help,

RLeon

[Mark_Solutions]

Ok - sorry if I have confused things - let me clarify

You have said you want to do snapshot backups - by this I was thinking that you would select the "Perform Snapshot Backups" in the policy attributes tab.

This makes it a special type of backups which specifically snapshots the items to be backed up and then backs up the snapshot.

If you use this option the snapshot will run before your bpstart_notify script and so will not do what you need.

If you do not select this option - just like a normal backup - then the bpstart_notify script will run first and that backup will still utilise VSS during the backup based on capturing any open files. This sounds like it will do exactly what you want unless you have a specific reason to need it to do a snapshot.

If you use the parent_start_notify this will run immediately but will not wait at all and so the snapshot will still run before the script does what you need

As your initial request is to fully snapshot the E drive then neither the bpstart_notify script or parent_start_notify script will do what you need

I can only see that you use a Windows scheduled task to kick of your commands followed by running a bpbackup -i -p policyname -s schedulename

You would then be able to have the bpend_notify script in place to run when it has finished the backup

Hope this helps

[RLeon]

  Thanks for clarifying.

Now that I fully understand what you mean, I am still confused as to why VSS is still used when the option "Perform Snapshot Backups" is explicitly disabled. I understand that this behavior is because Netbackup is trying to be helpful when it sees opened files on the drive, but I'd rather have more control. Am I right in saying that to achieve what I defined as "more control", I will have to explicitly disable "Open files backup" for the client in Master's host properties?

To answer your question, yes, I definately want Netbackup to do a VSS snapshot of E:\. There are two reasons:

1. E:\ is huge. It will take Netbackup hours to backup all its contents, regardless of whether I use flashbackup (raw block snapshot), or simple, old fashioned file level backup.

2. The program that is constantly reading and writing to E:\ cannot be shutdown for that long. It is only allowed less than a minute of down time per day. (Before phones would start ringing)

VSS snapshots could solve the above problem. It only needs seconds to create a snapshot of the entire E:\ drive for Netbackup to start backing up from, IF at the point of the snapshot creation there are no opened/locked files on the drive. (Or if they can be quiesced either manually, or by VSS writers)

So if I understand you correctly, there is no way for Netbackup to call my script BEFORE VSS snapshots the E:\ drive. And, to do what I want, I will have to use means outside of Netbackup's control (like the Windows scheduled task).

If that is indeed the case, then... Well, I will be a very sad Netbackup admin.

Thanks,

RLeon

[Mark_Solutions · Accepted solution]

If you wanted to disable VSS being used at all then yes you would have to set nothing to be used in the Master Servers host properties - having said that even if you did that it would still use it if you were backing up the Shadow Copy Components as it has to!

Unless someone else comes up with a better idea then I think you will be a sad NetBackup Admin!

Of course you could use a little trick ....

Two policies created - the one you really want plus one that backups up a text file on its C drive

The main one has schedules but no open windows so never runs on its own

The new text file one has a bpstart_notify.textpolicyname.bat that stops you processes and then calls your backup using the bpbackup command (no need to use  a -w, just let it fire it off), wait for a period of time to let the snapshot complete and then put run your final processes.

Note quite sure how you will get the timing perfect for all of this - what if your main job is in a queue waiting to start?

Perhaps you could have its own bpstart_notify.mainpolicyname.bat pipe to a text file for each stream and the first one keep checking for when stream 2 appears (which is when the snapshot will have completed)?

An entry such as:

@echo number %stream_number% >> "c:\program files\veritas\netbackup\bin\bpstart.log"
 @echo count %stream_count% >> "c:\program files\veritas\netbackup\bin\bpstart.log"
 @echo pid %stream_pid% >> "c:\program files\veritas\netbackup\bin\bpstart.log"
 @echo ============================================== >> "c:\program files\veritas\netbackup\bin\bpstart.log"

Would create a text file that looks like this:

number 1

count 3

pid 1844

==============================================

number 2

count 3

pid 6148

==============================================

A bit of a workaround I know but should work

Sorry it is not great - maybe someone else will have a "neater" way of doing it

[RLeon]

Your solution makes perfect sense, and I'm very impressed with the scripting suggestion!

I will have to have something in "bpstart_notify.textpolicyname.bat" that would keep checking "bpstart.log" for the 2nd stream. And as soon as the 2nd stream appears in "bpstart.log", the program will be restarted.

I agree this should work. I can see no flaw with the flow, but still...

It makes no sense that Netbackup does not cater to such needs. I mean, even Backup Exec supports Pre-Script-Before-Snapshot. Not to forget that many other enterprise grade backup products also have built-in support for Pre-Script-Before-Snapshot.

Thanks again for your help Mark. I too would like to see if someone here has a better way of doing it.

RLeon

[Mark_Solutions]

Ok - my sample has a double >> soit keeps appending to the file

You may want the bpend_notify to delete the file ready for a new one next time

[Mark_Solutions]

You may also need to add the client to the master servers server list to allow it to call the backup

[RLeon]

Understood. Oh and I probaby won't need the bpend_notify if I could make it delete itself. Better keep things neat.

Thanks,

RLeon

[RLeon]

I just found out from the Snapshot admin guide that the following 2 directives would do exactly what I wanted:

DB_BEGIN_BACKUP_CMD=your_begin_script_path
DB_END_BACKUP_CMD=your_end_script_path

However, the guide also states that the directives are not supported for Windows clients.

So am I correct in saying:
 - For non-Windows, use the two directives.
 - But for Windows, use the workaround you suggested.

If so, then do you know of an ETA for the fix for Windows? Or if there is any plans for getting Windows clients support for those directives?
It doesn't seem very sensible that, in this day and age, we still don't have a Nbu official way to do Pre-Script-Before-Snapshots on Windows.

Thanks,

RLeon

[RLeon]

Anyone tried using DB_BEGIN_BACKUP_CMD on Linux? Is there any GOTCHA's?

Thanks,

RLeon

[Mark_Solutions]

Sorry for not getting back to you earlier - been busy out on site ...

These look fine for Unix only (Flashbackup) policies and i have not noticed that the enhancement has yet been added for Windows

So yes - give them a go for unix, but looks like the workaround on Windows

http://www.symantec.com/docs/TECH127948 says it has been submitted but not heard anything since

Let us know how you get on

[RLeon]

Thanks for getting back to me on this.

Well lets hope they get it sorted out for Windows on Netbackup 7.5.1.

Thanks,

RLeon

[Mark_Solutions]

Not holding my breath!

[CRZ]

If this is something you would really like to see in a future version of NetBackup, you should probably make your sales engineer aware and try to get them to light a fire under the previously opened Etrack, which is 2023815.  Let them know which version of NetBackup you're running (I'm assuming it's 7.5) and that this is an important issue to you.

[RLeon]

Thanks for the suggestion and the Etrack number, I (we) will see what can be done about it.

RLeon