Forum Discussion

RLeon's avatar
RLeon
Level 6
13 years ago

Open file backups, VSS snapshots and bpstart_notify scripts

Hi all,

This query is mainly for Windows, but may possibly apply to *nix platforms.

If a drive (say drive E:\), is to be VSS snapshotted and backed up, is this snapshot created BEFORE OR AFTER the bpstart_notify.bat script is run?
This is important because I may want to use bpstart_notify.bat to stop all I/Os to some opened files on E:\ before the snapshot should be taken.

Here is an example (which is really just my presumptions):
A custom program not supported by Netbackup, and has no VSS writer, is deployed on a Windows host.
It reads/writes some files on E:\ and never releases the file locks unless the program itself is shut down.

A script is created that does 2 things:
1.
Gracefully shuts down the program.
This is so that the file locks on E:\ are gone, and that the drive can be fully quiesced and snapshotted.
2.
After the shutdown (step 1. above), count 15 seconds and start the program again.
This means VSS is given a generous 15 seconds to take a snapshot of E:\
NOTE: This script has to be a non-blocking script. (IMPORTANT)

A line is placed in bpstart_notify.bat to call the script.

When the backup job starts, the following happens in strict order:
1. bpstart_notify.bat calls the script.
2. The script shuts down the program, which releases the file locks on E:\
2a. Since the script is a non-blocking script, Netbackup doesn't wait for it to finish.
3. The script at this point is still running a 15 seconds countdown, but AT THE SAME TIME, VSS takes a snapshot of the entire E:\, which should be successful seeing that no files are opened/locked.
4. The script completes its 15 seconds countdown and restarts the program.
5. Netbackup can take however long it wants to backup the snapshot. 1 hour? 2 hours? no problem! What's important is that the program is running and the backup doesn't interfere with it.

The above is what I presume how bpstart_notify.bat works in this situation.
So is this correct?
What I worry most is: What if the Windows VSS snapshot is taken BEFORE bpstart_notify.bat is run?
That would mean the snapshot would fail because there are still open/locked files on E:\.
If that happens, then the script that shuts down the program would be pointless because a failed snapshot attempt would already have happened before the file locks are released.


Thanks!

RLeon

7.1.x and Earlier
Agents
Backing Up
Backup and Recovery
Basics
Best Practice
Configuring
NetBackup
Tip-How to
  • Mark_Solutions's avatar
    Mark_Solutions
    13 years ago

    If you wanted to disable VSS being used at all then yes you would have to set nothing to be used in the Master Servers host properties - having said that even if you did that it would still use it if you were backing up the Shadow Copy Components as it has to!

    Unless someone else comes up with a better idea then I think you will be a sad NetBackup Admin!

    Of course you could use a little trick ....

    Two policies created - the one you really want plus one that backups up a text file on its C drive

    The main one has schedules but no open windows so never runs on its own

    The new text file one has a bpstart_notify.textpolicyname.bat that stops you processes and then calls your backup using the bpbackup command (no need to use  a -w, just let it fire it off), wait for a period of time to let the snapshot complete and then put run your final processes.

    Note quite sure how you will get the timing perfect for all of this - what if your main job is in a queue waiting to start?

    Perhaps you could have its own bpstart_notify.mainpolicyname.bat pipe to a text file for each stream and the first one keep checking for when stream 2 appears (which is when the snapshot will have completed)?

    An entry such as:

    @echo number %stream_number% >> "c:\program files\veritas\netbackup\bin\bpstart.log"
     @echo count %stream_count% >> "c:\program files\veritas\netbackup\bin\bpstart.log"
     @echo pid %stream_pid% >> "c:\program files\veritas\netbackup\bin\bpstart.log"
     @echo ============================================== >> "c:\program files\veritas\netbackup\bin\bpstart.log"

    Would create a text file that looks like this:

    number 1

    count 3

    pid 1844

    ==============================================

    number 2

    count 3

    pid 6148

    ==============================================

    A bit of a workaround I know but should work

    Sorry it is not great - maybe someone else will have a "neater" way of doing it

16 Replies

Sort By
  • Mark_Solutions's avatar
    Mark_Solutions
    Level 6
    13 years ago

    If you select "Perform Snapshot Backups" in the Policy Attributes tab then it run the snapshot before it runs the bpstart_notify script

    The snapshot gets run immediately by the parent job and once complete it kicks off the child job which runs the bpstart_notify script

    If you do not select "Perform Snapshot Backups" it will still use VSS during the backup if that is what you have selected in the Master Server Host Properties - Client Attributes tab - but not a true "snapshot"

    You could however use the parent_start_notify.cmd script on the Master Server but any commands would then need to be directed remotely to the client - but this does not wait for it to complete.

    I guess the only answer is not to select the Perform Snapsot Backups option - if just backing up a single drive it should be fine that was and VSS does still get used - but at least the bpstart_notify will work at the right time

    Hope this helps

  • RLeon's avatar
    RLeon
    Level 6
    13 years ago

    Thanks for your input.

    Can you elaborate on the seemingly contradictory points:

     1. "If you do not select "Perform Snapshot Backups" it will still use VSS"

     2. "....... but not a true "snapshot"

     

    Also, do you mean I should use the parent_start_notify.cmd script instead? Do you mean I should put it in bin/ on the client? (because you said "on the Master Server")

    I guess I'm also confused on why I should "not to select the Perform Snapsot Backups option" when I clearly want Windows to make a VSS snapshot of the drive.

    And what do you mean by "but at least the bpstart_notify will work at the right time"..... Under what conditions?

     

    Sorry for the many questions. Netbackup is making this topic very confusing for me.

     

    Thanks for your help,

     

    RLeon

  • Mark_Solutions's avatar
    Mark_Solutions
    Level 6
    13 years ago

    Ok - sorry if I have confused things - let me clarify

    You have said you want to do snapshot backups - by this I was thinking that you would select the "Perform Snapshot Backups" in the policy attributes tab.

    This makes it a special type of backups which specifically snapshots the items to be backed up and then backs up the snapshot.

    If you use this option the snapshot will run before your bpstart_notify script and so will not do what you need.

    If you do not select this option - just like a normal backup - then the bpstart_notify script will run first and that backup will still utilise VSS during the backup based on capturing any open files. This sounds like it will do exactly what you want unless you have a specific reason to need it to do a snapshot.

    If you use the parent_start_notify this will run immediately but will not wait at all and so the snapshot will still run before the script does what you need

    As your initial request is to fully snapshot the E drive then neither the bpstart_notify script or parent_start_notify script will do what you need

    I can only see that you use a Windows scheduled task to kick of your commands followed by running a bpbackup -i -p policyname -s schedulename

    You would then be able to have the bpend_notify script in place to run when it has finished the backup

    Hope this helps

  • RLeon's avatar
    RLeon
    Level 6
    13 years ago

     Thanks for clarifying.

    Now that I fully understand what you mean, I am still confused as to why VSS is still used when the option "Perform Snapshot Backups" is explicitly disabled. I understand that this behavior is because Netbackup is trying to be helpful when it sees opened files on the drive, but I'd rather have more control. Am I right in saying that to achieve what I defined as "more control", I will have to explicitly disable "Open files backup" for the client in Master's host properties?

    To answer your question, yes, I definately want Netbackup to do a VSS snapshot of E:\. There are two reasons:

    1. E:\ is huge. It will take Netbackup hours to backup all its contents, regardless of whether I use flashbackup (raw block snapshot), or simple, old fashioned file level backup.

    2. The program that is constantly reading and writing to E:\ cannot be shutdown for that long. It is only allowed less than a minute of down time per day. (Before phones would start ringing)

    VSS snapshots could solve the above problem. It only needs seconds to create a snapshot of the entire E:\ drive for Netbackup to start backing up from, IF at the point of the snapshot creation there are no opened/locked files on the drive. (Or if they can be quiesced either manually, or by VSS writers)

    So if I understand you correctly, there is no way for Netbackup to call my script BEFORE VSS snapshots the E:\ drive. And, to do what I want, I will have to use means outside of Netbackup's control (like the Windows scheduled task).

    If that is indeed the case, then... Well, I will be a very sad Netbackup admin.

    Thanks,

    RLeon

  • Mark_Solutions's avatar
    Mark_Solutions
    Level 6
    13 years ago

    If you wanted to disable VSS being used at all then yes you would have to set nothing to be used in the Master Servers host properties - having said that even if you did that it would still use it if you were backing up the Shadow Copy Components as it has to!

    Unless someone else comes up with a better idea then I think you will be a sad NetBackup Admin!

    Of course you could use a little trick ....

    Two policies created - the one you really want plus one that backups up a text file on its C drive

    The main one has schedules but no open windows so never runs on its own

    The new text file one has a bpstart_notify.textpolicyname.bat that stops you processes and then calls your backup using the bpbackup command (no need to use  a -w, just let it fire it off), wait for a period of time to let the snapshot complete and then put run your final processes.

    Note quite sure how you will get the timing perfect for all of this - what if your main job is in a queue waiting to start?

    Perhaps you could have its own bpstart_notify.mainpolicyname.bat pipe to a text file for each stream and the first one keep checking for when stream 2 appears (which is when the snapshot will have completed)?

    An entry such as:

    @echo number %stream_number% >> "c:\program files\veritas\netbackup\bin\bpstart.log"
     @echo count %stream_count% >> "c:\program files\veritas\netbackup\bin\bpstart.log"
     @echo pid %stream_pid% >> "c:\program files\veritas\netbackup\bin\bpstart.log"
     @echo ============================================== >> "c:\program files\veritas\netbackup\bin\bpstart.log"

    Would create a text file that looks like this:

    number 1

    count 3

    pid 1844

    ==============================================

    number 2

    count 3

    pid 6148

    ==============================================

    A bit of a workaround I know but should work

    Sorry it is not great - maybe someone else will have a "neater" way of doing it

  • RLeon's avatar
    RLeon
    Level 6
    13 years ago

    Your solution makes perfect sense, and I'm very impressed with the scripting suggestion!

    I will have to have something in "bpstart_notify.textpolicyname.bat" that would keep checking "bpstart.log" for the 2nd stream. And as soon as the 2nd stream appears in "bpstart.log", the program will be restarted.

    I agree this should work. I can see no flaw with the flow, but still...

    It makes no sense that Netbackup does not cater to such needs. I mean, even Backup Exec supports Pre-Script-Before-Snapshot. Not to forget that many other enterprise grade backup products also have built-in support for Pre-Script-Before-Snapshot.

    Thanks again for your help Mark. I too would like to see if someone here has a better way of doing it.

    RLeon

  • Mark_Solutions's avatar
    Mark_Solutions
    Level 6
    13 years ago

    Ok - my sample has a double >> soit keeps appending to the file

    You may want the bpend_notify to delete the file ready for a new one next time

  • Mark_Solutions's avatar
    Mark_Solutions
    Level 6
    13 years ago
    You may also need to add the client to the master servers server list to allow it to call the backup
  • RLeon's avatar
    RLeon
    Level 6
    13 years ago

    Understood. Oh and I probaby won't need the bpend_notify if I could make it delete itself. Better keep things neat.

    Thanks,

    RLeon

  • RLeon's avatar
    RLeon
    Level 6
    13 years ago

    I just found out from the Snapshot admin guide that the following 2 directives would do exactly what I wanted:

    DB_BEGIN_BACKUP_CMD=your_begin_script_path
    DB_END_BACKUP_CMD=your_end_script_path

    However, the guide also states that the directives are not supported for Windows clients.

    So am I correct in saying:
     - For non-Windows, use the two directives.
     - But for Windows, use the workaround you suggested.

    If so, then do you know of an ETA for the fix for Windows? Or if there is any plans for getting Windows clients support for those directives?
    It doesn't seem very sensible that, in this day and age, we still don't have a Nbu official way to do Pre-Script-Before-Snapshots on Windows.

    Thanks,

    RLeon