We have a case where we need to ensure that a Postgres DB, NFS presented file system and virtual machine (currently centos) are backed up in a consistent state.
Our plan is to stop the application, take a VM snapshot of the server and the DB server (also centos) and a LUN snapshot then restart the application. Once all snapshots are taken we can then backup these snapshots at leisure.
The stop / snapshot / start events will take place outside of the NetBackup environment. We did consider trying to do these are part of the backup job but we currenty don't have an agent on the application server and couldn't see how to get the applications to start again in the pre notify scripts as at this point the snapshot had not been taken.
Our issue is how to mount the ESXi snapshot to allow Netbackup to back it up. We can't use NBU snapshots because of the need to stop and restart the application.
So ideally the plan is
at a pre defined time (probably 11pm)
1) Using scripting (either powershell on backup server, or bash from application server) shutdown the application and database
2) Create ESXi snapshot
3) Create LUN snapshot of NFS storage
4) Mount ESXi snapshot so it can be seen by Netbackup
5) Mount LUN snapshot so it can be seen by Netbackup
4) restart application and database
Then at a random time later on backup the two (ESXi and LUN) snapshot.
How do we mount and access these snapshots?
Am I going about this entirely the wrong way and is there a better way! I'm not responsible for the backups but have been given the task of scripting it all. Our backup engineers are not sure what the best way is either. Regular NBU snapshot ESXi backups work great as do NDMP backups of CIFS shares and other files system on the VNX storage but these don't use snapshots.
What you're describing is not (entirely) possible but I'm curious why this is a requirement. Is the application so sensitive that it can't tolerate or recover from an inconsistency (assuming any piece of the 3 tier app needs to be restored)?
Points 4 is not possible. You can't take a snapshot of either a VMWare datastore (assuming that is what you want to snapshot) and then mount it somewhere. A snapshot of the datastore can only be mounted back on vSphere and even if you do that you would need to register the VM again, discover it and then back it up via a VMware policy.
You can just take a VM backup of the application server, it will be consistent and even more so if you want to stop the application.
There is an agent for Postgres, why don't you just use that?
The NDMP backup of the NAS might take long but then again it depends on the size.
You should investigate the use and operation of the VMware pre and post snapshot scripts on the DB client and incorporating those into your standard VM backup methodolgy.
This requires VMware tools to be installed but that should be the case.
The pre snapshot script is called by VMware prior to performing the snapshot, and can basically do whatever you want.
This would be part of a standard NetBackup VMware policy that would initiate the snapshot via VMware, the script runs, does the required DB stuff, snapshot taken, then NBU can get a clean copy. Post snapshot script would reverse the steps and restart the DB.
Great question - and as responded already as I see it you have three options
1) perform a scripted backup of the DB to another share to backup on a VM (dump and sweep - more storage and time required) along with NFS and VM policies
2) script independently and perform a standard backup on a VM ( to get the VM and not touch db you could exclude data disks in the policy - not sure if thats 100% going to work if VM is centos so you'd need to be specific in the policy of drive choice, or make sure application is in quiesced state. You can use bpstart.notify and bpend.notify scripts to trigger from within backup job
3) use the agent to backup the DB in one policy and seperate policies for the VM and NFS shares - this will be the least amount of work, most supportable and quickest implementation and a place to start would be here: https://www.veritas.com/content/support/en_US/doc/129277259-129955710-0/v129276450-129955710
Hope that helps.
Thanks all for some useful comments
I'm not quite sure how sensitive everything is to being consistent but the tools owners have decreed that it all needs to be backed up at once and whilst shut down.
The VM snapshot was my big issue and RiaanBadenhorst has confirmed this is not going to be possible.
We've take a more detailed look at the content of the machines and have the following
The application is Atlassian BitBucket Data Center consisting of
Application Server - currrently contains the git repositories but we think we can move these to an NFS mount
DB Server - contains mainly security / config data so not really insync with the repos
LFS NFS mount - contains large file storage - e.g. anything binary and large that does not really fit into a git repository.
We can't use the bpnotify scripts as (please correct me if I'm wrong) these run before and after a client backup. This would mean the application would need to be down until the backup has finished. It's good for a client file system backup but not when we need to take snapshots to keep downtime to a minimum.
However the idea of using the vmware freeze and thaw scripts looks like it may be a winner - thanks davidmoline.
The new plan:
1) Trigger a VM backup of the DB server from Netbackup. This will cause ESXi to call the freeze script on the DB server (through vmtools). This script (bash) will shutdown the application and database
2) Create LUN snapshot of NFS storage and probably the repository NFS storage
3) Mount LUN snapshot so it can be seen by Netbackup
4) Freeze script returns to ESXi and the VM snapshot is taken as normal
5) Once the VM snapshot is complete ESXi calls the thaw script which will restart application and database.
6) Netbackup will now backup the VM snapshot and proceed to backup the LUN snapshot
7) The application VM can be backed up later out of step as we have moved all the critical data from it
Does this sound more achievable?
Your plan mostly looks okay with one important problem.
Step 4 5 & 6 need to be rethought. A VM snapshot creates a transaction log file for want of a better term. Updates while the snaphot is active go to this file to be replayed into the VMDK when the snapshot is deleted.
So you are actually backing up the VMDK files which are now idle due to the snapshot, you are not backing up the snapshot itself.
I think we are just refering to snapshots in different ways. I'm, probably incorrectly, refering the snapshot as the frozen files when technically it could be the delta files. It is the VMDKs I want to back up so we are backing up the frozen state of the machine. The transaction logs (deltas) as you say are the changes since that point and I don't want to backup those.
I'll be letting Netbackup / ESXi take care of what is actually being backed up as it would with a regular EXSi backup.