In my struggles to understand various instant snapshot technologies, I have come up with the following way to help me better understand the fundamental concepts behind all snapshot products.
Please note that everything covered here are deliberately oversimplified. I also refer to LUNs, disks, volumes and partitions interchangeably.
By sharing my understandings, I hope others who are as confused as I was would find this helpful.
I would also be very grateful if someone would be kind enough to point out any errors in my understandings. It helps everyone in the end and will be appreciated.
Please note, that although all my examples are based on Windows platforms, the fundamental concepts apply to *nix platforms too.
Snapshots - What are they:
To explain how and why it works is out of the scope of this simple post. In most cases it utilizes Copy-On-Write technologies, which I would recommend reading up on.
For the scope of this post, all you need to know is that it could make a clone of your entire disk instantly.
That's right, instead of taking minutes or hours to copy every single file off your disk, a snapshot could do it instantly. (Or tens of seconds, depending on your "snapshot level", explained later)
If you are still unclear what that means: Yes, a snapshot is like a full backup of your entire disk that takes less than a second to complete. It is a point in time copy of everything, hence the name Snapshot.
So then, after a snapshot is taken, you have your original disk, and an exact block-identical clone of it.
You could think of the snapshot as a mirror disk of the original, only that the original is still active, and changes are still being written to it. The snapshot on the other hand, is just a static, point-in-time copy of the original. Less and less up-to-date by the second compared to the original.
(Note: I will not be explaining where snapshots are stored. All this cloned data has got to go somewhere right? If you are interested, please read up on Copy-On-Write technologies. Different products and Snapshot Levels implement Copy-On-Write differently.)
What are snapshots good for:
Why, for Netbackup to backup from, of course!
Snapshots are usually useful for backups, but there are many other uses as well. If your snapshot is a disk snapshot, then you practically have two disks with identical data. You could then mount this "cloned disk" on a different machine for testing or other purposes. The point is, what you do with this "cloned disk" would not affect your precious original disk one bit (pun? pun.). They are like identical twins who got separated. There are many other uses for snapshots as well, some very advanced, but again, for the scope of this post, just think of a snapshot as an instantly cloned LUN/disk/volume/partition.
Blocks and Snapshots:
From Wikipedia: a block is a sequence of bytes or bits, having a nominal length (a block size).
That's it really. Your disk has a bunch of blocks on it, on which you write your file system and files.
In most cases, snapshots operate on the block level, meaning that whoever/whatever creating the snapshot doesn't really care what kind of file systems or files are stored on the blocks. It simply (blindly) copies all the blocks it sees in an instant, then name the resulting clone a snapshot.
OS Consistency and Application Consistency:
"OS" could be any OS really. Windows, *nix or whatever you use.
When a group of blocks is said to be OS Inconsistent, it means the OS considers the files on these blocks to be corrupted. If you are lucky, these OS Inconsistent blocks don't affect the OS-critical files, and you could still boot your machine and run it, somewhat.
An example is when your Windows machine is powered off abruptly. When it forces you to go in to chkdsk mode in the next boot, it is really just trying to correct the OS Inconsistent blocks, to make them, well, OS Consistent, or "uncorrupted".
Application Consistency works the same way, but just that it is on the application level, built one level up on top of the OS level.
There can be situations where a group of blocks is OS Consistent, but Application Inconsistent. That means while the OS doesn't think the files on those blocks are corrupted, to an application they are.
An example could be made with MS SQL. A database file (with .mdf extension) could be considered by Windows to be OS Consistent, but MS SQL may consider it to be corrupted and refuses to mount the DB. We call the blocks that hold the mdf file Application Inconsistent. This could happen when the SQL Server processes are shutdown abruptly while the database file is being written to. (There are many SQL HA solutions to protect against this, but let's not...)
As you can see, shutting down an application abruptly corrupts application-files, just as shutting down an OS abruptly corrupts OS-files. Just remember that all these files are stored on blocks. ("Composed of blocks" would be more accurate)
Block States:
OK, I'll admit I made this part all up. But in order to understand the rest of the post, please bear with me.
A block can have one of four states at any given instant, which may change in the next instant:
(Let's oversimplify: think of an instant as 1/1000000000000000 of a second)
State 1:
It is actively being written to (being changed). This state is not what you would call "stable" or "static".
State 2:
It is not being written to. It is possible that it is being read from, but it is definitely not being changed.
This state is OS Inconsistent - The OS hasn't finished writing to this block yet, but it is just not being written to at this instant. Any file(s) on the block is considered to be corrupted at this instant.
State 3:
It is not being written to.
It is OS Consistent - The OS has finished writing to the block, and the block is not considered to be corrupted.
It is Application Inconsistent - An application hasn't finished writing to this block yet, but it is just not being written to at this instant. Any file(s) on the block is considered BY THE APPLICATION to be corrupted at this instant.
This state could only be achieved after State 2.
State 4:
It is not being written to.
It is OS Consistent - The OS has finished writing to the block, and don't consider the file on it to be corrupted.
It is Application Consistent - The application has finished writing to the block, and don't consider the file on it to be corrupted.
This state could only be achieved after State 3.
Snapshot Levels:
OK I made this "Snapshot Level" thing up too. But I hope you will see how this would make sense.
Consider a disk that has a total of 25 blocks, 5 across and 5 down:
=Figure A=
1 1 1 1 1
1 1 1 1 4
1 3 1 1 1
1 1 1 1 1
1 1 1 1 1
The three left most columns are where the OS stores its data. (Tiny OS I know.)
The two right most columns are where the application stores its data.
The digits refer to the Block States at a specific instant, or point in time.
If a snapshot is taken of the disk at this point in time:
- The snapshot will not have any meaningful use because nearly everything is corrupted.
- Most of the OS blocks are in State 1. As far as the OS is concerned, the entire OS is corrupted, so don't even think about using this snapshot for any recovery purposes.
- Most of the application blocks are in State 1. As far as the application is concerned, nearly all its files are corrupted .The only State 4 "good" block could be of little use even if retrieved.
The above is basically what happens when you take a snapshot when the disk is constantly being written to. I.e., during heavy I/O.
I call it a Level 1 Snapshot. (Aka. The useless snapshot)
What about a Level 2 snapshot?
Just replace all the 1s in the above with 2s and take a snapshot.
For all practical purposes it is just as useless as a Level 1 snapshot, but this time the snapshot is taken in an instant when there are virtually no writings to the disk.
As an example, this is basically what happens when the OS is powered off abruptly and then you take a snapshot of its disk. (Can be done by storage hardware products, such as the EMC SnapView, as discussed in the NBU Snapshot Admin Guide.)
Now consider the following:
=Figure B=
3 3 3 3 3
3 3 3 4 3
3 3 3 4 3
3 3 3 3 4
3 3 3 3 3
If a snapshot is taken of the disk at this point in time:
- The OS part of the disk is in an OS Consistent state. If you mount this snapshot to a different machine (with the same hardwares), it will boot up and run without any problems. This snapshot can be used for OS recovery purposes.
- The OS data of this snapshot should be just as good as if you cold-cloned it using some boot CD program like Ghost. It is just that here you snapshotted it live while the machine is running (hot-cloned), and that's very impressive. Cold-clone products would be jealous.
- The application part of this snapshot is not as lucky. Although all its blocks (files, really) are considered "Good" by the OS, things are not so on the application level. Most blocks are Application Inconsistent, so the application will probably run with many errors.
If the application is smart enough, it may still be able to recover some useful data from the three State 4 blocks.
I call the above snapshot a Level 3 Snapshot.
Next we have:
=Figure C=
3 3 3 4 4
3 3 3 4 4
3 3 3 4 4
3 3 3 4 4
3 3 3 4 4
If a snapshot is taken of the disk at this point in time:
- It has all the benefits of a Level 3 Snapshot.
- All application blocks are Application Consistent.
- The way that all the blocks are OS/Application Consistent is equivalent to when the machine is powered down gracefully. I.e., this is what you would see if you look at the blocks of a properly powered down machine.
- If you mount this snapshot to a different machine (with the same hardware), both the OS and the application should run without any problems. It would be as if the machine was shutdown properly, and is now booted up.
- The data of this entire disk should be just as good as if you cold-cloned it using some boot CD program like Ghost. It is just that here you snapshotted it live while the machine is running (hot-cloned).
I call the above snapshot a Level 4 Snapshot. (Aka. The only useful snapshot)
Putting It All Together:
Normally when you take a snapshot, you really only want a Level 4 Snapshot, because it is the only one that is really useful.
When you take a snapshot of a disk for Netbackup to backup from, you don't really want any of the OS level files or application level files to be in a corrupted state.
Imagine you have a really big file that would take 1 hour to be backed up by traditional backup methods, but you can't really back it up because it is always being locked by the OS (or an application) for constant I/O.
Now suppose it is acceptable for you to stop I/O to the file, and make it OS/Application Consistent for 10 seconds per day, that would mean you only have 10 seconds to backup this file in its consistent state.
With Snapshots, those 10 seconds are more than enough to clone not just the file, but the entire disk. (You may in the end choose to only backup that one file from the disk snapshot, but the snapshot is still based on the entire disk. Per-file or per-block snapshots are out of the scope of this post).
With the snapshot of the disk taken, Netbackup could just start backing up from it, and take however long it wants. The point here is, while Netbackup is still spending time backing up the file (consistent) from the snapshot, the real file has already long resumed its I/O operations on the original disk. (I.e., it is locked again)
In Netbackup and VMware, when you backup an entire VM that is running (not powered off), the following is what actually happens, in strict order:
(I believe the same concept applies to Hyper-V)
1. Netbackup calls the ESX host via the vStorage API, telling it to take a snapshot of the entire VM.
2. The ESX host speaks to the VM Windows OS via the installed VMware Tools application.
3. VMware Tools speaks to the Windows VSS service and tells it that it wants an OS/Application Consistent snapshot of everything. (If interested, you may want to do some reading up on the Windows Volume Shadow Copy Service)
4. VSS speaks to all the applications that listen to it (SQL, Exchange, AD, etc.), to prepare for imminent snapshot by temporarily making all their blocks Application Consistent, aka, State 4. (Applications that "listen" are those with VSS Writers)
5. VSS speaks to all the OS components, to prepare for imminent snapshot by temporarily making all their blocks OS Consistent, aka, State 3.
6. As soon as everything is consistent (only "3"s and "4"s if you plot the blocks like I did), a snapshot is taken at this very instant.
7. As soon as the snapshot is successfully taken, all the Applications and OS components return to their I/O activity. (Many "1"s and "2"s would show up now.)
8. The snapshot is handled by the VSS Provider, and through the VMware Tools, transferred to the ESX. (If you check your VM now, you would see that a snapshot has been created)
9. This OS and Application Consistent snapshot (aka Level 4 Snapshot) is transferred directly from the ESX host to the Netbackup media server, by the magic of the vStorage API.
10. Netbackup backs up the VM snapshot peacefully and without hurry.
11. After the backup, the snapshot is deleted on the ESX.
FAQs:
Can snapshots backup opened/locked files, as in files under heavy I/O?
If you are talking about a Level 1 Snapshot, then yes. Just remember that everything you just snapshotted is probably corrupted as far as the OS/Application is concerned.
If you are talking about a Level 4 Snapshot, then technically, no, although the file would appear to still be opened/locked as the snapshot is being taken.
Referring back to the above 1 to 11 steps, from the beginning of step 4 to the end of step 7 it would normally only take 10 or so seconds, sometimes less.
In most cases it would appear as if the file I/O never stopped.
Whenever you hear about glorious open file backup solutions and how amazing they are, they are really talking about Level 4 Snapshots.
Steps 4 and 5 are usually referred to as Quiescing.
I hear snapshots should be instantaneous, why is it that some people insist that the OS/Applications get "paused" for like a minute?
Read my answer to the first question.
If it takes that long, then the hardware is probably very slow, or there are problems with VSS. (Could be the service, the writers, or the provider)
What's the difference between LUN, disk, volume or partition snapshots?
Not alot really. Some products support all, some support less.
In netbackup I think it snapshots individual partitions by default, unless you set it to snapshot all drives at the same time, in which case, it would be equivalent to a disk snapshot. (If we are talking about 1 disk and many partitions)
Storage hardware vendors, such as EMC, HP, IBM, etc., can do LUN level snapshots (which happens on the SAN level).
If you snapshot the LUN without regards to what the OS is doing to it, then it is going to be a Level 1 snapshot.
Some hardware LUN snapshot methods have some kind of ways to interface with the LUN owner machine's Windows VSS, much like in the VMware example.
If that is the case, then the same steps apply (mostly), and the snapshot can be a Level 4 Snapshot.
Why couldn't I find much information about all this? And when I do find some why do they look contradictory?
As I have explained, I made up all the "Snapshot Levels" in this post, so that I (or others) could more easily understand the different... levels of snapshots.
When you read about snapshots in some product documents, they usually only focus on the Snapshot Level that is relevant to the product.
For example, a storage hardware that could do LUN snapshots may focus on the fact that their snapshots are instantaneous, with no waiting time.
That probably means that it could ignore all the VSS and quiescing stuff and do Level 1 Snapshots (Yey).
When you read about a different product that "only" needs around 10 seconds of zero I/O to make a snapshot, it is probably talking about a Level 4 Snapshot.
What is the difference between a hot-clone, a cold-clone, and a snapshot of a disk?
A cold-clone usually means you had to power down the machine, boot it up with something like Ghost and copy every block off the disk.
A hot-clone does the same as the above, but does it without the machine having to be powered down.
The OS and its applications can be fully running during the "hot-cloning process".
A Level 4 Snapshot is essentially a hot-clone.
Who or what exactly creates snapshots?
On the Windows OS level, the VSS service coordinates and creates the snapshots.
On the VM level, the ESX host creates the snapshots of entire VMs. (But really it just delegates some of the tasks to the OS on the VM.)
On the SAN level, the storage hardware controller creates the snapshots of the LUNs. In most cases this happens outside the control and awareness of the hardware or OS using the LUN.
All these snapshots can be presented to Netbackup for backing up, via one way or another.
What does it mean to have a Level 3 Snapshot?
It means that if you do a complete restore from it, your OS would run fine, but the application(s) would have errors because the application blocks/files are not Application Consistent.
The same would happen if you mount this snapshot to a different machine (same hardware) and boot from it.
Always go for a Level 4 Snapshot if possible. Levels 1 and 2 Snapshots (sometimes lv 3 too) are referred to as Crash Consistent Snapshots. Because they, well, crash.
A little more information about VSS?
The Windows VSS is essentially a tool to quiesce all blocks/files just right before a snapshot is taken.
In other words, it makes all OS blocks State 3 and all application blocks State 4 (A disk state as if the machine is powered down properly), and then it takes a snapshot of the disk.
The resulting snapshot is what I'd call a Level 4 snapshot, where nothing is in a corrupted state.
The whole VSS thing is supposed to work very quickly. So quickly that people just call it some kind of open/lock file backup.
But in actuality, the opened/locked files had to have been stopped being opened/locked for a very brief period, and made consistent (quiesced), for a Level 4 Snapshot to be taken.
Can Windows VSS quiesce every application?
No. Only those that has a VSS Writer.
In other words, only those applications that were designed/coded to work with VSS. Examples are the Windows OS itself, MS SQL, Exchange, AD, Sharepoint, etc.
Some other non-MS products such as Oracle for Windows also have VSS Writers, and could be asked by the VSS service to have their files quiesced.
I have an application that constantly locks its files (Block State 1), and it doesn't have a VSS Writer, what now?
If it doesn't have a VSS Writer, it doesn't support VSS. If you forcefully take a snapshot of the disk on which the application stores its files, the blocks/files will be Application Inconsistent.
I.e., if you restore the application files (corrupted) from the snapshot, your application will have a lot of errors and may not run properly, if at all.
The only way to take Application Consistent snapshots for such an application, is to have it manually quiesced.
In most cases that means manually doing a graceful shutdown of the application (so its blocks are in State 4), take a snapshot, then manually restarting it.
People usually do this using a pre-snapshot script, a post-snapshot script, and run them using the OS's own scheduler tool.
This is so that, even without VSS support, the whole App Consistent snapshot task can still be somewhat automated.
What about non-Windows?
See the answer to the previous question.
VSS is for Windows only.
For non-Windows platforms, the OS itself could usually be quiesced by the installed software that calls for the OS snapshot. (Such as a backup software agent)
People usually use pre/post-snapshot scripts to quiesce their applications on non-Windows platforms, to make sure the resulting snapshot is App Consistent.
So it goes like this:
1. Pre-snapshot script shutdown application
2. Snapshot of the disk taken, together with the OS.
3. Post-snapshot script restarts the application
A Level 4 Snapshot is created with minimum application down time.
If you got this far down, I thank you for taking time to read through the whole thing.
I'm sure there are probably many errors and misconceptions (and grammatical/spelling errors too), but as I have said, inputs and corrections will be greatly appreciated if someone would be kind enough to point them out, so that everyone could learn.
Thanks all,
RLeon
