NetBackup Instant Recovery for VMware enables you to boot up and run multiple virtual machines directly from backup image. As far as the end user of the virtual machine is concerned, there is no functional difference between an instance of VM run from production storage or the one run from NetBackup backup storage. NetBackup presents the virtual machine files from the backup storage as an NFS data store for the ESXi host. Let us take a look at the under the hood details on how this capability is implemented.
A bit of history first. This concept of Instant Recovery (i.e. a backup copy functioning as the production copy) is not new in NetBackup. Symantec has supported this capability for over a decade, since the release of Snapshot Client option in NetBackup 4.5. In simple terms, NetBackup would treat a supported snapshot (e.g. snapshot taken using Symantec Storage Foundation, Supported array snapshot methods like EMC TimeFinder, Hitachi ShadowImage, HP Business Copy etc.) as if it were a backup image and catalog its data. At the time of recovery, the entire snapshot may be rolled back as production (known as Instant Recovery Rollback) or items may be copied from the snapshot (known as Instant Recovery Restore).
Similarly, presenting the backup image as a secure NFS volume is also not new in NetBackup. This was introduced in NetBackup 6.5.3 released over half a decade ago. When we developed a method to present backup image as a file system so as to enable application item recovery from enterprise applications like Microsoft Exchange, Active Directory etc., we had to choose a protocol that will work across platforms as NetBackup supports various UNIX flavors, Enterprise Linux flavors and Windows as media servers. NFS naturally filled this cross-platform requirement but our implementation needed to be secure, as backup of production workloads cannot be exposed via general-purpose NFS servers.
We solved this problem by building a purpose-built NFS server for NetBackup, known as nbfsd (NetBackup File System Daemon) that publishes backup image as a file system with a self-generated set of credentials based on globally unique identifiers (GUIDs) that are required for mounting/unmounting on/from a client. The NetBackup client package also includes a purpose-built program called nbfs (NetBackup File System) that uses these credentials to mount/unmount file systems. Unlike standard NFS, NetBackup's implementation requires just two ports open on the media server -- port 7394 (configurable) and port 111 (the standard RPC port mapper access port). This makes the solution firewall-friendly and also adds to security.
NetBackup Instant Recovery for VMware, introduced in NetBackup 7.6 is a fusion of these two well established, proven technologies enhanced further for use in virtualized environments. Although the ‘classic Instant Recovery’ in NetBackup required storage level snapshots, the newer Instant Recovery for VMware eliminates that need. Now when combined with NetBackup’s ability export a backup image as a secure NFS volume, the ESXi host can mount that volume as a data store and boot up the virtual machine. In the virtualized world, we wanted to meet a few requirements to meet the needs of enterprise grade applications.
- As always, security is paramount to Symantec. As ESXi host has a general-purpose NFS client, we wanted to implement a solution for ESXi that will not compromise security. We established this by building a proxy server that will inherit the credentials and attaches to the secure port to facilitate the mount operation on ESXi host. The whole operation is given a very small time window to finish, typically less than a minute. Once the image is mounted (or the timer expires), the NetBackup backup image is no longer available for mount operations. Thus, even if you have a copy of GUID based credentials generated at the time of secure NFS export; it won’t work anymore after the timer expires. A new set of GUID based credentials need to be issued for further attempts. Thus NetBackup prevents an insider from mounting a backup image using a general purpose NFS server including those present in ESXi hosts.
- The virtual machine running from backup storage is useful only if it accommodates writes and reads efficiently. Redirecting write operations to local data store enhances the user/application performance as if the VM is running from production storage!
- NetBackup 7.6 also has made significant enhancements to read I/O from deduplication storage that includes content-aware pre-fetching to smoothen read performance. These performance enhancements for virtual machine backup workloads were essential to make Instant Recovery suitable for enterprise grade workloads. (Hint: This is an important reason why NetBackup Instant Recovery is supported only from NetBackup’s native storage units like Media Server Deduplication Pool, Advanced Disk Pool etc.) Let us save the new self-healing architecture and other improvements made in NetBackup Deduplication for another blog.
- From 2 & 3, NetBackup accommodates running multiple VMs from the same backup storage unit. Our goal in this release had been to provide enough concurrency to accommodate a distributed application like Exchange or SharePoint. For example, in case of a site loss; if backups were made available via NetBackup Auto Image Replication at an alternate site, you can bring up the Exchange VMs as well as the required Active Directory from the same deduplication storage at the alternate master server.
While NetBackup is serving the VM from backup storage, you can also initiate storage vMotion of the same VM onto production storage. Once storage vMotion is complete simply cancel the Instant Recovery job from NetBackup Activity at a convenient time; the virtual machine is being served from production and NetBackup cleans up the NFS data store.
What are the use cases for NetBackup Instant Recovery for VMware?
- High RTO applications where the virtual machine must come online immediately after an outage in production storage
- Performing development/testing using a copy production workload data in an isolated network (e.g. testing the impact of an OS/application patch, testing a workflow etc.)
- Disaster Recovery for high RTO applications at an alternate site in conjunction with NetBackup Auto Image Replication
- Granular level recovery for virtual machines and applications where Symantec V-Ray enabled recovery is not yet available (e.g. granular recovery from Solaris x86 virtual machines, recovering DB2 from Linux virtual machines etc.)
