A Brief History of Backup

Saving your assets.jpg

"Study the past if you would define the future."
– Confucius

A lot has changed in the backup industry over the past two decades. In honor of World Backup Day, we thought it might be fun to take a look back and see how things have changed over that time. We produced this Infographic to give you the big picture, but read on if you want to know more of the details and how NetBackup and Backup Exec have played an essential role in shaping the backup industry as it exists today.


  • Three-tier architecture – NetBackup’s unique three tier architecture transformed the management of enterprise backup by allowing all policies and management tasks to be orchestrated from a single master server which then controlled a number of media servers that performed the actual data movement, backup, and restore operations. The result allowed a smaller IT staff to manage backup operations for a much larger IT environment.
  • Open file backup – It’s hard to imagine, but there was a time when you had to shut down your applications in order to back them up. In the mid-90s Backup Exec and NetBackup promoted a new open file backup technology based on file system snapshots that enabled live data to be backed up while in use, greatly simplifying the daily backup task and ensuring important files were not skipped. Backup administrators could sleep a lot better after that.
  • Email message-level restore – In the early days of Microsoft Exchange, email backups were database-only, so if you only wanted to restore a specific message you had to first recover the entire email database to an alternate location and then browse for the message you needed. To remedy this problem, Backup Exec pioneered the first generation of Granular Restore Technology, allowing a single email message to be selected and restored from the Backup Exec interface, forever changing how email is restored.
  • Shared storage over SCSI – Large tape libraries had become the norm in the mainframe environment, providing economies of scale for large enterprises. But the open systems environment still required storage to be tethered to a single server, resulting a many small islands of storage including smaller tape libraries and autoloaders, one per server. That was until NetBackup introduced a ground breaking technology that allowed multiple servers to share a single tape library over SCSI. Tape management was never the same after that.
  • Tape multiplexing – In the 90s tape drive performance had grown to the point where the bottleneck shifted to the network. Despite their fast performance, tape drives often sat idle waiting for data to travel over the network. NetBackup solved this problem with tape multiplexing, allowing multiple data streams from different sources to be interleaved into fewer, high performance data streams that could keep the tape drives spinning as fast as they could go. Backup windows were manageable again.
  • Network Data Management Protocol (NDMP) – Filers introduced the NDMP interface as a means of protecting filer data via remote control without agents. NetBackup quickly implemented its NDMP support and later extended its capabilities to allow several different backup methods. Today NDMP is the default, low cost method for protecting filer data.
  • Bare Metal Restore (BMR) – Originally introduced under the name Intelligent Disaster Recovery, the concept of Bare Metal Restore was pioneered by Backup Exec and later proliferated by NetBackup. With BMR technology, full system recoveries were automated and no longer required restore operators to know the details of configuring hardware or installing operating systems. Such automation improved recovery times and eliminated much of the human error common in high-pressure recovery situations.
  • Database Block Level Incremental backup (BLI) – Protecting very large databases (VLDB) has always pushed the limits of backup technology. NetBackup’s early solution to this problem was through tight integration with Veritas Storage Foundation storage checkpoint technology, allowing only block-level changes within databases to be copied while ignoring everything else. BLI backups of Oracle and DB2 could be performed in a fraction of the time, making VLDBs a lot less scary than they used to be to the backup team.
  • FlashBackup – Some of the largest applications at the time, such as the Human Genome project, were now hosting millions of files on a single system. Backing up these systems proved to be very slow as the file system became the primary bottleneck when the number of files reached a certain point. NetBackup designed a revolutionary solution to this problem, bypassing the file system with an image-style backup, but then performing a block map of the file system as part of the catalog post-process. The result was FlashBackup, an ideal way to quickly back up a massive file system without losing the ability to restore a single file.
  • Shared storage over a Fiber Channel SAN – In the 90s large enterprises were using very large tape libraries to protect their mainframes, but could not leverage these investments to protect their LAN systems. That all changed when storage area networks (SAN) were born. Backup Exec launched Shared Storage Option, the world’s first backup solution based on Fiber Channel SANs in partnership with Compaq (now HP), allowing those huge tape libraries to finally be shared by many systems connected via a SAN.
  • Hardware snapshot integration – NetBackup introduced its first generation of snapshot management integration with EMC Timefinder, enabling the NetBackup administrator to incorporate snapshots into their backup strategy. With this capability, NetBackup could orchestrate data synchronization and 3rd-mirror break-off backups on EMC Symmetrix arrays.
  • Vertex Initiative – In the shadow of the Internet Bubble, the convergence of backup and snapshot technology leaped forward with something called the Vertex Initiative, NetBackup’s second generation of snapshot integration. NetBackup delivered integration with several hardware snapshot technologies and allowed customers to perform nondisruptive, snapshot-assisted backups, including off-host and server-free backups.
  • Virtual Tape Libraries (VTL) – As the costs of disk began to compete with tape, the first wave of disk-based backup technologies were dominated by virtual tape libraries. While supporting VTLs was a straightforward qualification effort, NetBackup took it one step further and provided a level of integration that allowed VTLs to communicate with NetBackup so image duplication and data retention could be accomplished within the NetBackup policy framework.
  • Disk-to-Disk-to-Tape (D2D2T) – The next wave of disk-based backup began to incorporate standard JBOD systems into the mix as the first stage of a tiered backup architecture that combined disk and tape. NetBackup pushed the state-of-the-art forward with its flexible and advanced disk-based backup technologies, allowing disk capacity to be more easily managed using water marks and more logical operations that broke free of the limits of legacy tape methods.
  • Global data deduplication – The continued growth of disk as a backup target offered new opportunities to exploit the advantages of disk over tape. One of the most important innovations was deduplication technology which allowed data redundancies to be eliminated on a scale far beyond legacy compression technologies. NetBackup introduced its first generation of data deduplication in 2005, reducing storage requirements by as much as 90% or more. With this technology, disk finally became more cost-effective than tape as a primary backup target.
  • V-Ray – As VMware took the IT world by storm, NetBackup was first out of the gate to deliver truly advanced capability to this platform. Later dubbed “V-Ray,” NetBackup’s first generation of VMware integration did something nobody else could do: select and restore a single file from a VMDK-level backup. No longer would backup admins have to choose between VMDK image-level backups or VM guest-level backups. Cited by the analysts as one of NetBackup’s most distinctive and industry-leading capabilities, V-Ray technology helped NetBackup win Best of VMWorld in 2007, the first of many awards to follow.
  • OpenStorage Technology (OST) – The next wave of disk-based backup saw the proliferation of intelligent storage devices and appliances. Rather than put together a string of ad-hoc integrations with different vendors, NetBackup created an interface to allow these devices to “plug-in”, allowing NetBackup to directly leverage the intelligent capabilities of these devices. Launched as the OpenStorage Technology API, this interface has all but become a de facto standard in the backup industry.
  • NetBackup 5200 Integrated Backup Appliance – NetBackup decided to enter the backup appliance market with solution that combined software and hardware into a fully integrated platform. Since then the NetBackup Appliance has outpaced the growth of all other backup appliances in the market. Unlike so called target appliances, the NetBackup Appliance replaces storage, media servers, and storage interconnects, greatly simplifying the deployment and maintenance of NetBackup infrastructure.
  • Accelerator – The latest innovation to transform the backup world came with NetBackup 7.5. NetBackup combined several key technologies including optimized synthetics and changed block tracking to create a method of backup that permanently eliminates the need to perform full backups. When combined with deduplication technology, daily backup times can be reduced by an order of magnitude over traditional full backup methods. Time-consuming full backups are now a thing of the past.

Despite all that has changed over the past two decades, one thing hasn't: a reliable backup remains as the final solution when all else fails. No matter how technology evolves in the future, having a spare copy in your back pocket will always bail you out.

1 Comment

This blog post made me feel old - thanks for that!

 

I wanted to add one item for completeness. I remember talking to Curtis Preston a few years ago and he was recalling the one thing that sold him on NetBackup in the early days - it was policy directives. Instead of having to enter in the pathnames, mountpoints or drivenames individually and adjusting them as required when changes occurred, you could enter in a  directive like "ALL_LOCAL_DRIVES".  This small innovation took the notion of policies which could manage many systems as one, and made it easier to ensure that the right data was protected even in a changing environment. While this feature sounds small when you look in the rear view mirror, it's still a very important concept that we continue to extend. How cool is it that you can test and tune your environment with GEN_DATA directives? How cool is it that you can protect all the Virtual Machines on an virtual server or just a subset, without having to manually change your policies?

 

It just keeps getting better.