Backup & Recovery

Overview:
My Organization is an IT based system providing Custom Software’s for Banking, Insurance and more. It has got lot of branches in India and US. Each Branch has its own Data Center with 200 more Servers and 2000 more Desktops. I am staying in Head office; so obviously my Office is little bigger than other branches. And recently we have introduced VMware in our environment and currently my VM Data Center is running with 2 Virtual Centers, 20 ESX Servers, and 200 more VM Machines.

Roadblocks:
As we got lot of Development projects users keep on updating the contents to the servers and some servers running based on SAN Storage; and the data size is in 100 TB’s so it is not an easy task to maintain backup for all servers. Also it involves more Human Resources to handle these operations and in case of issues it is too hard to troubleshoot and make the data available at any time. As we got more Banking production servers, it is not acceptable to lose even a single file in case of disaster. Now a day’s everything is getting virtualized, my Data center is getting filled with ESX Servers. Faster and granular recovery of VM Machines should be considered. Also we have to consider the Cost spending on Resources like Backup Software, and Add in feature License Cost.
Developers develop their Project modules from their Desktops and upload the contents to the Servers, so we should consider the Anti Virus and Spyware functionalities to keep all Developers desktops more Stable. Because Viruses from Users Desktop can get into the Servers and do malfunctioning. Also all Desktop’s should be updated properly to protect from upcoming Viruses. All this update process should be automated so that no one need not sit to do these things. Update missed Desktops should be reported properly with details, so that Antivirus Admin can troubleshoot easily. And finally the Cost spend to implement and maintain this setup must be considerable.

Total Analysis:
We had formed an Internal Team to analysis the cost and time involved in doing these operations. They have produced a detailed survey Report; here is the Major Fields of the Report:
1. Cost:
     a. Cost spend on Backup Software Purchase and Add in Feature License.
     b. Labor cost on Backup, manage and troubleshooting Issues.
     c. Cost spends on Antivirus Software purchase.
     d. Labor cost spends for Scanning and Patching the Desktop Clients.

2. Time:
     a. Time spend to implement, manage and maintain data backup for all servers.
     b. Time taken to troubleshoot Backup issues
     c. Time taken to recover the Data’s in case of disaster.
     d. Time taken to Patch and scan Antivirus updates on Desktops.

Breaking the Roadblocks:

As per this Analysis Report we came up with requirement sheet that will lead will Company profit to roof.


Backup and Recovery:


Single Console Management:
Backup software should provide a Centralized Console to Implement and Manage Backup and Recovery. More compatible with DISK Storage and as well as VTL. It should allow to customization so that it can fit into the Customer need. More reliable to retain data’s based on the retention.

Heterogeneous Support:
We use more Windows Servers and some servers running in UNIX and Mac. It should work with more platforms.

Application Support with advanced recovery:
We use MS SQL, MS Exchange, MS Share point, VMware, and Oracle and more; Backup software should be very much integrated to applications to protect and recover data’s. It should provide granular recovery with minimized time window. It should fit more with VMware as all our servers becoming virtualized

Minimized time Window:
My Production Servers should not be disturbed for more time. Backup should complete within limited time window and recovery of individual files and Full Machine should be carried out in less time period.

Backup copies to Off Shore Locations:
My Backed up data should be duplicated to send for Off Shore locations to avoid data loss in case of disaster. Backup Duplication should not extend my Backup time window.

Disaster Recovery:
Recovery of full Physical Machine as well as VM Machine should be supported.

Reporting:
Backup successful and failure Reports should be generated in day wise and mailed to respective person. Backup admin should be alerted the completions and failure of backup instantly. Also it should allow customizing the report as per requirement.


Virus Protection:

Unauthorized access and Virus:
Anti Virus software should allow to configure Policy rules to protect unauthorized access and unknown files, file types.

Centralized console:
It should have centralized console so that antivirus administrator can easily configure policies and access rights. Patch report and non complaint machine reports should be easily generated.

Automated Patching:
Patching of updates on Desktops should be automated and it should allow choosing the time window based on the user requirement.

Machine Performance:
Full System Scan or Software patching should not disturb the Machine performance and users data.

Products Implemented:
Symantec Products gave us a best suit for our requirements; we have implemented VERITAS NetBackup 6.5 and Endpoint Security.

Transformation:

VEITAS NetBackup Implementation:
We were running a Backup Application that backs up data to Tape Storage unit. That gave us limited features and also it didn’t fit into our environment. We have deployed VERITAS NetBackup as follows:
First we have created a test environment to test all our needs like Application backup and granular recovery, Tape to Disk Backup’s, Deduplication.
Backup & Restore Test got 100% successful rate, we have prepared a Master Server brought up that to live.
Configuration done as per Veritas Guides.
Initially we removed all Development servers and installed Veritas NetBackup agent on those servers.
Segregated the Application servers and configured agent backups for those servers.
We kept that Old Backup Server ahead, whenever we require restoration from old data we get it restored from that server.
We had a full Backup Selection analysis with Server admin guys to get Backup Selection of individual servers and this help us to remove unwanted data’s getting backed up.
Configured Disk to Tape Storage unit schedules. And started a Full Backup of those Development servers.
Everything went fine; we got a request from High level Management to migrate all Production servers to VERITAS NetBackup.
We have migrated all Production servers to VERITAS.
We have Configured VMware proxy server to test the VM Machines backup and that also went fine, both full image level and file level recovery was successful.
We have moved all VM’s to VCB Backup.
Finally we have scheduled all backups, so need of Backup Admin interaction.
For reporting purpose we have implemented NOM with the help of Symantec Technical Support.
Now Our Backup server is my Pet, It do what I say, I see my master only I got any bad alerts from NOM.

Endpoint Security Implementation:
We were using Multiple Anti Virus Software’s in our setup that makes us to go for multiple servers for managing these clients. It has got lot and lot more limitations in PC Scanning and Policy deployment and management. We have rolled out to Symantec Endpoint Security as follows:
Prepared a list of total machines in our setup. Moved all Machines to Domain and Configured Domain Admins Accounts to have admin access on all machines.
Created a Script to remove all Antivirus packages from 10 % of the total machines.
Executed that script to remove all other Antivirus Packages.
Configured the Anti Virus update server as per document.
Configured Policies to Control and patch the Machines.
Configured patch policy to patch all machines with recently downloaded patch at everyday night 3 PM
After a complete patch update process, generated a report to verify the Patch Level of clients.
Our organization policy not allows playing any media files and not allowed to use USB devices, we have disabled all this restrictions from our Antivirus Server from a single console.
As per our today’s report, My non compliance machines rate is 1 %
Also I need not worry about audits; I can get all details in minutes.

Thanks to:
VERITAS NetBackup 6.5
Symantec Endpoint Security
NetBackup Operation Manager.
Special thank to NBU 6.5.3 – It gave us granular recovery of VM, SharePoint, and Exchange.

Thanks,
Karthikeyan Sundaram.

Comments
Awaiting your reply as you have implemented it on Vmware & why y0u have not used VMDK for Backup & spend on 3rd party backup tools

Prachia,

What you mean by "VMDK for Backup"?  Are you pointing to VMware build in VM Backup
Veritas NetBackup is certified for VMware Virtual machine backup and it's giving you the flexible options for Backup and Restore.
"One single path backup-two level restore(Image level and file level)" and as well as Incremental backup for VM
No need to install/licence veritas for individual VM
BTW,here are the challenges that we face.
1. We were unable to run multiple copies parallel copies of VM backup
      Remedy: We implemented inline backup
2. Some of the snapshot deletions failed with some VM ware errors
    Remedy: solved by scheduling VCB script to delete the missed out snapshots
3. We were unable to run backup for VM that do not have NIC Card(We got a project they require machines without network card)
    Remedy: Implemented display name based backup
4. We were Unable to do automatic VM restore converstion (restore all virtual machine files to virtual machine server automatically using VMware     
     converter)
    Remedy: Downgrade to VMware convertor 3.0.2


Hello friends,
we have more than 20 years of manufacturing the auto horizontal pillow packaging machine. if you are interested in, pls visit our website:www.qdfengye.cn
Thanks & best regards,
Ms.Maggie Liu