Security practices for malware scanning of backup images

[sgk]

Veritas NetBackup introduced malware scanning for backups a few versions ago, which has since matured. Some competitors refer to it as threat hunting or threat analysis. This blog explores the potential risks of those who search for threats inadvertently becoming targets themselves. 

At a high level, this process involves scanning backups of hosts and data repositories for malware or ransomware rather than scanning in the production environment. Initially, it may seem straightforward, but the complexity arises when examining how backup software conducts malware scanning. 

Different backup vendors have adopted varying strategies for malware scanning of backups.   

• Threat hunting, defined as search for known file signatures 
• Live mounting backups and exposing them through a running virtual machine 
• Exposing files and folders using CIFS/NFS protocols 
• Directly restoring backup data versus virtually exposing the data 

Each of these strategies must consider the potential security risks and how to both protect the data and how to protect the environment.  

Security from the direction of the scanning software: When it comes to security in scanning software, whether it searches for various indicators of compromise (IOCs), such as signatures of known bad files within the product, or relies on a trusted third-party security scanning tool to do similar search or matching, malware scanning is a critical consideration. The design and implementation of this process must always remember that the files being assessed may not be harmless; they are being evaluated to detect potential malware. 

To draw an analogy, this is not like a casual gathering at a family wedding, where you warmly welcome guests you already met. Instead, it's more akin to airport security, where security personnel treat everyone as a potential security concern. They don't conduct screenings in a haphazard manner. Instead, they ask you to empty your pockets, remove jackets and shoes, and raise your hands to ensure the safety of everyone during the scan. Any possible weapons in the carry-on luggage are not accessible to the commuter during the safety scan. The carry-on luggage gets scanned separately without any potential chance of being used for any harm.

Similarly, regardless of the method used for potential malware scanning, it must be executed with great care. The scanning session should be isolated, ensuring that other users who can log into the scanning host cannot see or access the files being scanned for potential malware. Even if the same user initiates another session, that session should not have access to the files under scrutiny for potential malware. This isolation is crucial for maintaining the security and integrity of the scanning process. 

Security from the direction of the backup image exposure:  

Let’s look at three alternatives for exposing backup images for malware scanning. There are a few important considerations to keep in mind.  

1. One way to conduct malware scanning is by using snapshots stored within the backup image. For instance, with virtual machines, you can create a VM directly from the backup image. However, this approach introduces some challenges. It adds extra load to the VM host environment, potentially allowing malicious code to remain undetected. Furthermore, this method grants full access to the entire VM from the scan host using methods like SMB, which is not always straightforward or feasible. 
2. A second method involves mounting the snapshot on the scan host. In the case of virtual machines, although the backup of the VM is immutable, meaning it can't be altered, there is no control over what can or cannot be executed from the scan host's side. The same considerations apply to NAS backups when using the snapshot method. In this scenario, due to immutability, there's no way to modify file permissions for the exposed backup image, to prevent accidental execution of files. 
3. There is a third method of threat scan used by at least one data protection vendor that may seem simple, but is highly risky. It involves restoring files to the scan host, scanning them, and then "immediately" deleting them. This approach essentially places trust(?) in potential malware not to take any malicious action while it's temporarily free to run on the scan host without any restrictions, even if only for a short period. 

Operating System and mounting method: Compared to the first two areas of consideration, this one is relatively straightforward. Share type choices are NFS and SMB. Window and Linux (RHEL, SLES) scan hosts are the choices of OS for a VM or server which can scan both the share types. Linux scan host with NFS share type and Windows with SMB share type may be most common.  

This covers the various combinations considered while offering malware scans of backups. It is important to try and cover all these combinations in every way possible. 

What is the secure way to do it and how Veritas NetBackup solves for above considerations?

To enhance the security of NetBackup's malware scans, there are several behind-the-scenes measures in place. One crucial element is the VPFS, or, Veritas Provisioning File System, which acts like a pseudo file system using the backup images from NetBackup as its foundation. VPFS also helps in supporting the browsing functionality on block or object storage used for storing backups. Many other vendors only support backups stored on block-based storage for such instant browsing. 

This “virtual” file system exposes a view of an image without allowing the end user, or the scan process to change the original files. It is more like a “copy on write” snapshot in many ways, but it isn’t a full recovery, so you don’t have to move it across the network. Some index engines require making a fully rehydrated copy, thus either slowing the speed of the scans, or, requiring costly resource additions.  

The best approach to malware scanning lives in answers for two simple questions.  

1. Can NetBackup change the permissions of the files in the backup image when NetBackup exposes those to scan hosts without changing the original backup image? If the answer is yes, NetBackup does it.  
2. Can NetBackup ensure that the scan host's session is isolated from other users and sessions on that host? If it's possible, NetBackup takes that step as well.  

Think of it as conducting a virus test in a controlled laboratory environment requiring extreme caution. Malware scanning is no different. The key distinction is that we can't perform a secure scan on production data because we can't make any alterations there, even at the presentation layer. By conducting the scan on the backup data, we gain the flexibility needed for thorough and secure malware scanning. As a result, for restore time scan, one could restore files with just the read permissions for the scan and apply original permissions after the scan. 

You have learnt the ingredients of a secure malware scanning of backup images. 

Intrigued? Join our Veritas REDLab group to see such unique topics discussed. Don’t worry, it is free and easy to join. Sign up on vox.veritas.com, and then go to this group URL and join. https://vox.veritas.com/t5/Veritas-REDLab/gh-p/VeritasREDLab